WAF test with serious shortcomings
We welcome any effort to test security products for their effectiveness and efficiency rather than just rating them based on their feature set (as many other analysts do). SecureIQLab last conducted such a test in Q1/2024. Unfortunately, the current test results from SecureIQLab do not come close to the protection potential of Airlock Gateway. There are two main reasons for this:
- Insufficient Airlock configuration: Important protection functions of Airlock Gateway were not active at all.
- Missing test scenarios: In many areas, the test did not include any realistic attacks that Airlock Gateway could have defended against.
The table below lists concrete examples of such test deficiencies.
Shallow tests give a false impression of high security
At first glance, the list of tests carried out by SecureIQLab is impressively long. Numerous formulas also give the impression of a scientifically sophisticated validation. However, a closer look reveals a different picture. Many attack scenarios are so superficial that the results have little to say about the effectiveness of the products tested:
- Questionable bot tests:
The Bot Protection section checks whether the WAF blocks if an attacker identifies themselves via the user agent string. This "defense" is not very effective in practice because even an inexperienced attacker is unlikely to voluntarily reveal his identity. Any reasonably clever (bad) bot rather disguises its identity by pretending to be a normal user. To do this, the bot simulates a browser with an unsuspicious user agent string. Smart bots even interpret JavaScript code and imitate human mouse movements. A more meaningful benchmark for WAFs would be to also detect bots that do not fall for a simple user agent check. To prevent smart bots, the protection needs to be much more intelligent. This includes looking at behavioral characteristics such as the timing and sequence of page views. Behavior-based bot protection analyzes how the authorized users of a specific application behave. Airlock Anomaly Shield uses this to calculate a tailored machine learning model for each application, which can be used to quickly identify conspicuous behavioral outliers and thus also block unwanted bots. Unfortunately, Anomaly Shield was not used in this test.
- Filter Evasion not tested:
The report gives the impression that many WAFs offer 100% protection against attacks such as SQL injection or XSS, although their protection filters can be easily bypassed. These types of WAFs are typically optimized for publicly available attack examples and offer no effective protection against filter evasion attacks. Filter evasion is an obfuscation tactic used by an attacker to bypass the filters of a web application firewall. Airlock Gateway is currently the only WAF on the market that pays out a reward for every filter evasion found as part of a bug bounty program. This program has now been running for 4 years, and it has allowed us to develop the most secure injection filters on the market. This protection goes far beyond blocking bots or pentesting tools and effectively prevents the exploitation of an injection vulnerability. Filter evasion was hardly taken into account in the test, which gives a misleading picture of the level of protection provided by the products.
These examples make it quite obvious: To fend off complex attacks in a WAF, you need much more intelligent protection functions and a lot of experience. This is why we have been analyzing attack risks on web applications and APIs for 20 years and incorporating our findings into our products. Our Swiss engineering team is constantly investing in new technologies, from which our customers regularly benefit with new versions of Airlock Gateway.
We are very grateful to the testers at SecureIQLab for drawing our attention to a vulnerability in the management interface of Airlock Gateway. Thanks to the immediate reaction of our development team, this gap was closed within 48 hours with a hotfix and all customers were informed. The professional and quick reaction is one of the reasons why our customers and partners have so much confidence in us.
We hope that SecureIQLab will take our feedback into account, so that the next iteration of the WAAP tests can take full advantage of Airlock's protection features.
Specific test deficiencies
In the test by SecureIQLab, numerous Airlock Gateway protection features were either not active at all, or the test did not contain any suitable scenarios for this kind of attack prevention.
Airlock | Potential | Test |
|---|---|---|
| Strong multi-factor authentication Upstream multi-factor authentication in combination with Airlock IAM can compensate for weaknesses in the application login. |
| ⚠️ Airlock protection was inactive |
| Hardened Deny Rules Block lists for generic detection of common attacks such as the OWASP Top 10 | SQL Injection, Command Injection, XSS etc. kombiniert mit Techniken zur Filter Evasion | ⛔️ No test scenario for Filter Evasion |
| Anomaly Shield Behavior-based anomaly detection, tailored to the respective application/API. | Malicious Bots, Vulnerability Scans, Brute Force Attacks, Content Scraping, Denial-of-Service, Account Hijacking, Password Spraying, Click Fraud | ⚠️ Airlock protection was inactive |
| API Protection: OpenAPI Schema Validation Schema validation is used to block API calls that do not correspond to the interface defined by the provider. | API Abuse, Brute Force Attacks, Parameter Tampering, Enumeration Attacks, Invalid Data Submission, Type Confusion Attacks | ⚠️ Airlock protection was inactive |
| Dynamic IP blacklisting Temporary blocking of an IP address in the event of repeated attacks. |
| ⚠️ Airlock protection was inactive |
| Threat Intelligence (Webroot IP Reputation Feed) Thanks to a real-time feed, all access from known botnets, compromised systems and other untrusted addresses is blocked. | DoS- und andere Angriffe von bekannten Botnetzen, kompromittierten Systemen und nicht vertrauenswürden Adressen. | ⚠️ Airlock protection was inactive |
| Cookie Protection Cookies are encrypted or hidden in a cookie store for protection against spoofing and manipulation. | Cookie Manipulation, Cookie Stealing, Session Hijacking | ⛔️ No test scenario for cookie protection |
| URL Encryption & Form Protection prevents the manipulation of HTML form fields and the URL (e.g. changing/adding parameters) | Forceful Browsing, Parameter Tampering, Injection-Angriffe, CSRF-Attacken, Path Traversal etc. | ⚠️ Airlock protection was inactive |
| CSRF Tokens stop CSRF vulnerabilities from being exploited. This prevents sensitive actions from being carried out unnoticed in the name of the victim. | CSRF-Attacks | ⛔️ No test scenario for CSRF |
| Request Rate Limiting can prevent DoS attacks or login bruteforce attacks by limiting the request frequency on certain paths. | Bruteforce Attacks, DoS Attacks | ⚠️ Airlock protection was inactive |