Engineering 12 min read

Second benchmark: WAFio v2, SafeLine and ModSecurity through three test tools

We ran WAFio v2, SafeLine 9.4.4 and ModSecurity with the OWASP Core Rule Set through blazehttp, GoTestWAF and WAFtester on one machine with the same limits. The numbers, the weak spots, and what the tools do not tell.

The WAFio logo and the words WAFio v2 in a dark test lab, where three beams of light from measuring instruments pass through a teal, a grey and an amber glass panel

This is the follow-up to our first benchmark. That post measured one build of WAFio v2 with one tool, blazehttp, and took the numbers of the other products from the blazehttp README. This time we ran three products ourselves, on one machine, through three test tools.

The engines also moved on since the first post. On the same blazehttp samples it reported 90.61 % detection with 0.06 % false positives. The build tested here reaches 94.78 % with 0.02 %.

What was measured

  • WAFio v2: the WAF agent and its semantic engines, built on 17 November 2026 from the main branches (agent commit 61edcfd). It ran with the default rules of the product, with two things switched off because every test comes from one address: the block policy and the L7 flood limit. It is the proxy with the engines, without a control plane, so there is no bot list, no GeoIP and no IP reputation in this test.
  • SafeLine 9.4.4, the community edition, in two settings: the modes it ships with, and all seventeen detection modules set to strict.
  • ModSecurity v3.0.17 with the OWASP Core Rule Set 4.30.0, the official container image, in blocking mode with its default anomaly thresholds, at paranoia level 1 (the default) and at level 4.
  • Three tools: blazehttp with the sample set of the SafeLine makers (575 attack samples and 33,094 normal ones), GoTestWAF 0.5.8 and WAFtester 2.9.50.
  • One machine: a virtual machine with 4 virtual CPUs (AMD EPYC 9354P 32-Core Processor) and 15.6 GiB of memory. Every product got the same limits, 2 CPUs and 4 GB, and the same backend, an nginx that answers 200. Only one product ran at a time.

blazehttp: detection and false positives on 33,669 samples

ProductDetection (blocked / attacks)False positives (blocked / normal)AccuracyAverage time
WAFio v294.78 % (545 / 575)0.02 % (8 / 33,094)99.89 %7.43 ms
ModSecurity CRS, level 170.78 % (407 / 575)18.22 % (6,030 / 33,094)81.59 %37.92 ms
ModSecurity CRS, level 493.91 % (540 / 575)50.98 % (16,870 / 33,094)49.79 %65.81 ms
SafeLine, default61.91 % (356 / 575)0.05 % (17 / 33,094)99.30 %5.95 ms
SafeLine, strict69.22 % (398 / 575)0.14 % (45 / 33,094)99.34 %6.13 ms

Colours: green: best light green: second yellow: middle orange: second to last red: last of the five products in that column. Equal values share a colour; when all five are equal, none has one.

blazehttp on 575 attack and 33,094 normal samples, 17 November 2026. All five rows are our own measurements.

Detection rate (higher is better)

WAFio v2
94.78 %
ModSecurity CRS, level 1
70.78 %
ModSecurity CRS, level 4
93.91 %
SafeLine, default
61.91 %
SafeLine, strict
69.22 %

False positive rate (lower is better)

WAFio v2
0.02 %
ModSecurity CRS, level 1
18.22 %
ModSecurity CRS, level 4
50.98 %
SafeLine, default
0.05 %
SafeLine, strict
0.14 %

ModSecurity at level 4 finds nearly as many attacks as WAFio, 93.91 % against 94.78 %, but it also blocks 50.98 % of the normal samples. At level 1 it blocks 18.22 % of them and finds 70.78 % of the attacks. SafeLine blocks very few normal samples, 0.05 % by default and 0.14 % in strict, and finds 61.91 % and 69.22 % of the attacks.

One thing favours WAFio and SafeLine here, and we measured how much. The tools address the WAF by its IP address, and the Core Rule Set adds points to every request whose Host header is a number. When the tools use a host name, ModSecurity blocks fewer normal samples: 14.79 % at level 1 and 48.91 % at level 4. Its detection does not change, and neither do the numbers of WAFio and SafeLine.

ModSecurity was also the slowest in this test: 37.92 ms and 65.81 ms on average, against 7.43 ms for WAFio and 5.95 ms for SafeLine. It used all of its 2 CPUs during the run.

GoTestWAF: 675 attacks and 141 normal requests

ProductAttacks blocked / sentNormal requests blocked / sentOverall score of the tool
WAFio v2633 / 6750 / 14198.42
ModSecurity CRS, level 1325 / 67513 / 14163.39
ModSecurity CRS, level 4502 / 675141 / 14168.66
SafeLine, default314 / 6750 / 14158.28
SafeLine, strict400 / 6753 / 14175.30

Colours: green: best light green: second yellow: middle orange: second to last red: last of the five products in that column. Equal values share a colour; when all five are equal, none has one.

WAFio blocks 633 of 675 attacks and none of the 141 normal requests. ModSecurity at level 4 blocks 502 attacks, and every one of the normal requests. SafeLine in strict mode blocks 400 attacks and 3 normal requests.

WAFio is not complete here. It lets through 13 of the 24 mail injection cases, the five large-body command injection cases of the community set, two of three raw-request command injection cases and three of seven XML injection cases. The table below has every test set.

Test setWAFio v2ModSecurity level 1ModSecurity level 4SafeLine defaultSafeLine strict
community (body size cases)10 / 1512 / 1212 / 120 / 5-
community/community-lfi8 / 88 / 88 / 82 / 82 / 8
community/community-lfi-multipart1 / 2--0 / 20 / 2
community/community-rce4 / 42 / 44 / 40 / 44 / 4
community/community-rce-rawrequests1 / 33 / 33 / 30 / 33 / 3
community/community-sqli12 / 1212 / 1212 / 128 / 128 / 12
community/community-user-agent8 / 96 / 96 / 91 / 91 / 9
community/community-xss99 / 10498 / 104104 / 10482 / 10496 / 104
community/community-xxe2 / 20 / 11 / 10 / 20 / 2
owasp/crlf7 / 76 / 77 / 70 / 70 / 7
owasp/ldap-injection24 / 242 / 2418 / 240 / 240 / 24
owasp/mail-injection11 / 243 / 2415 / 240 / 240 / 24
owasp/nosql-injection50 / 5012 / 5033 / 5010 / 5010 / 50
owasp/path-traversal19 / 206 / 2014 / 202 / 202 / 20
owasp/rce6 / 62 / 65 / 60 / 62 / 6
owasp/rce-urlparam6 / 93 / 99 / 93 / 93 / 9
owasp/rce-urlpath3 / 31 / 32 / 30 / 31 / 3
owasp/shell-injection32 / 326 / 3224 / 320 / 328 / 32
owasp/sql-injection48 / 4820 / 4832 / 4848 / 4848 / 48
owasp/ss-include24 / 2412 / 2415 / 240 / 240 / 24
owasp/sst-injection24 / 2411 / 2415 / 248 / 248 / 24
owasp/xml-injection4 / 72 / 77 / 70 / 70 / 7
owasp/xss-scripting216 / 22490 / 223142 / 223144 / 223194 / 223
owasp-api/graphql-----
owasp-api/graphql-post-----
owasp-api/grpc-----
owasp-api/non-crud2 / 22 / 22 / 21 / 21 / 2
owasp-api/rest7 / 74 / 77 / 73 / 75 / 7
owasp-api/soap5 / 52 / 55 / 52 / 54 / 5

Colours: green: best light green: second yellow: middle orange: second to last red: last of the five products in that row. Equal values share a colour; when all are equal, and for a dash, there is none.

Each cell shows the share of attacks blocked and, under it, blocked / attacks the tool could judge. A total smaller than in the other columns, or a dash, means the tool could not judge some requests. Read the colours with the table above: ModSecurity at level 4 is green in many rows and also blocks every normal request.

WAFtester: 3,473 tests in 24 categories

WAFtester sends two kinds of test. The first kind is attacks, which a WAF should block. The second is about 412 requests the tool considers harmless, which a WAF should let through. A good WAF blocks many attacks and few of the requests that should pass. The table below keeps the two apart.

ProductAttacks blockedRequests that should pass, but were blockedTests the tool skipped
WAFio v283.48 % (2,497 of 2,991)123 of 41275
ModSecurity CRS, level 183.62 % (2,558 of 3,059)140 of 4112
ModSecurity CRS, level 4100.00 % (3,061 of 3,061)412 of 4120
SafeLine, default33.17 % (930 of 2,804)17 of 41099
SafeLine, strict39.00 % (987 of 2,531)20 of 412575

Colours: green: best light green: second yellow: middle orange: second to last red: last of the five products in that column. Equal values share a colour; when all five are equal, none has one.

ModSecurity at level 4 blocks every attack, and also all 412 requests that should pass. A WAF that blocks everything cannot be used, so that 100 % is not a good result. The score WAFtester itself reports does not separate the two kinds of test: it counts every blocked test as a success. That is why level 4 gets 100.00 % in the report of the tool, and why this table does not use that score.

WAFio and ModSecurity at level 1 are level. WAFio blocks 83.48 % of the attacks and wrongly blocks 123 requests that should pass. ModSecurity at level 1 blocks 83.62 % of the attacks and wrongly blocks 140. We have not gone through those wrongly blocked requests one by one yet: some are requests a WAF may well refuse, such as malformed protocol input, and some are false blocks we have to fix.

WAFio is weak in several categories: it blocks none of the 60 GraphQL tests, 27 of 96 service-specific tests, 105 of 227 authentication tests and 27 of 49 bypass tests. ModSecurity at level 1 does better in all four.

The SafeLine numbers of this tool are the least firm. WAFtester stops sending when it thinks it has been banned, and with SafeLine it skipped hundreds of tests. Over three runs on the same day the share of attacks SafeLine blocked moved between 31.79 % and 34.64 % by default and between 39.00 % and 42.73 % in strict. For WAFio and ModSecurity at level 1 it moved by less than half a point.

CategoryWAFio v2ModSecurity level 1ModSecurity level 4SafeLine defaultSafeLine strict
injection500 / 540508 / 599599 / 599143 / 58991 / 407
xss396 / 408389 / 408408 / 408345 / 404205 / 229
auth105 / 227161 / 228354 / 35440 / 22544 / 225
ssrf270 / 277231 / 277277 / 277121 / 277141 / 262
waf-bypass214 / 241215 / 242242 / 24212 / 15177 / 240
traversal187 / 195190 / 206206 / 20644 / 19961 / 74
protocol128 / 158144 / 161161 / 1612 / 1413 / 141
sqli153 / 160159 / 160160 / 160139 / 151139 / 150
logic147 / 147147 / 147147 / 1470 / 1460 / 146
waf-validation83 / 10975 / 107110 / 11013 / 10721 / 109
service-specific27 / 9642 / 96100 / 1007 / 9619 / 96
fuzzing89 / 9090 / 9090 / 902 / 302 / 28
deserialization80 / 8079 / 8080 / 801 / 744 / 74
fuzz4 / 3010 / 3780 / 800 / 300 / 30
upload52 / 7017 / 7070 / 702 / 702 / 70
cache60 / 6060 / 6060 / 6015 / 6018 / 60
graphql0 / 6040 / 6060 / 600 / 600 / 60
ratelimit2 / 92 / 951 / 510 / 91 / 9
bypass27 / 4937 / 4950 / 5019 / 4921 / 49
ai16 / 1620 / 2046 / 4613 / 1415 / 15
obfuscation32 / 4032 / 4040 / 4020 / 4028 / 40
media14 / 1415 / 1533 / 336 / 1111 / 13
owasp-top1022 / 2622 / 2526 / 263 / 244 / 24
regression12 / 1213 / 1323 / 23--

Colours: green: best light green: second yellow: middle orange: second to last red: last of the five products in that row. Equal values share a colour; when all are equal, and for a dash, there is none.

Each cell shows the share of tests blocked and, under it, blocked / tests with a result. A total smaller than in the other columns means the tool skipped tests. These categories hold tests the tool expects a WAF to let through, so a high share there is not all good: auth, waf-bypass, protocol, logic, waf-validation, service-specific, fuzzing, deserialization, fuzz, ratelimit, bypass, ai, media, owasp-top10, regression.

How stable the numbers are

We ran everything three times: the run reported above, the same run again 45 minutes later, and a run where the tools use a host name. blazehttp and GoTestWAF gave the same counts for every product in the first two runs, with one exception: ModSecurity at level 4 blocked 16,870 normal samples the first time and 16,871 the second.

ProductRunblazehttp attacks blockedblazehttp normal blockedGoTestWAF attacks blockedWAFtester attacks blocked
WAFio v2first545 / 5758 / 33,094633 / 67583.48 %
WAFio v2repeat545 / 5758 / 33,094633 / 67583.50 %
WAFio v2host name545 / 5758 / 33,094633 / 67583.60 %
ModSecurity CRS, level 1first407 / 5756,030 / 33,094325 / 67583.62 %
ModSecurity CRS, level 1repeat407 / 5756,030 / 33,094325 / 67583.63 %
ModSecurity CRS, level 1host name407 / 5754,895 / 33,094325 / 67583.90 %
ModSecurity CRS, level 4first540 / 57516,870 / 33,094502 / 675100.00 %
ModSecurity CRS, level 4repeat540 / 57516,871 / 33,094502 / 675100.00 %
ModSecurity CRS, level 4host name540 / 57516,186 / 33,094500 / 67598.37 %
SafeLine, defaultfirst356 / 57517 / 33,094314 / 67533.17 %
SafeLine, defaultrepeat356 / 57517 / 33,094314 / 67531.79 %
SafeLine, defaulthost name356 / 57517 / 33,094314 / 67534.64 %
SafeLine, strictfirst398 / 57545 / 33,094400 / 67539.00 %
SafeLine, strictrepeat398 / 57545 / 33,094400 / 67542.01 %
SafeLine, stricthost name398 / 57545 / 33,094400 / 67542.73 %

Colours: green: best light green: second yellow: middle orange: second to last red: last of the five products in that column, within the same run.

What these numbers do not say

  • None of the three is a blind test for us. All three sample sets are public, and we have worked on misses from each of them. A WAF tuned on a test does better on that test.
  • The other products ran with their defaults. A ModSecurity installation that is looked after has exclusions for its application and blocks far fewer normal requests than the numbers here. SafeLine ran with its containers limited to 2 CPUs and 4 GB and off the host network, which is not how it installs itself.
  • This is not the whole of WAFio. The test covers the proxy and the semantic engines. Bot checks, GeoIP, IP reputation and the control plane were not part of it.
  • A test tool measures what its authors chose to send. The three tools disagree about the same product, as the tables show. No single number here is the detection rate of a WAF.

Conclusion

  • WAFio v2 blocked the most attacks without blocking normal traffic in two of the three tools. blazehttp: 94.78 % detection with 0.02 % false positives. GoTestWAF: 633 of 675 attacks and none of the 141 normal requests.
  • ModSecurity with the Core Rule Set finds many attacks, at the price of normal traffic. At level 4 it comes close to WAFio on blazehttp (93.91 %), but it blocks 50.98 % of the normal samples there and every normal request of GoTestWAF. At level 1 it finds 70.78 % and still blocks 18.22 % of the normal samples before any tuning. On the WAFtester attacks, level 1 and WAFio are level.
  • SafeLine blocks very little normal traffic, and finds fewer attacks. Its false positive rate on blazehttp is 0.05 % to 0.14 %, and on WAFtester it wrongly blocks the fewest requests (17 to 20, against 123 for WAFio). It finds 61.91 % to 69.22 % of the blazehttp attacks and 314 to 400 of the 675 GoTestWAF attacks.
  • WAFio has work left. GraphQL, authentication, service-specific and bypass tests in WAFtester, mail injection in GoTestWAF, and the 123 WAFtester requests it blocks although they should pass.
  • Test on your own traffic before you choose. These are public sample sets and default settings. They show how the products differ, not how each one behaves in front of your application.

Versions and method

  • When. Every measurement in this post was taken on 17 November 2026.
  • What was tested. WAFio v2 at agent commit 61edcfd. SafeLine 9.4.4. ModSecurity v3.0.17 with the Core Rule Set 4.30.0, from the official image owasp/modsecurity-crs:4.30.0-nginx-alpine-202611160811.
  • How GoTestWAF judged. An answer with status 403 counts as blocked. An answer with status 200 or 404 counts as let through.
  • How WAFtester ran. The tool stops sending when many tests in a row are blocked. To get every test sent, we ran it once for each category and severity, 96 runs for each product, and joined the results. Each test is counted once.
  • How blazehttp ran. With its default settings: 10 connections at a time and a limit of one second for each request. It sent every sample file once.
  • Raw data. We keep the output of every tool as it printed it. If you want to check a number or repeat the test against your own setup, write to us through the support page.