This page contains the performance results of PVS-Studio Java analyzer on the open OWASP Benchmark test suite for evaluating SAST tools.
OWASP Benchmark is a test suite that evaluates how well a static analysis tool detects potential vulnerabilities in Java applications. The benchmark contains a total of 2,740 tests: 1415 tests contain a potential vulnerability classified according to CWE, and 1325 contain safe code. Each test is a small part of a web application.
Tests containing safe code are necessary to ensure that scores account for both potential vulnerabilities and false positives. Otherwise, a tool that issues warnings for everything would produce good results. According to the OWASP Benchmark, however, such a tool would score 0%.
The benchmark repository includes a list of expected results that specifies whether each test contains a potential vulnerability. The analyzer warnings are checked against this list, and the results fall into one of four groups:
They produce two metrics:
TP / (TP + FN)) is how many real potential vulnerabilities the analyzer detected.FP / (FP + TN)) is how many tests with safe code the analyzer got wrong.The final score equals TPR – FPR.
The following configuration is used to evaluate the analyzer:
The table below shows the metrics from running PVS-Studio Java analyzer on all 2740 benchmark tests:
|
Metric |
Value |
|---|---|
|
TPR |
86,00% |
|
FPR |
16,53% |
|
Final score |
69,47% |
The final values are calculated as the arithmetic mean across 11 vulnerability categories.
The results for each category are shown in the following table:
|
Category |
CWE |
TP |
FN |
TN |
FP |
Total |
TPR |
FPR |
Score |
|---|---|---|---|---|---|---|---|---|---|
|
Command Injection |
78 |
92 |
34 |
98 |
27 |
251 |
73,02% |
21,60% |
51,42% |
|
Insecure Cookie |
614 |
36 |
0 |
31 |
0 |
67 |
100,00% |
0,00% |
100,00% |
|
LDAP Injection |
90 |
24 |
3 |
23 |
9 |
59 |
88,89% |
28,12% |
60,76% |
|
Path Traversal |
22 |
116 |
17 |
97 |
38 |
268 |
87,22% |
28,15% |
59,07% |
|
SQL Injection |
89 |
241 |
31 |
172 |
60 |
504 |
88,60% |
25,86% |
62,74% |
|
Trust Boundary |
501 |
51 |
32 |
33 |
10 |
126 |
61,45% |
23,26% |
38,19% |
|
Weak Encryption Algorithm |
327 |
130 |
0 |
116 |
0 |
246 |
100,00% |
0,00% |
100,00% |
|
Weak Hashing Algorithm |
328 |
89 |
40 |
107 |
0 |
236 |
68,99% |
0,00% |
68,99% |
|
Weak Randomness |
330 |
218 |
0 |
275 |
0 |
493 |
100,00% |
0,00% |
100,00% |
|
XPath Injection |
643 |
14 |
1 |
12 |
8 |
35 |
93,33% |
40,00% |
53,33% |
|
XSS (Cross-Site Scripting) |
79 |
208 |
38 |
178 |
31 |
455 |
84,55% |
14,83% |
69,72% |
|
Total |
1219 |
196 |
1142 |
183 |
2740 |
||||
|
Overall (average across categories) |
86,00% |
16,53% |
69,47% |
In OWASP Benchmark, one detail often distinguishes tests with secure code from those with vulnerable code. For example, user input reaches a dangerous call only in a branch of a condition that is never executed. To avoid issuing a redundant warning for such code, the analyzer must not only see the path the data follows, but also determine whether the data will actually travel along that path. That is why the benchmark result primarily shows how accurately the analyzer tracks data and variable values in a program.
You can get the PVS-Studio results on OWASP Benchmark yourself:
mvn install command.pom.xml and enable the diagnostic rules from the OWASP group in the settings. Then, run the analysis using the mvn pvsstudio:pvsAnalyze command.plog-converter and put the result in the results directory of the BenchmarkJava repository.createScorecards.sh (or createScorecards.bat on Windows). This creates an HTML page listing the overall results sorted by category.