Our website uses cookies to enhance your browsing experience.
Accept
to the top
>
OWASP Benchmark Java coverage

OWASP Benchmark Java coverage

This page contains the performance results of PVS-Studio Java analyzer on the open OWASP Benchmark test suite for evaluating SAST tools.

What is OWASP Benchmark

OWASP Benchmark is a test suite that evaluates how well a static analysis tool detects potential vulnerabilities in Java applications. The benchmark contains a total of 2,740 tests: 1415 tests contain a potential vulnerability classified according to CWE, and 1325 contain safe code. Each test is a small part of a web application.

Tests containing safe code are necessary to ensure that scores account for both potential vulnerabilities and false positives. Otherwise, a tool that issues warnings for everything would produce good results. According to the OWASP Benchmark, however, such a tool would score 0%.

How the score is calculated

The benchmark repository includes a list of expected results that specifies whether each test contains a potential vulnerability. The analyzer warnings are checked against this list, and the results fall into one of four groups:

  • TP (True Positive) means that a potential vulnerability exists, and the analyzer detected it;
  • FN (False Negative) means that a potential vulnerability exists, but the analyzer ignored it;
  • FP (False Positive) means that the code contains no potential vulnerability, but the analyzer issued a warning;
  • TN (True Negative) means that the code contains no potential vulnerability, and the analyzer did not issue a warning.

They produce two metrics:

  • TPR (True Positive Rate, TP / (TP + FN)) is how many real potential vulnerabilities the analyzer detected.
  • FPR (False Positive Rate, FP / (FP + TN)) is how many tests with safe code the analyzer got wrong.

The final score equals TPR – FPR.

Analyzer evaluation

The following configuration is used to evaluate the analyzer:

The table below shows the metrics from running PVS-Studio Java analyzer on all 2740 benchmark tests:

Metric

Value

TPR

86,00%

FPR

16,53%

Final score

69,47%

The final values are calculated as the arithmetic mean across 11 vulnerability categories.

The results for each category are shown in the following table:

Category

CWE

TP

FN

TN

FP

Total

TPR

FPR

Score

Command Injection

78

92

34

98

27

251

73,02%

21,60%

51,42%

Insecure Cookie

614

36

0

31

0

67

100,00%

0,00%

100,00%

LDAP Injection

90

24

3

23

9

59

88,89%

28,12%

60,76%

Path Traversal

22

116

17

97

38

268

87,22%

28,15%

59,07%

SQL Injection

89

241

31

172

60

504

88,60%

25,86%

62,74%

Trust Boundary

501

51

32

33

10

126

61,45%

23,26%

38,19%

Weak Encryption Algorithm

327

130

0

116

0

246

100,00%

0,00%

100,00%

Weak Hashing Algorithm

328

89

40

107

0

236

68,99%

0,00%

68,99%

Weak Randomness

330

218

0

275

0

493

100,00%

0,00%

100,00%

XPath Injection

643

14

1

12

8

35

93,33%

40,00%

53,33%

XSS (Cross-Site Scripting)

79

208

38

178

31

455

84,55%

14,83%

69,72%

Total

1219

196

1142

183

2740

Overall (average across categories)

86,00%

16,53%

69,47%

In OWASP Benchmark, one detail often distinguishes tests with secure code from those with vulnerable code. For example, user input reaches a dangerous call only in a branch of a condition that is never executed. To avoid issuing a redundant warning for such code, the analyzer must not only see the path the data follows, but also determine whether the data will actually travel along that path. That is why the benchmark result primarily shows how accurately the analyzer tracks data and variable values in a program.

How to reproduce the results

You can get the PVS-Studio results on OWASP Benchmark yourself:

  • Download the BenchmarkJava repository.
  • Download the BenchmarkUtils tool and install it using the mvn install command.
  • Add the PVS-Studio Maven plugin to the BenchmarkJava project. To do this, add it to pom.xml and enable the diagnostic rules from the OWASP group in the settings. Then, run the analysis using the mvn pvsstudio:pvsAnalyze command.
  • Convert the analyzer report to SARIF using plog-converter and put the result in the results directory of the BenchmarkJava repository.
  • Run createScorecards.sh (or createScorecards.bat on Windows). This creates an HTML page listing the overall results sorted by category.