Philosophy of Static Code Analysis: We Have 100 Developers, the Analyzer Found Few Bugs, Is Analyzer Useless?

16 Oct 2017

Author: Evgenii Ryzhkov

People often think incorrectly about static code analysis.

It is often possible to meet such an opinion expressed in the following paragraph:

We have a large project. Dozens (hundreds) of developers work on it. This is that very case about which you are writing, that needs static code analysis application. I downloaded the code analyzer, performed a analysis and found very few bugs. Obviously, the code analyzer is simply useless!

Unfortunately, it seems that the way of promoting static code analysis via checking open source projects plays tricks on some people. People EXPECT that having run the analyzer on a random project, they will NECESSARILY find a bunch of errors. Of course this is not true, here is why.

Let us suppose we have a project involving 10 developers, 3 QAs and 1 project manager. We will consider an interval of one month (21 working days). The interval was selected randomly and it is not that important.

Let in the first working day the developers wrote code and introduced 5 errors. The next day, the QAs found these 5 bugs and wrote back to programmers. On the third day the programmers fixed 5 bugs. Here we have formed such a "tuple" or a cohort: one day they make errors, the second day they find them, the third day they correct them.

Obviously, this cohort moves through time, and every day the bugs are added, errors are detected and fixed. Although in every moment of time there are not not so many errors, but this "window" moves and the general number of fulfilled work on detecting and fixing bugs is quite significant. On the last day of the 21-days working cycle, a project manager gets tired, he shouts at everybody, and all errors are fixed. As a result, at the end of the month, there is 0 errors in a project. This, of course, will not last long, but the manager is happy about the nice month totals.

Let us suppose that on this 21st day someone from programmers downloaded the code analyzer, performed the check, got 0 errors (and a couple of stupid false positives) and made the obvious, but wrong conclusion: code analyzers just don't work and aren't needed!

Now we will look at the moving of the cohort if the analyzer is used regularly.

On the first day of the programmers also made 5 errors. But thanks to the automatic code analysis, 3 bugs have been found and were fixed immediately even at the stage of coding. So, the next day, the testers will find just 2 errors. On the third day the programmers will have much less work on edits, because some errors have been fixed on the first day. It turns out that the analyzer has reduced the amount of work on errors fixing more than twice that significantly accelerates the development process as a whole.

Of course, the numbers above are relative. The most important thing is that we can make the right important conclusion. Static analysis is the most useful when it is used not as a one-time activity, but as a regular process.

#StaticAnalysis