How we have solved an engineering task for several years in PVS-Studio

Aug 03 2011

Author: Evgenii Ryzhkov

At first I wanted to title this post "How PVS-Studio enables cheap integration of static code analysis into the development process" but I decided not to do it because of the ambiguous interpretation of the word "cheap". So I will tell you about one engineering problem we had to solve constantly to enable people to use our product. Going a bit ahead I want so say that we seem to have solved it.

So, having developed the first full version of our static code analyzer (that was called Viva64 at the time and served to detect 64-bit errors) back in 2007, we faced an issue of tool's integration by our users. Our clients are companies that have at least several tens of developers and at least several hundred thousand code lines. Any static analyzer will generate plenty of warnings on a code of that size. For instance, our tool generated up to several thousand messages on one project.

Yes, of course, the trouble here is with false reports of analyzers. But any analyzer has false reports and you can't help it. A question rose what users were to do with a lot of messages they got? That is, the problem looked as follows: a potential user downloads a program (trial-version), launches it and gets ten thousand messages. It certainly makes him sad, he uninstalls the program - and one more client is lost for us.

The first thing we did is removing double messages at once. The analyzer checks C/C++ projects and sometimes it happens that an error in an .h-file is reported during the check of several .cpp-files that use it. We don't have this doubling. Then we added capabilities of analysis result filtering (and then we constantly improved them): filtering by error code, by message text, a capability not to check files by masks and so on. All this allowed us to significantly reduce the number of messages but only after customization. At the first use, a person would get a pile of messages all the same. So, message filtering is an important tool but it did not solve the initial problem - difficulty of integrating the tool into the development process.

Then the analyzer acquired the new mechanism "Mark as False Alarm". Its principle is adding comments of a special kind (//-V112) into the code to suppress the analyzer-generated messages. Having marked the code in such a way, in future you will get error-reports only for those code fragments that do not have this marking. Ideally it will be only newly added code. Although the problem of integrating the analyzer into the team development process became a bit simpler, still it required that several members of the team marked the code first to remove rubbish messages.

The next step towards solution of the integration problem was the capability to check only files modified during the last several days. It was much closer to the idea of starting to get profit from static analysis at once. But the problem remained. A user doesn't know about this feature and if we enable it by default it won't be clear why so few files are checked. But I repeat this again - the direction seemed right to us.

So we went on and made a new super feature "Incremental Analysis after Build". The analyzer now launches right after the compilation and checks only those files which have been affected by user editing. Unlike checking files for the last several days (when editing of the developer team was could be checked), the user now sees errors ONLY in the code he directly handles.

Programmers now won't worry about large sizes of code they do not deal with. Perhaps this code is older than 5 years; it virtually doesn't endure any modifications and most defects in it have been fixed. You don't need to rush and check this code first of all, and the analyzer doesn't. The programmer will see warnings only in the fresh code. And if he has some time left, he may check the project in full and peep into the most rarely visited places.

Yes, the analyzer still produces false reports. Yes, filters haven't become less important. But there is another thing that counts. We have managed to reduce the cost of INTEGRATION (costs of people's efforts on startup) of the static analyzer to zero. That is, a person now downloads the static analyzer, installs it and IMMEDIATELY starts getting profit from it without any additional efforts.

But there remained one last thing to complete the task. All was good, but it was difficult to notice it when the analyzer found errors. Color change of the PVS-Studio window's icon (as we made it in the beginning) is not so visible while in Visual Studio 2005 it doesn't work at all. The solution was to make a pop-up message.

Surely, we all don't like all those importunate pop-up notes. But in this case it will be useful to programmers for sure and it will appear rarely on condition that they do not make a lot of errors in code.

So, the engineering task of integrating static analysis into the development process has been solved. That is why the Incremental Analysis after Build mode will be enabled by default in Incremental Analysis after Build.

The conclusion is simple. Now developers should not get afraid of difficulties of static code analysis integration because they can simply download PVS-Studio, install it and study errors that will be detected in newly developed code.