To get a trial key
fill out the form below
Team License (standard version)
Enterprise License (extended version)
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Request our prices
New License
License Renewal
--Select currency--
USD
EUR
GBP
RUB
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
To get the licence for your open-source project, please fill out this form
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
To get the licence for your open-source project, please fill out this form
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
I am interested to try it on the platforms:
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Message submitted.

Your message has been sent. We will email you at


If you haven't received our response, please do the following:
check your Spam/Junk folder and click the "Not Spam" button for our message.
This way, you won't miss messages from our team in the future.

>
>
Viewing Bug Statistics, or "Look, …

Viewing Bug Statistics, or "Look, They've Got Charts in PVS-Studio!"

Jul 30 2015

This post will be first of all of utmost interest to project and technical managers whose teams use the PVS-Studio code analyzer in their projects. We have recently implemented a new feature to let you monitor the efficiency of the team tool use. You can finally have the necessary figures to prove to your boss that the analyzer which cost your team or company a few thousands of dollars is bringing a real, tangible profit. But don't worry – this article is not about ROI at all.

0341_Excel/image1.png

So what's the problem with static analysis tools besides them being pretty costly? Not every team that has purchased an analyzer can boast of really using it. And as for us, we are more than anyone else interested in our analyzer being used as efficiently as possible. We won't be satisfied if a customer just pays for a license and forgets all about the tool. You see, more than half of our customers renew their licenses for another year. So we need to prove to those who consider renewing their licenses that the tool really benefits their project.

That's why, starting with PVS-Studio 5.27, our tool can draw charts of the number of bugs detected in your project. The idea behind this feature is very simple:

  • If you want to fix all the errors reported by PVS-Studio, your chart will gradually drop to zero.
  • If you are ready to put up with old bugs but get rid of all the new ones, your chart shouldn't go too far up.
  • If you have any other chart, it means your developers simply don't use PVS-Studio – to both our regret (you won't renew the license) and yours (you've spent money on the license to no purpose).

Before studying any particular types of charts, let me first say a few words about the mechanism behind this new feature. For more details, please refer to the documentation.

Every time you analyze your project in full (the Check Solution command), the analyzer creates a small XML file with the information about the number of messages in the %AppData%\Roaming\PVS-Studio\Statistics folder. In PVS-Studio menu, you will find the Analysis Statistics command to open the statistics viewing settings dialog. In this dialog, you can select the desired time interval, rule sets, and message severity levels. After that, click on the "Show in Excel" button to open Microsoft Excel installed on your computer and view the resulting chart.

There's no point speaking about this dialog in detail as everything is said in the documentation, so let's rather examine the examples of the possible chart types.

What does a chart look like with all the bugs (both old and new) fixed?

Here's the first example. Imagine your company has purchased PVS-Studio and set about integrating it into the project. Suppose we are interested in General Analysis errors of the 1-st and 2-nd severity levels only (for a start). After the first analysis, you've got about 1800 messages. Your programmers will first of all eliminate any apparent false positives – by suppressing them in macros, for example. Then they'll go on to fix the simplest bugs that don't take much time. Then the work will slow down a bit. That is why our chart looks like a curved line, not a straight one.

Errors are getting fewer and fewer until we get total 0 at some point. The chart type below corresponds to this bug fixing mode in PVS-Studio:

0341_Excel/image2.png

Figure 1 – Simple errors get fixed in the first place, then more complex ones, and the most complex bugs are fixed in the end.

OK, we've got all the messages gone at last and have 0 warnings in the last report. Can we relax now? No, you can't. Because if you do and turn off the analyzer (stop fixing new bugs), the number of errors will start growing all over again. Figure 2 shows what the chart looks like in this case.

0341_Excel/image3.png

Figure 2 – Having fixed all the errors, developers relaxed and stopped fixing new analyzer messages.

Do you see the rise of the message number in this chart? It is because the developers stopped fixing bugs in new code after they had achieved 0 warnings for old code. Fortunately, the project manager noticed it in time and gave his team a talking-to, after which they eliminated all the bugs again and then kept their number at zero further on.

An example of how you may draw incorrect conclusions about the team's work results relying on the chart

Well, what about the chart in Figure 3? What conclusions does it suggest?

0341_Excel/image4.png

Figure 3 – An example of an ambiguous chart.

At first glance, you may think the team has given in to laziness and stopped fixing anything, which results in the bug number growth – and quite a fast one, from 2700 up to 3700. However, only a person completely unfamiliar with the project can think so. If you know what the developers actually do, then you'll get it right. You see, they have added big chunks of new code into the project (new project files in the general .sln file) lately. There's nothing to worry about in this chart. When the number of projects is settled, the amount of errors will stop growing that much if the team starts fixing them in time.

This chart is an example to warn you against "head-on" interpretation of statistics data. To draw correct conclusions, you also need to fully understand what your team is doing at any given time.

What are those "steps"?

A perfect chart for evenly timed work would often resemble a staircase with steps as in Figure 4.

0341_Excel/image5.png

Figure 4 – A stepped chart.

A stepped chart like this is the best one, for it means that the bug fixing process is even and steady. Have you guessed what those "steps" mean, where the number of errors remains unchanged? They're weekends, of course.

What does a chart for "new bugs only" look like?

OK, until now we were discussing examples when a team fixes every bug found by the analyzer. But we also have the "mass message suppression" feature in our tool whose purpose is to let developers ignore tons of messages generated on old code they are aware of but not willing to fix right now, and focus on new bugs only instead. What will the bug chart look like in this case? Well, take a look at Figure 5.

0341_Excel/image6.png

Figure 5 – A bug number chart for new code only.

Though it looks somewhat chaotic at first glance, it is exactly what a normal code analyzer driven workflow should look like. What do I mean? Look at the Y axis scale. The largest value reached by the chart is 8 – it is the maximum amount of errors (diagnostic triggerings) for one day. You think it's too much? But considering that we deal with a team of 50 developers and project of 9 million code lines, that's quite a nice result, don't you agree? But what I like most of all is that these bugs will get fixed not later than the next day.

What exactly do I do to get an "ideal chart" in PVS-Studio?

Now we have figured out what both "incorrect" and "correct" charts look like. But how should you arrange the entire development process to achieve the "correct" bug chart? What should you do as a project or technical manager?

The following are the steps that will ensure the best result:

  • Set up daily analysis of your project by PVS-Studio on the build server. It will help you monitor and manage the bug fixing process and collect the work history that can be easily visualized later.
  • Install PVS-Studio on every team member's computer and turn on the incremental analysis mode: in this mode, PVS-Studio keeps tracking which files have been modified on a particular computer and automatically analyzes them once they are successfully compiled. Should it detect any bugs, it will inform you right away. The advantage of this mode is that the developer doesn't need to manually run the analysis. Once they've got a message, they can fix the code at once. In this case, bugs will never even get to the version control system.
  • Even if they do, the night analysis will reveal them for sure. It's very important that developers see the night analysis report in the morning so that they can fix any errors right away. PVS-Studio automatically creates an .html log file that can be sent to everyone involved in the development.

If the team succeeds in adopting this work scenario, the purchase and integration of the analyzer won't be in vain. And the question of renewing the license for another year will be easily resolved, too.

Popular related articles
Technologies used in the PVS-Studio code analyzer for finding bugs and potential vulnerabilities

Date: Nov 21 2018

Author: Andrey Karpov

A brief description of technologies used in the PVS-Studio tool, which let us effectively detect a large number of error patterns and potential vulnerabilities. The article describes the implementati…
Free PVS-Studio for those who develops open source projects

Date: Dec 22 2018

Author: Andrey Karpov

On the New 2019 year's eve, a PVS-Studio team decided to make a nice gift for all contributors of open-source projects hosted on GitHub, GitLab or Bitbucket. They are given free usage of PVS-Studio s…
Static analysis as part of the development process in Unreal Engine

Date: Jun 27 2017

Author: Andrey Karpov

Unreal Engine continues to develop as new code is added and previously written code is changed. What is the inevitable consequence of ongoing development in a project? The emergence of new bugs in th…
PVS-Studio for Java

Date: Jan 17 2019

Author: Andrey Karpov

In the seventh version of the PVS-Studio static analyzer, we added support of the Java language. It's time for a brief story of how we've started making support of the Java language, how far we've co…
How PVS-Studio Proved to Be More Attentive Than Three and a Half Programmers

Date: Oct 22 2018

Author: Andrey Karpov

Just like other static analyzers, PVS-Studio often produces false positives. What you are about to read is a short story where I'll tell you how PVS-Studio proved, just one more time, to be more atte…
The Ultimate Question of Programming, Refactoring, and Everything

Date: Apr 14 2016

Author: Andrey Karpov

Yes, you've guessed correctly - the answer is "42". In this article you will find 42 recommendations about coding in C++ that can help a programmer avoid a lot of errors, save time and effort. The au…
The way static analyzers fight against false positives, and why they do it

Date: Mar 20 2017

Author: Andrey Karpov

In my previous article I wrote that I don't like the approach of evaluating the efficiency of static analyzers with the help of synthetic tests. In that article, I give the example of a code fragment…
Appreciate Static Code Analysis!

Date: Oct 16 2017

Author: Andrey Karpov

I am really astonished by the capabilities of static code analysis even though I am one of the developers of PVS-Studio analyzer myself. The tool surprised me the other day as it turned out to be sma…
The Last Line Effect

Date: May 31 2014

Author: Andrey Karpov

I have studied many errors caused by the use of the Copy-Paste method, and can assure you that programmers most often tend to make mistakes in the last fragment of a homogeneous code block. I have ne…
Characteristics of PVS-Studio Analyzer by the Example of EFL Core Libraries, 10-15% of False Positives

Date: Jul 31 2017

Author: Andrey Karpov

After I wrote quite a big article about the analysis of the Tizen OS code, I received a large number of questions concerning the percentage of false positives and the density of errors (how many erro…

Comments (0)

Next comments

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
This website uses cookies and other technology to provide you a more personalized experience. By continuing the view of our web-pages you accept the terms of using these files. If you don't want your personal data to be processed, please, leave this site.
Learn More →
Accept