To get a trial key
fill out the form below
Team License (standard version)
Enterprise License (extended version)
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Request our prices
New License
License Renewal
--Select currency--
USD
EUR
GBP
RUB
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
To get the licence for your open-source project, please fill out this form
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
To get the licence for your open-source project, please fill out this form
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
I am interested to try it on the platforms:
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Message submitted.

Your message has been sent. We will email you at


If you haven't received our response, please do the following:
check your Spam/Junk folder and click the "Not Spam" button for our message.
This way, you won't miss messages from our team in the future.

>
>
>
A Note of Caution about Using PVS-Studi…

A Note of Caution about Using PVS-Studio on godbolt.org (Compiler Explorer)

Jun 08 2020
Author:

We have added an option allowing you to experiment with the PVS-Studio static analyzer on the godbolt.org (Compiler Explorer) website. It supports analysis of C and C++ code. We believe this to be an enjoyable and super-easy way to explore results produced by the analyzer when run on different example snippets of code. At the same time, we are concerned that programmers may feel tempted to evaluate the analyzer's abilities based on how it handles synthetic examples without running it on real code. Let's try to figure out why relying on synthetic examples to evaluate tools is a bad approach.

0739_godbolt_and_caution/image1.png

Compiler Explorer is an interactive tool that lets you type code in one window and see the results of its compilation in another window.

PVS-Studio is a tool for detecting bugs and potential vulnerabilities in the source code of programs written in C, C++, C#, and Java.

We have made these two tools friends so that you could run a quick check and see whether PVS-Studio can detect a certain bug in source code. All you need to do is:

  • Go to the godbolt.org website,
  • In the compiler output tab, click the "Add tool ..." button,
  • In the drop-down list, select "PVS-Studio".

Now you can check your code by simply entering it in a window on the site. All versions of GCC and Clang compilers for x86 and x64 platforms are currently supported.

This may be promising from the perspective of satisfying one's curiosity, writing articles, and so on. But there's a downside to it too: rather than using synthetic examples to explore or try out the tool, people may start relying on them to evaluate and compare it against other analyzers. And this is a very bad approach because the results will be unreliable and dependent on how the test examples are written.

Our team has been facing this problem for quite a while. Back in 2017, I wrote an article "Why I Dislike Synthetic Tests". Do read it – it won't take long. In this post, I'd like to share some more examples.

The first thing you may want to find out is whether PVS-Studio can detect a memory leak. I just know from experience that for some reason this test is programmers' favorite :). The test example is very likely to be written as follows:

int main()
{
  int *p = new int;
  *p = 1;
  return 1;
}

Yes, PVS-Studio can detect memory leaks. But it will keep silent on this particular example.

The reason might be unclear to you and there's a risk that you will draw the wrong conclusions. The problem here is that it's particularly common with developers not to free memory in the main function. The tendency is so strong that we decided to make this case an exception. There's actually nothing bad about not freeing memory in main. When it terminates, the memory will be freed anyway.

Sure, technically it's still a bug. But this practice is so common that distracting people with warnings about it wouldn't be a reasonable idea. Actually, I recall some users asking us to turn off warnings triggered by leaks in main.

But change the name of the function – and the warning will appear.

int foo()
{
  int *p = new int;
  *p = 1;
  return 1;
}

On this code, the analyzer will issue the warning as expected: <source>:5:1: error: V773 The function was exited without releasing the 'p' pointer. A memory leak is possible.

Here's another example with the main function. Let's make an infinite loop in it:

int main()
{
  unsigned i = 0;
  while (1)
  {
      i++;
  }
  return 1;
}

The analyzer keeps silent. Doesn't it see this suspicious infinite loop? Yes, it does. It's just that this pattern is very common too. It's typically found in the code of microcontroller firmware. Almost all such programs use infinite loops doing something. There's no reason for the main function to exit in such cases and it never does. There's actually nowhere to exit to :).

Again, just change the name of the function, and the analyzer will issue the expected warning.

int foo()
{
  unsigned i = 0;
  while (1)
  {
      i++;
  }
  return 1;
}

Here, it says: <source>:4:1: note: V776 Potentially infinite loop.

Any static analyzer is about maintaining a fragile balance between an infinite number of nearly useless warnings and the risk of failing to inform the developer about a real bug. Now it should be clear why code analyzers are so sensitive to the way test examples are written. You must be particularly careful when making synthetic examples. And twice that when comparing static analysis tools based on such small tests. This is just not the right thing to do.

If you want to have a better understanding of our philosophy behind the choice of diagnostics to implement and false positives to eliminate, see these two articles:

The best way to try the analyzer and see what it's worth is to run it on your own real-life project. Simply download and run PVS-Studio. By the way, open-source developers can get a free license.

Now that we have warned you, feel free to go ahead and play with the analyzer. Just be careful about the conclusions you draw.

Here are two links to make it easier for you and save you time looking for a button that starts a check. Just modify the code as you like and watch the results:

Thanks for reading!

Popular related articles
The Evil within the Comparison Functions

Date: May 19 2017

Author: Andrey Karpov

Perhaps, readers remember my article titled "Last line effect". It describes a pattern I've once noticed: in most cases programmers make an error in the last line of similar text blocks. Now I want t…
PVS-Studio for Java

Date: Jan 17 2019

Author: Andrey Karpov

In the seventh version of the PVS-Studio static analyzer, we added support of the Java language. It's time for a brief story of how we've started making support of the Java language, how far we've co…
Characteristics of PVS-Studio Analyzer by the Example of EFL Core Libraries, 10-15% of False Positives

Date: Jul 31 2017

Author: Andrey Karpov

After I wrote quite a big article about the analysis of the Tizen OS code, I received a large number of questions concerning the percentage of false positives and the density of errors (how many erro…
The Ultimate Question of Programming, Refactoring, and Everything

Date: Apr 14 2016

Author: Andrey Karpov

Yes, you've guessed correctly - the answer is "42". In this article you will find 42 recommendations about coding in C++ that can help a programmer avoid a lot of errors, save time and effort. The au…
Technologies used in the PVS-Studio code analyzer for finding bugs and potential vulnerabilities

Date: Nov 21 2018

Author: Andrey Karpov

A brief description of technologies used in the PVS-Studio tool, which let us effectively detect a large number of error patterns and potential vulnerabilities. The article describes the implementati…
Appreciate Static Code Analysis!

Date: Oct 16 2017

Author: Andrey Karpov

I am really astonished by the capabilities of static code analysis even though I am one of the developers of PVS-Studio analyzer myself. The tool surprised me the other day as it turned out to be sma…
Static analysis as part of the development process in Unreal Engine

Date: Jun 27 2017

Author: Andrey Karpov

Unreal Engine continues to develop as new code is added and previously written code is changed. What is the inevitable consequence of ongoing development in a project? The emergence of new bugs in th…
The way static analyzers fight against false positives, and why they do it

Date: Mar 20 2017

Author: Andrey Karpov

In my previous article I wrote that I don't like the approach of evaluating the efficiency of static analyzers with the help of synthetic tests. In that article, I give the example of a code fragment…
Free PVS-Studio for those who develops open source projects

Date: Dec 22 2018

Author: Andrey Karpov

On the New 2019 year's eve, a PVS-Studio team decided to make a nice gift for all contributors of open-source projects hosted on GitHub, GitLab or Bitbucket. They are given free usage of PVS-Studio s…
How PVS-Studio Proved to Be More Attentive Than Three and a Half Programmers

Date: Oct 22 2018

Author: Andrey Karpov

Just like other static analyzers, PVS-Studio often produces false positives. What you are about to read is a short story where I'll tell you how PVS-Studio proved, just one more time, to be more atte…

Comments (0)

Next comments

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
This website uses cookies and other technology to provide you a more personalized experience. By continuing the view of our web-pages you accept the terms of using these files. If you don't want your personal data to be processed, please, leave this site.
Learn More →
Accept