A Scoop from PVS-Studio: "An Ideal Project with Zero Bugs Found!"

Apr 30 2014

Author: Evgenii Ryzhkov

We were recently contacted by a user who was claiming that our tool had detected zero bugs in his pretty heavy project (2000 files)! Nor general errors, nor 64-bit issues, nor any other defects. Since things are never like this in our world, we sent him just a few e-mails to find out some details and very soon were working on his computer remotely trying to figure the whole thing out ourselves. All the settings were right; no silly evident faults of the analyzer's operation – all was fine. After a few test launches, we grew totally convinced that our analyzer wasn't actually generating a single warning for the project! Want to know why?

You see, the PVS-Studio analyzer provides the user with a number of ways to hide irrelevant diagnostic messages. You can turn them off inside the code; you can hide all messages relating to certain files by message name or text; you can hide messages for files in a certain folder. The very first idea to occur to anyone is that the guy added the entire project folder into the exceptions list. Doing so is possible, but we checked this version first of all, and it proved false.

However, the analyzer was behaving just as if the whole project had been excluded from analysis. We went on to examine intermediate configuration files (file.PVS-Studio.cfg) which are created before launching the analyzer for each file. Among other data, they include information about the folders excluded from analysis. This part of an intermediate file usually looks like this:

exclude-path = *\boost\*
exclude-path = *\zlib\*
...
exclude-path = c:\program files (x86)\microsoft visual studio 11.0
exclude-path = c:\program files (x86)\microsoft sdks
...

Some of the folders are system folders, others are standard libraries, and sometimes there may also be user folders.

But this time it was the entire project folder among other exceptions, although no one seemed to have added it there. Because of that the analyzer didn't show a single warning for the code.

So, what happened?

In our analyzer, in order to exclude system libraries from analysis, we have the paths from Additional Includes automatically added into the exceptions list, these paths starting with "C:\Program Files" and "C:\Program Files (x86)".

Additional Includes may include paths defined in relation to the folder the compiler is launched from (which is usually the folder of the project file). For example, you can legally add a path like "../Common" into the Additional Includes list.

Before checking the paths for being nested into "Program Files", we used to normalize them with the help of the DirectoryInfo class:

String NormIncludePath = 
  new DirectoryInfo(list[i].Trim('"')).FullName.ToLower();

However, the DirectoryInfo class would at the same time restore relative paths in relation to the current process folder, which is obviously incorrect. Now imagine the following situation. The IDE's devenv.exe process is launched from its standard folder – say, we simply open the IDE through a desktop shortcut. If the project contains relative paths in Additional Includes, these paths are filtered off as if they were system folders, for the standard folder of devenv.exe is c:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\. Therefore these paths get into exceptions.

The bug was easy to fix, we just needed to expand the relative paths correctly:

String NormIncludePath = list[i].Trim('"');
ExpandRelativePath(
  ref NormIncludePath, CompilerWorkingDirectory);
NormIncludePath = NormIncludePath.ToLower();

To sum it all up, what impact could that bug in our analyzers have on your code? If you use relative paths in Additional Includes in your project and launch the IDE in a certain way (through the desktop Visual Studio shortcut), you might have missed some of the messages from the analyzer. This bug was both in PVS-Studio. The fresh versions of this product don't have it anymore.

P.S. Unfortunately, there hardly exists an absolutely perfect project our analyzer cannot find bugs in. Yet we keep seeking.