We have been heavily participating in various IT conferences both in Russia and abroad lately. We try to have our own booth at most events and give talks sometimes. Participating in conferences is a good way for us not only to promote our product (PVS-Studio static code analyzer) but also - and most importantly - get to know our potential and current users better. When talking to guests at our booth, our team have to answer the same questions, some of which are pretty funny. In this article, I'll try to answer the most frequently asked questions about static analysis in general and PVS-Studio in particular.
Let's start with a funny "expectations-VS-reality" story. We write quite a lot of articles and naively believe that since people read them, they definitely know - if only in general terms - about our product and its distinctive features. But, sadly, it's not always so. As an example, here's a real-life conversation we had with a guest at one of the recent event. He said, "We read your articles on Habr. They're cool, but it's a pity we can't use your analyzer. You see, we have to keep all our source files secret and can't let them out, while you have all your gear working through the cloud."
I guess our marketing department has some work to do.
Other questions typically asked by an average visitor at our exhibition booth are given below along with the answers, arranged as an improvised dialog.
Q: I think I heard something about you. Where are you from, what do you do, and where can I learn more about you?
A: We are a small independent developer team from Tula. We develop PVS-Studio, a static code analyzer for C, C++, C#, and Java. You can find out more about us on the PVS-Studio website. We also write articles and post them on our blog, and some other websites.
Q: What's static analysis? Our team does pretty well with only compiler warnings and peer reviews.
A: The compiler can detect only the most blatant and fairly obvious errors such as unreachable code or use of uninitialized variables. Detecting errors is not the compiler's primary task. Compilers have indeed become pretty smart today, but any specialized bug-searching tool is way better at this than compilers are. These tools are designed to carry out thorough code analysis and catch typos, logical errors, potential use of null pointers/references, and so on.
Peer reviews are a good, time-proven practice, but it also has its own disadvantages implied by having to engage other developers in working on someone else's code. This makes the development more expensive, while there's still a risk of overlooking a bug or adding a new one since it's natural for humans to make mistakes.
Static analysis is the checking of a program's source code without executing it. We recommend using it as an additional barrier in bugs' way.
Q: Does PVS-Studio check executables or the source code?
A: If we talk about the search of vulnerabilities, analyzing executables is more like what antiviruses do when scanning the binary code for signatures stored in their database.
PVS-Studio analyzes the source code. This enables us to detect a much wider range of errors and potential vulnerabilities, speed up the analysis process, and get more reliable results.
Sure, you could try disassembling the code and tracking down a defect in the algorithm. But you should keep in mind that much of the information gets lost when converted into binary code. For instance, an always-true condition will be simply deleted during compilation, and there's no way you could find out something was wrong with that condition.
Q: What about integration options? Do you have plugins for any IDEs?
A: As of today, PVS-Studio can integrate into Visual Studio 2010-2017 as a plugin. We also have a Java-plugin for IntelliJ IDEA. Besides, thanks to a specialized module, you can manage CMake projects in Qt Creator and CLion.
The analyzer is constantly evolving, so keep track of the new capabilities through our website.
Q: We use SonarQube.
A: Good pick! The PVS-Studio plugin for SonarQube supports import of analysis results into SonarQube to handle them in the usual way.
Q: OK, I have to find a way to incorporate your tool into our build system.
A: PVS-Studio can be used from the command line. The analyzer provides a bunch of settings to address a wide range of tasks. Besides, no matter what use scenario you choose, we help our users get started at the integration stage and provide them with support during the license period.
Q: How can I try your analyzer?
A: Visit the download page, download the suitable distribution, and run PVS-Studio in the trial mode. If you need more time to estimate our product or you are not comfortable with the demo version's limitations, please email us.
One more thing. Our guests at exhibitions can also get a one-month Enterprise key. To get it, simply visit our booth with the unicorn logo. In addition to the license, you can also participate in a lottery and win a prize from the PVS-Studio team.
Q: I'm a student. Can I use PVS-Studio for free?
A: Yes, you can do that if you add special comments to your source code. Files marked with those comments will be checked without any limitations. The free-use mode is discussed in detail in the article "How to use PVS-Studio for Free".
Q: Who are your customers?
A: At present, we have more than 200 companies among our customers all over the world, working in a wide range of areas. A complete list of our current customers is available on our website.
Q: So, can I use your tool locally or not?
A: PVS-Studio is installed locally on a dedicated computer(s) and can run in complete isolation. An Internet connection is needed only to download updates and get to online documentation (diagnostics descriptions, etc.) from the plugins' interface. We are currently thinking over a possible "cloud" mode, but it's going to be just a complement to the standard mode anyway.
Q: What exactly makes your tool better than, say, Coverity?
A: There's no simple and exhaustive answer to this question. All our attempts at comparing our tool with other static analyzers failed. We were accused of bias, artificially incrementing the figures, manipulating the sample database, and other deadly sins. Another problem is that you can't compare analyzers "head-on". Every tool is unique and has its own strong and weak points. Some put emphasis on performance, others focus on code "smells" and style improvements. We look for programming mistakes and potential vulnerabilities.
If you wish, and if you have a set of appropriate methods, you are welcome to do a comparison of our analyzer with others and then publish the results. We would even give you a temporary free unlimited license for that research. But be ready to face criticism.
Q: Wait, you look for defects only in uncompiled code, which hasn't even been checked by the compiler?
A: No, that's not so. PVS-Studio looks for and finds bugs only in compiled programs - the ones that really work. These programs don't simply compile with zero errors - some of them are even regularly checked by other static analysis tools. And we still manage to find bugs in them. We often write articles for our blog, telling our readers about the results of checking various open-source projects.
Q: What types of defects can PVS-Studio detect? Only typos?
A: Of course, typos make a group of defects traditionally caught by static analysis. But apart from typos, PVS-Studio detects several hundreds of error patterns such as null pointer dereferencing, division by zero, always true/false conditions, incorrect index operations, "array index out of bounds", and many more. A complete list of the patterns detected by our tool is available on the documentation page.
Q: Can I be sure there are no bugs left in my program after checking it with your tool?
A: No, you can't. PVS-Studio is not a tool for verifying the correctness of programs - this is the job of another class of tools. Our analyzer's job is to point out a potential bug in code as quickly and reliably as possible. It's always the developer who decides, relying on their knowledge of the context and specifics of the project, if a given construct is flawed or not. The analyzer just helps you by trying to minimize the number of false positives and providing additional mechanisms to handle the warnings.
Q: How exactly does PVS-Studio work? What bug search mechanisms do you use? You must be using regular expressions, right?
A: Regular expressions are an extremely inefficient mechanism. It works only with the most primitive defects like mistakes in conditions (where you have two identical subexpressions compared):
if ((a+b+c) == (a+b+c)) {....}Even a small change to the code (not affecting its logic) would most likely make the analyzer confused:
if ((a+b+c) == (b+a+c)) {....}PVS-Studio is much smarter than that and makes use of the following mechanisms:
All of this was discussed in detail by my colleague Andrey Karpov in a recent article "Technologies used in the PVS-Studio code analyzer for finding bugs and potential vulnerabilities".
Q: OK, we have a C/C++ project of five million lines of code, which took us 15 years to develop. Is there a way we could start using PVS-Studio having gone that far?
A: Yes, there is. You'll have to get started by running a complete check of your project. Once it's over, you can mark all the issued messages (there'll likely be a pile of them) as irrelevant until later (that is, temporarily suppress them) so that you could address this technical debt later. After that, you can continue to use PVS-Studio for routine checks of freshly written code. To learn more about this and other capabilities, please see the documentation page.
Q: How often should I run the checks? Do you mean I have to check the whole code base every time?
A: The most efficient use of static analysis implies checking new code as often as possible. Specifically for that purpose, PVS-Studio provides incremental analysis mode, where it checks only the files that were modified since the last build. There are other modes as well. All of that enables the tool to detect errors as early as at the development stage, which makes it much harder for them to slip into the release version.
Q: We are using PVS-Studio. The analyzer does catch bugs, but many of them sit in unused code or tests. Is it OK?
A: Yes, it's quite normal. One of the specifics of static analysis, unlike dynamic analysis, is that it checks the entire code base, not only currently used code. Suppose you spent a lot of time and effort debugging your code and got things working. But there's one function that you don't use often or don't use at all for a while. This function is very likely to be faulty, and when you start using it someday, something may go wrong. Static analysis can help minimize this risk.
There could also be knowingly incorrect constructs (which is typical of tests), so you may want to exclude them from the check (by specifying the projects or paths to files) through the PVS-Studio's settings tab. However, tests themselves might be faulty. Such cases are difficult to detect, and it's the developer who has to decide if they are correct.
Q: What about the detection of vulnerabilities?
A: PVS-Studio is a SAST (Static Application Security Testing) tool, which means it can detect potential vulnerabilities classified under the CWE (Common Weakness Enumeration). CWE warnings overlap with the classic warnings to a large extent. To learn more about SAST, see the documentation. Keep in mind that potential vulnerabilities don't necessarily mean real ones, exploitable by hackers. As for detected instances of vulnerabilities, these are classified under the CVE (Common Vulnerabilities and Exposures). That said, eliminating potential vulnerabilities definitely makes your program more secure and minimizes the risk of running into real vulnerabilities someday.
Q: I am a team leader. How exactly can I benefit from using PVS-Studio?
A: In addition to improving the quality and reliability of your code, PVS-Studio will also help you manage purely administrative tasks such as separation of concerns. The analyzer comes with a utility called BlameNotifier, which automatically identifies developers who submitted flawed code to the version control system and sends email notifications both to the "guilty" developer and the team leader.
You can also have bug reports converted into any desired format, including detailed html reports, which can be conveniently used by the team leader.
Lastly, if you use SonarQube, you can employ all the handy mechanisms offered by this tool to ensure continuous code quality control by importing PVS-Studio analysis results through a special plugin.
Q: Do you use or plan to use machine learning?
A: This is a vast and interesting topic. We are going to write a critical article about it soon. For now, I'll just make a few short statements.
You don't need machine learning to build a calculator. There is a certain rule (formula) you should apply to the code and draw a conclusion from that. Why set up a neural network to detect a new bug pattern when those patterns are already known and you just need to apply them properly? And - what's most important - where do we get a base for such learning? Where do we get those hundreds of thousands of projects with identified bugs for the neural network to learn on?
The only use of machine learning in our context that we can think of is filtering of false positives.
Q: Do you check the PVS-Studio's code with PVS-Studio?
A: Sure! What's more, whenever a bug is detected, the guilty ones are announced publicly and banned from the freezer with ice cream. Jokes aside, we believe it's very beneficial to be a user of your own product. This helps you look at it from an ordinary user's perspective and notice things that need improving.
Q: How can I get your cool desktop spiral-bound statuses and branded ushanka-hat?
A: Drop by our booth with the unicorn logo at the nearest exhibition. We'll see what we can do about that :)
Hopefully, I managed to answer the most frequently asked questions of our guests. There are some more complicated questions, of course, but they might well take a separate article to answer.
The main idea we try to get across when talking to the guests is this: static analysis is not a cure-all, but it's very good for health (of your programs, I mean). Stick with PVS-Studio and keep well!
To review, here's a list of useful links from the article: