One of the most frequently asked questions we get from the readers of our articles is, "Do you use your analyzer to check its own code?" We usually answer that we have a practice of checking our code right in the course of writing it with the help of incremental analysis (it is a mode when individual files are analyzed right after compilation). Besides, we regularly run night checks of the whole code. Because of that, we, "unfortunately", will never get a chance to write an article about bugs found in our own software products.
Unfortunately, we are no longer developing or supporting the CppCat static code analyzer. Please read here for details.
In today's article, however, I'm going to tell you about one bug we found in our product and how much it cost us to fix it. Got interested?
To give this story a certain dramatic touch, let me introduce the main characters first:
One Friday, February 21-st, we published the article "A Spin-off: Firebird Checked by PVS-Studio" which was then commented by a number of readers among whom Mikhail showed up with a comment of his:
CppCat is just the thing every developer needs! You should write a separate article about it. I'm going to convince my boss to buy it - it's quite affordable, after all :)
My colleague Andrey smiled and replied that we had already published three articles about CppCat. However, he got a telling-off right away for our new product still being unknown to a wide audience. It happened at 16:16 Moscow time...
As soon as at 16:21, I got an e-mail from Mikhail with a "wish" concerning CppCat. In particular, he wanted CppCat's diagnostic messages to be displayed in the standard Visual Studio error list window instead of a custom CppCat one as it is now. As "the product keeper", I found it to be unwise and wrote a response e-mail to Mikhail trying to explain that we had deliberately set up the tool that way so that the user would not confuse analyzer-generated messages and Visual Studio's messages; that it is very convenient since it allows us to implement a specialized interface for our own window, and so on. But Mikhail just wouldn't give in. His argument was a simple one:
"And how do I know about the bugs found after file recompilation? Wouldn't it be better to show them in the window right away?.."
Well, but we have a killer feature for this particular purpose - incremental analysis! Once files are recompiled, the analyzer starts checking them automatically. Should any issues be found, you will be informed by a tray pop-up message and a caption bar of the analyzer's window will change accordingly. A message is displayed in the window to tell you that new bugs were found. You can enable or disable the incremental analysis mode in the menu.
"Right," Mikhail said. "I have the Incremental analysis option enabled. But how do I know if issues were found?"
After five e-mails with attached screenshots from Mikhail, it finally got to me that despite everything being set up right, the analyzer did not show the message in the tray, nor it changed the text of the window's caption bar. The time was already past five p.m. of a Friday evening...
With the thought "I haven't touched checkers in a long time... well, since the day before yesterday!" (an allusion to a phrase from the book "Dead Souls" by the Russian writer Nikolai Gogol - a comment by translator) on my mind, I asked Mikhail to let me into his system remotely through the AMMYY Admin client. After poking my program for some time, I got totally convinced that it was failing to show the message about new messages being found, though it seemed to be working correctly. The time was about six p.m., and I visited my colleague Pavel responsible for the analyzer's integration with Visual Studio. I explained the problem to him and described my exact actions on the user's computer.
Thirty seconds later Pavel said:
"Evgeniy, how could you... How could you forget that we deliberately made it so that incremental analysis won't start after running the command "Rebuild All"? We did it consciously and that's alright. You should spend more time with your product to know it better. But you would rather let yourself rest answering e-mails and writing articles about business..."
Ashamed, I went back to check it all anew... However, I found that it didn't matter whether one or several files were recompiled - no notification was shown anyway. I tried it on another project, and the result was the same - it just wouldn't work. The worst thing was that it was a special, very important function, a killer feature of our tool, that didn't work. As you remember, I was working remotely on the user's computer. I visited Pavel again, but it was almost six p.m., and both the user and Pavel wanted to go home... Perhaps this fact prevented us from figuring out together why the tool was not working. Finally, we decided to start fresh on Monday with building a version of the analyzer with included debugging logging, and went home.
I was thinking over this problem all weekend long. Was it our tool to blame? If yes, how could we have let such a bug into the code? And how many users HAD NOT BOUGHT the product because of a very important option got broken?
In this sad mood, Monday passed. We built a special version of the tool with debugging logging for Mikhail. He sent us the logs. However, they didn't gave us any clue. We've increased the logging detail level and worked out a thorough guide on reproducing the bug... But all was in vain. Logs suggested that our code wasn't working, though it just could not be true.
During Monday and Tuesday, we sent Mikhail a few versions with highly detailed logging. But we still got no idea of what was happening. The notification about new messages just would not be displayed - on his computer, actually. On ours, everything was just fine all the time.
On Tuesday, in the afternoon, Pavel called me:
"You won't believe that, Evgeniy... but I don't get new message notifications for CppCat anymore!"
You think it were the debugging logs that helped us to figure it all out? No. It was just that Pavel ACCIDENTALLY clicked on the auto-hide button in CppCat's window inside Visual Studio, and it caused our program's window to be hidden automatically when moving the cursor away from it. Since none of our team uses the auto-hide button, it just could have never occurred to us that our notification system wouldn't work in this mode!
That is, if auto-hide is disabled (the default mode), notification messages are displayed; if auto-hide is enabled, they are not.
I asked Mikhail to check our guess, and he confirmed that we were right.
Once the problem was located, the rest was a pie. We found that the plugin's code was incorrectly processing a certain window style. I asked Pavel to write a detailed description of the problem and its solution for programmers, but I won't cite this text here as it will only make sense for Visual Studio plugin developers.
So we managed to reproduce the bug on Tuesday, in the afternoon. On Wednesday, we sent a fixed beta-version to Mikhail.
While Pavel was writing the patch, I corresponded with Mikhail concerning the licensing policy of our products CppCat and PVS-Studio. Although Mikhail was initially interested in CppCat, it turned out later that his team might also need PVS-Studio in their work.
A brief note for those readers not familiar with us and our tools. We develop, promote and sell two software products: CppCat for single developers at $250 and PVS-Studio with its cheapest license for a team of up to 9 developers being offered at €5000.
So, after we finished discussing the licensing matters, Mikhail informed his boss, and later on the same day, Tuesday, I was discussing the PVS-Studio licensing policy with him. It didn't take us long, and he finally decided that they needed PVS-Studio rather than CppCat.
OK, but what about that bug? On Wednesday, we sent Mikhail a fixed beta-version; he checked it and said that all was working well. Later on the same day, I sent an invoice for payment for PVS-Studio to Mikhail's boss. On Thursday, I got a payment receipt. And then I thought I should tell you this story -some may find it interesting.
This is a nice story about how fixing one bug in a product costing $250 let us sell another product at a much higher price. And all that took us just one week, for today (February 28th) you are already reading this story!
If you ask me why I have written this article, I'll give you a simple answer. I am proud of my team, proud that we can fix bugs so quickly and that our customers appreciate it.
P.S. Many thanks to Mikhail who helped us to diagnose that error - this story would just have been impossible but for him. Our users, programmers, are the best in the world!
P.P.S. And of course, Mikhail's firm got a discount for the purchase.