Earlier we wrote about our quiz for C++ developers. Since the quiz launch, we have been carefully collecting your feedback. Some of the comments were related to the quiz errors, that we obviously sought to fix.
For those who don't know yet what quiz we are talking about, let us briefly explain. We made a quiz so you could challenge yourself in code review. For this quiz, we've picked errors detected by our analyzer in Open Source projects. You'll see 10 random code fragments, each containing an error. In 60 seconds, you need to spot the bug and click on it.
You can read more about our quiz in the note, or finally try to spot the error!
But let's get back to the topic. We received a lot of comments from people who took our quiz. And some of the replies pointed out problems with the mechanics of the quiz. We, the advocates of code quality, always strive to enhance our own projects as well.
That's why we dug into the comments everywhere we posted our quiz. In social networks, on Habr, at thematic forums and websites. As a result, we identified two issues to be fixed. The first one got a lot of comments — people complained about the way code errors are selected in the quiz and the way answers taken as correct/incorrect. The second issue was not as common, since we got a single comment on it. And, ironically, we have knowingly made this issue possible. But first things first. Let's discuss.
Our quiz originally allowed to select multiple code elements for an answer. In the code fragment below, the error is clip->GetSequence.
Previously, you could select both the correct answer, and (either deliberately or accidentally) select the symbols = or (), and even the entire line as well. As a result, you could find an actual error, but an answer can be regarded as incorrect due to extra symbols being selected.
Now you can select no more than one element in the code fragment. But that doesn't make the selected element to be the only correct. At the same time, a correct answer sometimes includes several elements of code to be selected, that's why an answer would be correct if you select any of them.
You can easily see the difference in the videos attached:
We recently summed up the results of our quiz for two months and told you that the average score of correct answers among nearly 2,000 developers is 3.5 out of 10. But we can't deny that the way errors were counted may have affected the final results of the quiz.
It's always embarrassing to admit your own mistakes, but we have to. Let's go back to the time when the quiz was created. At first, the quiz contained a bug: as soon as the question appeared on the screen, you got the "time's up" message. This was the reason:
In other words:
the task is opened -> the timer resets to zero -> the check for === 0 is done and the "time is up" message is issued.
Instead, it was supposed to work this way:
the task is opened -> the timer resets to zero -> the timer starts -> after 60 seconds the check for === 0 is done and the "time is up" message is issued.
So as not to waste time, we decided to set the timer from 60 to 1. After that, the time reset to zero, and the "time is up" message was issued when checking "if the timer === 1".
The solution was not elegant, but obviously fast. And we were sure that no one would probably cosplay 90s action movie heroes, pressing the "Answer" button at the last second. But, if developers believe that everything would work exactly the way they made it, there is bound to be someone who "stumbles on a kludge".
For 2 months we safely forgot about this mistake. Until we saw a screenshot in one of the comments:
It turned out that even though we promised the quiz participants 60 seconds for each answer, we actually gave them only 59 seconds. Well, we pleaded guilty and corrected the mistake. So, now you can safely press the "Answer" button at the last second. But it always better not to waste time, particularly when you have already found an answer.
This case has proven the essential role of feedback in any project.
In fact, we covered all these issues in advance while making the C# quiz. You can already challenge yourself in it. Try it yourself, share the quiz with your C# teammates, and of course, share your own results with us! And if you suddenly notice a bug, don't hesitate to write us – we will fix it.
Also, don't forget that those who passed the quiz get a promo code for a 30-day PVS-Studio license.