To get a trial key
fill out the form below
Team License (a basic version)
Enterprise License (an extended version)
* By clicking this button you agree to our Privacy Policy statement

Request our prices
New License
License Renewal
--Select currency--
USD
EUR
GBP
RUB
* By clicking this button you agree to our Privacy Policy statement

Free PVS-Studio license for Microsoft MVP specialists
* By clicking this button you agree to our Privacy Policy statement

To get the licence for your open-source project, please fill out this form
* By clicking this button you agree to our Privacy Policy statement

I am interested to try it on the platforms:
* By clicking this button you agree to our Privacy Policy statement

Message submitted.

Your message has been sent. We will email you at


If you haven't received our response, please do the following:
check your Spam/Junk folder and click the "Not Spam" button for our message.
This way, you won't miss messages from our team in the future.

>
>
>
Complementing Unit Testing with Static …

Complementing Unit Testing with Static Analysis, with NUnit as an Example

Aug 17 2016
Author:

When discussing static analysis tools for C# projects, programmers will often deny the necessity of static analysis arguing that most errors can be caught through unit testing. So, I decided to find out how well one of the most popular unit-testing frameworks, NUnit, was tested and see if our analyzer could find anything of interest there.

0420_NUnit/image1.png

Introduction

NUnit is a popular unit-testing library for .NET projects ported from Java to C#. Its code is open and can be downloaded from the project website http://www.nunit.org/.

It should be noted that JUnit - the project that NUnit was ported from - was created by such renowned programmers as Erich Gamma, a co-author of the textbook on object-oriented design patterns, and Kent Beck, the creator of the test-driven development and extreme programming methodologies. I recall reading his book Test Driven Development By Example once, where he explains test-driven development by the example of creating a unit-testing framework, like JUnit, following all of his methodologies. What I mean to say is that there is no doubt that JUnit and NUnit were developed in keeping with the best traditions of unit testing, which is also confirmed by Kent Beck's comment at the NUnit site: "... an excellent example of idiomatic design. Most folks who port xUnit just transliterate the Smalltalk or Java version. That's what we did with NUnit at first, too. This new version is NUnit as it would have been done had it been done in C# to begin with."

I looked through NUnit's source files: there are piles of tests; it looks like they have tested everything that could be tested. Taking into account the project's great design and the fact that NUnit has been used by thousands of developers over a number of years, I didn't expect PVS-Studio to find a single bug there. Well, I was mistaken: it did find one bug.

About the bug found

It triggered the V3093 diagnostic, which deals with an issue when programmers use operators & and | instead of && and ||. This issue may cause troubles when it is critical that the right part of an expression should not execute under certain conditions. Let's see what this error looks like in NUnit.

public class SubPathConstraint : PathConstraint
{
    protected override bool Matches(string actual)
    {
        return actual != null &
            IsSubPath(Canonicalize(expected), Canonicalize(actual));
    }
}
public abstract class PathConstraint : StringConstraint
{
    protected string Canonicalize(string path)
    {
        if (Path.DirectorySeparatorChar !=
            Path.AltDirectorySeparatorChar)
            path = path.Replace(Path.AltDirectorySeparatorChar,
                                Path.DirectorySeparatorChar);
        ....
    }
}

Even if the Matches method receives the value null as the actual parameter, the right operand of the & operator will be evaluated anyway, which means that the Canonicalize method will be called, too. If you look at this method's definition, you'll see that the value of its path parameter is not tested for null and method Replace is called on it right away - this is where NullReferenceException might be raised. Let's try to reproduce this issue. For that purpose, I wrote a simple unit-test:

[Test]
public void Test1()
{
    Assert.That(@"C:\Folder1\Folder2", Is.SubPathOf(null));
}

Now let's run it and here's what we get:

0420_NUnit/image2.png

Here it is: NUnit crashed with NullReferenceException. PVS-Studio managed to find a real bug even in such a well-tested product as NUnit is. Note that it was no harder than writing a unit-test: you just run project analysis from the menu and check the grid with the results.

Conclusion

Unit testing and static analysis are not alternative, but complementary software-development strategies [1]. Download PVS-Studio analyzer and run it on your projects to see if it can find errors that tests didn't.

References

Popular related articles
Static analysis as part of the development process in Unreal Engine

Date: Jun 27 2017

Author: Andrey Karpov

Unreal Engine continues to develop as new code is added and previously written code is changed. What is the inevitable consequence of ongoing development in a project? The emergence of new bugs in th…
Free PVS-Studio for those who develops open source projects

Date: Dec 22 2018

Author: Andrey Karpov

On the New 2019 year's eve, a PVS-Studio team decided to make a nice gift for all contributors of open-source projects hosted on GitHub, GitLab or Bitbucket. They are given free usage of PVS-Studio s…
The Last Line Effect

Date: May 31 2014

Author: Andrey Karpov

I have studied many errors caused by the use of the Copy-Paste method, and can assure you that programmers most often tend to make mistakes in the last fragment of a homogeneous code block. I have ne…
Characteristics of PVS-Studio Analyzer by the Example of EFL Core Libraries, 10-15% of False Positives

Date: Jul 31 2017

Author: Andrey Karpov

After I wrote quite a big article about the analysis of the Tizen OS code, I received a large number of questions concerning the percentage of false positives and the density of errors (how many erro…
Technologies used in the PVS-Studio code analyzer for finding bugs and potential vulnerabilities

Date: Nov 21 2018

Author: Andrey Karpov

A brief description of technologies used in the PVS-Studio tool, which let us effectively detect a large number of error patterns and potential vulnerabilities. The article describes the implementati…
The way static analyzers fight against false positives, and why they do it

Date: Mar 20 2017

Author: Andrey Karpov

In my previous article I wrote that I don't like the approach of evaluating the efficiency of static analyzers with the help of synthetic tests. In that article, I give the example of a code fragment…
Appreciate Static Code Analysis!

Date: Oct 16 2017

Author: Andrey Karpov

I am really astonished by the capabilities of static code analysis even though I am one of the developers of PVS-Studio analyzer myself. The tool surprised me the other day as it turned out to be sma…
How PVS-Studio Proved to Be More Attentive Than Three and a Half Programmers

Date: Oct 22 2018

Author: Andrey Karpov

Just like other static analyzers, PVS-Studio often produces false positives. What you are about to read is a short story where I'll tell you how PVS-Studio proved, just one more time, to be more atte…
The Ultimate Question of Programming, Refactoring, and Everything

Date: Apr 14 2016

Author: Andrey Karpov

Yes, you've guessed correctly - the answer is "42". In this article you will find 42 recommendations about coding in C++ that can help a programmer avoid a lot of errors, save time and effort. The au…
PVS-Studio for Java

Date: Jan 17 2019

Author: Andrey Karpov

In the seventh version of the PVS-Studio static analyzer, we added support of the Java language. It's time for a brief story of how we've started making support of the Java language, how far we've co…

Comments (0)

Next comments
This website uses cookies and other technology to provide you a more personalized experience. By continuing the view of our web-pages you accept the terms of using these files. If you don't want your personal data to be processed, please, leave this site.
Learn More →
Accept