To get a trial key
fill out the form below
Team License (standard version)
Enterprise License (extended version)
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Request our prices
New License
License Renewal
--Select currency--
USD
EUR
GBP
RUB
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
To get the licence for your open-source project, please fill out this form
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
To get the licence for your open-source project, please fill out this form
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
I am interested to try it on the platforms:
* By clicking this button you agree to our Privacy Policy statement

** This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Message submitted.

Your message has been sent. We will email you at


If you haven't received our response, please do the following:
check your Spam/Junk folder and click the "Not Spam" button for our message.
This way, you won't miss messages from our team in the future.

>
>
>
PVS-Studio wanted but couldn't find bug…

PVS-Studio wanted but couldn't find bugs in robots.txt

Jul 11 2019

The other day Google revealed the sources of the robots.txt parser. Why not give a run for the already far and wide checked project using PVS-Studio and possibly find a bug. So said so done. But I wish we could find something meaningful. Well, then let it be just a reason to give full marks for Google developers.

0638_Robots_Parser_Google/image1.png

robots.txt - is an index file that contains rules for search robots. It works for https, http and FTP protocols. Google made the parser of the robots.txt file available for everyone. Read more about this news here: Google opens the source code of the robots.txt parser

I think most of our readers know what PVS-Studio does. But in case it's your first time on our blog, I'll give a brief reference. PVS-Studio is a static code analyzer that allows you to find a variety of bugs, vulnerabilities, and flaws in projects written in C, C++, C# and Java. In other words, PVS-Studio is a SAST solution and it can work both on user machines, build servers and in the cloud. The PVS-Studio team also likes writing articles on checks of various projects. So let's get to the point and try to find errors in the source code of the parser from Google.

Unfortunately, but to the delight of everyone else, no mistakes were found. Only a couple of minor flaws, which I will tell about. Well, I have to write something about the project :). The lack of errors is due to the small amount of the project and high quality of the code itself. This doesn't mean that there are no hidden errors, but static analysis was helpless at that moment.

So this article happened to be in the spirit of another our post "The Shortest Article about a Check of nginx".

I found a case with possible optimization:

V805 Decreased performance. It is inefficient to identify an empty string by using 'strlen(str) > 0' construct. A more efficient way is to check: str[0] != '\0'. robots.cc 354

bool RobotsTxtParser::GetKeyAndValueFrom(char **key, ....)
{
  ....  
  *key = line;
  ....
  if (strlen(*key) > 0) {
    ....
    return true;
  }
  return false;
}

It's inefficient to call the strlen function to find out if a string is empty. This check can be much simpler: if (*key[0] != '\0'). This way you don't have to traverse the entire string, if it's not empty.

V808 'path' object of 'basic_string' type was created but was not utilized. robots.cc 123


std::string GetPathParamsQuery(....)
{
  std::string path;
  ....
}

The string is declared, but not used further. In some cases, unused variables may indicate an error. In this case, it looks like this variable was used somehow, but after making changes it became unnecessary. Thus, the analyzer often helps to make the code cleaner and helps to avoid errors by simply removing prerequisites for their appearance.

In the next case, the analyzer recommends to add a default return after the entire main is executed. Perhaps it is worth adding a return statement at the very end in order to understand that everything has really worked out. However, if such behavior was intended, nothing needs to be changed. If you don't want to see this warning, in PVS-Studio you can suppress it and never see it again :).

V591 The 'main' function does not return a value, which is equivalent to 'return 0'. It is possible that this is an unintended behavior. robots_main.cc 99

int main(int argc, char** argv)
{
  ....
  if (filename == "-h" || filename == "-help" || filename == "--help")
  {
    ShowHelp(argc, argv);
    return 0;
  }
  if (argc != 4)
  {
    ....
    return 1;
  }
  if (....)
  {
    ....
    return 1;
  }
  ....
  if (....)
  {
    std::cout << "...." << std::endl;
  }
}

I also found that two functions below which had different names were implemented in the same way. Perhaps this is the result of the fact that earlier these functions had different logic, but came to one. It may be that a typo crept somewhere, so such warnings should be carefully checked.

V524 It is odd that the body of 'MatchDisallow' function is fully equivalent to the body of 'MatchAllow' function. robots.cc 645

int MatchAllow(absl::string_view path, absl::string_view pattern)
{
  return Matches(path, pattern) ? pattern.length() : -1;
}

int MatchDisallow(absl::string_view path, absl::string_view pattern)
{
  return Matches(path, pattern) ? pattern.length() : -1;
}

It's the only place I'm suspicious of. It should be checked by the project's authors.

Thus, the check of the robots.txt parser from Google showed that this project, which have been checked multiple times and is widely used, is of great quality. Even some found flaws cannot spoil the impression of cool Google coders writing this project :).

We suggest you as well to download and try PVS-Studio on the project you're interested in.

Popular related articles
The Evil within the Comparison Functions

Date: May 19 2017

Author: Andrey Karpov

Perhaps, readers remember my article titled "Last line effect". It describes a pattern I've once noticed: in most cases programmers make an error in the last line of similar text blocks. Now I want t…
Static analysis as part of the development process in Unreal Engine

Date: Jun 27 2017

Author: Andrey Karpov

Unreal Engine continues to develop as new code is added and previously written code is changed. What is the inevitable consequence of ongoing development in a project? The emergence of new bugs in th…
How PVS-Studio Proved to Be More Attentive Than Three and a Half Programmers

Date: Oct 22 2018

Author: Andrey Karpov

Just like other static analyzers, PVS-Studio often produces false positives. What you are about to read is a short story where I'll tell you how PVS-Studio proved, just one more time, to be more atte…
The Last Line Effect

Date: May 31 2014

Author: Andrey Karpov

I have studied many errors caused by the use of the Copy-Paste method, and can assure you that programmers most often tend to make mistakes in the last fragment of a homogeneous code block. I have ne…
Appreciate Static Code Analysis!

Date: Oct 16 2017

Author: Andrey Karpov

I am really astonished by the capabilities of static code analysis even though I am one of the developers of PVS-Studio analyzer myself. The tool surprised me the other day as it turned out to be sma…
The way static analyzers fight against false positives, and why they do it

Date: Mar 20 2017

Author: Andrey Karpov

In my previous article I wrote that I don't like the approach of evaluating the efficiency of static analyzers with the help of synthetic tests. In that article, I give the example of a code fragment…
PVS-Studio for Java

Date: Jan 17 2019

Author: Andrey Karpov

In the seventh version of the PVS-Studio static analyzer, we added support of the Java language. It's time for a brief story of how we've started making support of the Java language, how far we've co…
Technologies used in the PVS-Studio code analyzer for finding bugs and potential vulnerabilities

Date: Nov 21 2018

Author: Andrey Karpov

A brief description of technologies used in the PVS-Studio tool, which let us effectively detect a large number of error patterns and potential vulnerabilities. The article describes the implementati…
Free PVS-Studio for those who develops open source projects

Date: Dec 22 2018

Author: Andrey Karpov

On the New 2019 year's eve, a PVS-Studio team decided to make a nice gift for all contributors of open-source projects hosted on GitHub, GitLab or Bitbucket. They are given free usage of PVS-Studio s…
Characteristics of PVS-Studio Analyzer by the Example of EFL Core Libraries, 10-15% of False Positives

Date: Jul 31 2017

Author: Andrey Karpov

After I wrote quite a big article about the analysis of the Tizen OS code, I received a large number of questions concerning the percentage of false positives and the density of errors (how many erro…

Comments (0)

Next comments

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
This website uses cookies and other technology to provide you a more personalized experience. By continuing the view of our web-pages you accept the terms of using these files. If you don't want your personal data to be processed, please, leave this site.
Learn More →
Accept