04.06.2021

Commit and Pull Request analysis in Travis CI, Buddy and AppVeyor

April 6, 2021

File list checking mode
General principles of Pull Request analysis
Travis CI
Buddy
AppVeyor

This article describes how to check Pull Requests on Linux and macOS. To learn about checking Pull Requests on Windows, use documentation for the PVS-Studio_Cmd and CLMonitor utilities (the "Specification of individual files for analysis" section).

Pull Request analysis is available only under PVS-Studio's Enterprise license. You can request a trial Enterprise license at the downloads page.

File list checking mode

Starting with version 7.04, PVS-Studio has a source file list checking mode for Linux and macOS. It works for projects whose build system can generate the 'compile_commands.json' file. The analyzer uses it to extract information about specific files' compilation. If your build system does not support the 'compile_commands.json' file generation, you can try using the Bear utility to generate this file.

You can also use the file list checking mode together with 'strace' (pvs-studio-analyzer trace) - a log that contains intercepted compiler invocations. To do this, you can build the entire project in trace mode. This way, the analyzer gets complete information about compilation flags for all files.

However, this approach has a significant drawback. Each time you run an analysis, you will need to build an entire project and do its complete trace. This contradicts the idea of a quick commit check. Alternatively, you can cache the trace result. But if you change source file dependency structure (for example, add a new #include to a source file), this can make subsequent analyzer runs incomplete.

This is why, if you need to check commits or Pull Requests, we do not recommend combining the file list checking mode with the trace log feature. If, when checking a commit, you can do an incremental build, do look into the incremental analysis mode.

In the file list checking mode, a list of source files is stored to a text file. Then you can use the '-S' parameter to pass this data to the analyzer.

pvs-studio-analyzer analyze ... -f build/compile_commands.json -S check-list.txt

This file contains relative or absolute file paths. Each path must start on a new line. The analyzer ignores any text that is not a source code file path. You can use this specificity to add comments when indicating files manually. However, often the file list is generated during analysis in CI. Thus, the list may include files from a commit or a Pull Request.

At the start of a project's first analysis, the analyzer generates a cache file that is required for the file checking mode to operate correctly. The file contains dependencies of the project's all source files from header files. The need for this file is a peculiarity of C/C++ file analysis. In the future, the file with dependencies will be cached and the analyzer will update it automatically.

To generate or update the dependency file without starting the analysis, use the '--regenerate-depend-info' flag with the 'skip-analysis' key:

pvs-studio-analyzer analyze ... -f build/compile_commands.json \ 
                                --regenerate-depend-info skip-analysis

To analyze a list of files or an entire project, and force dependency cache update, use the '--regenerate-depend-info' flag with the 'run-analysis' key:

pvs-studio-analyzer analyze ... -S source_files \
                                -f build/compile_commands.json \
                                --regenerate-depend-info run-analysis

Now you can use the file list checking mode to quickly check new code before you commit it to the master development branch. In order for the system to recognize the analyzer's messages, the 'plog-converter' utility provides the '--indicate-warnings' flag:

plog-converter ... --indicate-warnings ... -o /path/to/report.tasks ...

When this flag is set and the analyzer's report contains warnings, the converter returns a non-zero code. You can use the return code to block the pre-commit hook, commit or Pull Request. You send the analyzer's generated report by email, or show it on the screen.

General principles of Pull Request analysis

It may take a long time to check an entire project. Usually, it is a good idea to check the project's select part. The main problem is separating new files from the project's other files.

Take a look at this at this sample commit tree with two branches:

Let's say, most of the code from commit 'A1' has already been checked. A little earlier, a developer created the 'hotfix' branch from the 'A1' commit and changed some files.

As we see from the example above, after 'A1', two more changes were committed to the 'master' branch. As soon as the 'hotfix' branch is ready to be merged, the developer opens a Pull Request to merge 'B3' and 'A3' changes.

You can check the entire result of merging two branches. This approach is slow and not justified, because only a few files were modified. It's more efficient to check only them.

To do this, when in the HEAD of the branch that you intend to merge into 'master', you need to get the diffs between the branches:

git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list

We will discuss the '$MERGE_BASE' variable further in this article. Not every CI service provides the necessary information about the diffs, so in each scenario finding new ways to get this data may be necessary. Below we will provide a detailed description for each described web-service.

After you get the diffs, or to be precise, a list of files that were changed, pass the '.pvs-pr.list' file to the analyzer:

pvs-studio-analyzer analyze -j8 \
                            -o PVS-Studio.log \
                            -S .pvs-pr.list

After the analysis convert the log file (PVS-Studio.log) to a convenient format:

plog-converter -t errorfile PVS-Studio.log --cerr -w

This command will output warnings into stderr (the standard error message output stream).

In addition to listing errors, you can notify the build and test service about these problems. To do this, add the '-W' ('--indicate-warnings') flag to 'plog-converter'. If there is at least one warning, the 'plog-converter' utility return code will change to 2, which, in its turn, will tell the CI service about potential errors in the Pull Request files.

Travis CI

Travis CI stores its configuration in the '.travis.yml' file. For convenience, create a separate bash-script with functions that will be called from the '.travis.yml' file ('bash script_name.sh function_name').

Add the necessary code to the 'bash' script. Indicate the following in the 'install' section:

install:
  - bash .travis.sh travis_install

Open the '.travis.sh' file and add installing the analyzer to the 'travis_install()' function:

travis_install() {
  wget -q -O - https://files.viva64.com/etc/pubkey.txt \
    | sudo apt-key add -
  sudo wget -O /etc/apt/sources.list.d/viva64.list \
    https://files.viva64.com/etc/viva64.list
  
  sudo apt-get update -qq
  sudo apt-get install -qq pvs-studio 
}

Add the analysis launch to the 'script' section:

script:
  - bash .travis.sh travis_script

And to the bash-script:

travis_script() {
  pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY
  
  if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then
    git diff --name-only origin/HEAD > .pvs-pr.list
    pvs-studio-analyzer analyze -j8 \
                                -o PVS-Studio.log \
                                -S .pvs-pr.list \
                                --disableLicenseExpirationCheck
  else
    pvs-studio-analyzer analyze -j8 \
                                -o PVS-Studio.log \
                                --disableLicenseExpirationCheck
  fi
  
  plog-converter -t errorfile PVS-Studio.log --cerr -w
}

Run this code after the project is built, for example, after you've built your project on CMake:

travis_script() {
  CMAKE_ARGS="-DCMAKE_EXPORT_COMPILE_COMMANDS=On ${CMAKE_ARGS}"
  cmake $CMAKE_ARGS CMakeLists.txt
  make -j8
}

This is the result:

travis_script() {
  CMAKE_ARGS="-DCMAKE_EXPORT_COMPILE_COMMANDS=On ${CMAKE_ARGS}"
  cmake $CMAKE_ARGS CMakeLists.txt
  make -j8
  
  pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY
  
  if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then
    git diff --name-only origin/HEAD > .pvs-pr.list
    pvs-studio-analyzer analyze -j8 \
                                -o PVS-Studio.log \
                                -S .pvs-pr.list \
                                --disableLicenseExpirationCheck
  else
    pvs-studio-analyzer analyze -j8 \
                                -o PVS-Studio.log \
                                --disableLicenseExpirationCheck
  fi
  
  plog-converter -t errorfile PVS-Studio.log --cerr -w
}

Travis CI independently declares the '$TRAVIS_PULL_REQUEST' and '$TRAVIS_BRANCH' environment variables:

'$TRAVIS_PULL_REQUEST' stores the Pull Request number, or, if it is a regular branch, the value is 'false';
'$TRAVIS_REPO_SLUG ' stores the project repository name.

The diagram below demonstrates how the function operates:

Travis CI can recognize return codes. Thus, the presence of warnings will instruct the service to tag the commit as one with errors.

Let's take a closer look at this line of code:

git diff --name-only origin/HEAD > .pvs-pr.list

Travis CI merges branches automatically during Pull Request analysis:

In this case, the analyzer processes 'A4' instead of 'B3->A3'. This specificity makes it necessary to calculate the difference with 'A3' that is at the top of the branch from 'origin'.

The last important detail is caching dependencies of header files from compiled translation units (*.c, *.cc, *.cpp etc). During the first run in the file list checking mode, the analyzer calculates these dependencies and stores them to the '.PVS-Studio' directory. Travis CI allows to cache folders, which is why we'll save the '.PVS-Studio' directory data:

cache:
  directories:
    - .PVS-Studio/

Add this code to the '.travis.yml' file. The '.PVS-Studio' directory stores various data collected after the analysis. This data significantly speeds up subsequent runs of the file list or incremental analysis. If you do not cache dependencies, the analyzer will process all files every time.

Buddy

Like Travis CI, Buddy enables you to automatically build and test projects stored on GitHub. In contrast to Travis CI, Buddy is configured in the web interface (bash support is available), so you do not need to store configuration files in the project.

First, add a new action to the pipeline:

Specify the compiler you used to build the project. Note the docker container, installed in this step. For example, there's a special container for GCC:

Now install PVS-Studio and the necessary utilities:

Add the following lines to the editor:

apt-get update && apt-get -y install wget gnupg jq

wget -q -O - https://files.viva64.com/etc/pubkey.txt | apt-key add -
wget -O /etc/apt/sources.list.d/viva64.list \
  https://files.viva64.com/etc/viva64.list

apt-get update && apt-get -y install pvs-studio

Now switch to the 'Run' tab (the first icon). Add the following code to the editor:

pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY

if [ "$BUDDY_EXECUTION_PULL_REQUEST_NO" != '' ]; then
  PULL_REQUEST_ID="pulls/$BUDDY_EXECUTION_PULL_REQUEST_NO"
  MERGE_BASE=`wget -qO - \
    https://api.github.com/repos/${BUDDY_REPO_SLUG}/${PULL_REQUEST_ID} \
    | jq -r ".base.ref"`

  git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list
  pvs-studio-analyzer analyze -j8 \
                              -o PVS-Studio.log \
                              --disableLicenseExpirationCheck \
                              -S .pvs-pr.list
else
  pvs-studio-analyzer analyze -j8 \
                              -o PVS-Studio.log \
                              --disableLicenseExpirationCheck
fi

plog-converter -t errorfile PVS-Studio.log --cerr -w

If you read the Travis-CI section, you already know this code. However, now there's a new step:

Buddy does not analyze the result of branch merging. It analyzes the HEAD of the Pull Request's branch:

This is why if your commit is 'B3', you need to compare it to 'A3' and get the diffs:

PULL_REQUEST_ID="pulls/$BUDDY_EXECUTION_PULL_REQUEST_NO"
  MERGE_BASE=`wget -qO - \
    https://api.github.com/repos/${BUDDY_REPO_SLUG}/${PULL_REQUEST_ID} \
    | jq -r ".base.ref"`
git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list

To define 'A3', use GitHub's API:

https://api.github.com/repos/${USERNAME}/${REPO}/pulls/${PULL_REQUEST_ID}

Buddy provides the following environment variables:

'$BUDDY_EXECUTION_PULL_REQEUST_NO' - the Pull Request number;
'$BUDDY_REPO_SLUG' - a combination of the user name and the repository name (for example, max/test).

Use the button below to save the changes, and start Pull Request analysis:

As opposed to Travis CI, there is no need to specify the '.PVS-Studio' folder for caching, because buddy automatically caches all files for subsequent runs. So the last step is to save your username and password for PVS-Studio in Buddy. After saving changes, go back to the 'Pipeline', open variable settings and add the login and key for PVS-Studio:

After this, adding a new Pull Request or commit will run the analysis. If the commit contains errors, Buddy will indicate this on the Pull Request page.

AppVeyor

AppVeyor, same as Buddy, is configured in the web interface. You do not need to add the '*.yml' file into the project repository.

In the project overview, switch to the 'Settigs' tab:

Scroll the page down and check "Save build cache in Pull Requests":

Now choose the 'Environment' tab. Indicate the image used for the build and specify required environment variables:

If you read previous sections, you already know these two variables - 'PVS_KEY' and 'PVS_USERNAME'. AppVeyor uses them to check the PVS-Studio license.

Scroll the page down and set the cached directory:

If you do not specify this setting, the analyzer will process the entire project instead of a few files, while the output will contain information only on the specified files. This is why it's important to enter a correct directory name.

Then proceed to writing the script that manages the project's analysis. Open the 'Tests' tab and select 'Script':

Paste the following code into the 'Test script' textbox:

sudo apt-get update && sudo apt-get -y install jq

wget -q -O - https://files.viva64.com/etc/pubkey.txt \
  | sudo apt-key add -
sudo wget -O /etc/apt/sources.list.d/viva64.list \
  https://files.viva64.com/etc/viva64.list

sudo apt-get update && sudo apt-get -y install pvs-studio

pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY

PWD=$(pwd -L)
if [ "$APPVEYOR_PULL_REQUEST_NUMBER" != '' ]; then
  PULL_REQUEST_ID="pulls/$APPVEYOR_PULL_REQUEST_NUMBER"
  MERGE_BASE=`wget -qO - \
    https://api.github.com/repos/${APPVEYOR_REPO_NAME}/${PULL_REQUEST_ID} \
    | jq -r ".base.ref"`

  git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list
  pvs-studio-analyzer analyze -j8 \
                              -o PVS-Studio.log \
                              --disableLicenseExpirationCheck \
                              --dump-files --dump-log pvs-dump.log \
                              -S .pvs-pr.list
else
  pvs-studio-analyzer analyze -j8 \
                              -o PVS-Studio.log \
                              --disableLicenseExpirationCheck
fi

plog-converter -t errorfile PVS-Studio.log --cerr -w

Pay special attention to the following code fragment:

PWD=$(pwd -L)
if [ "$APPVEYOR_PULL_REQUEST_NUMBER" != '' ]; then
  PULL_REQUEST_ID="pulls/$APPVEYOR_PULL_REQUEST_NUMBER"
  MERGE_BASE=`wget -qO - \
   https://api.github.com/repos/${APPVEYOR_REPO_NAME}/${PULL_REQUEST_ID} \
   | jq -r ".base.ref"`

  git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list
  pvs-studio-analyzer analyze -j8 \
                              -o PVS-Studio.log \
                              --disableLicenseExpirationCheck \
                              --dump-files --dump-log pvs-dump.log \
                              -S .pvs-pr.list
else
  pvs-studio-analyzer analyze -j8 \
                              -o PVS-Studio.log \
                              --disableLicenseExpirationCheck
fi

Note how the '$PWD' variable is assigned the 'pwd' command's value. The procedure may seem unusual, but it facilitates the analyzer's correct operation. If you do not set this variable, it will contain a value AppVeyor had specified for its internal purposes.

Then follows the algorithm you already know:

Now take a look at this code fragment:

PULL_REQUEST_ID="pulls/$APPVEYOR_PULL_REQUEST_NUMBER"
MERGE_BASE=`wget -qO - \
  https://api.github.com/repos/${APPVEYOR_REPO_NAME}/${PULL_REQUEST_ID} \
  | jq -r ".base.ref"`

This code gets diffs for branches involved in the Pull Request. The code uses the following environment variables:

'$APPVEYOR_PULL_REQUEST_NUMBER' - the Pull Request number;
'$APPVEYOR_REPO_NAME' - the user name and the repository name.