Now it's hard to imagine software development without automated project builds and testing. There are various ready-made solutions to minimize the time expenses for the integration of the modifications into the project. In this article I am going to speak about the way PVS-Studio team changed the continuous integration server from CruiseControl.NET to Jenkins I will also be talking about the motives behind this decision, the goals we tried to pursue and the issues we had to deal with during that process.
Continuous integration - is an automated process of building, deploying and testing software. This practice is popular both among large development teams and individual developers. There is quite a number of solutions for this practice. In this article we are going to talk about free open source projects CruiseControl.NET and Jenkins.
CruiseControl.NET(CCNet) — a tool for continuous integration of software implemented on the .NET Framework. There are also variants of the tool in Java (CruiseControl) and a version for the Ruby-environments (CruiseControl.rb). You can view and manage the information about the builds through web-interface or a desktop utility. It integrates with different version control systems. Is an open source project but, unfortunately, is not developing since 2013.
Jenkins — a tool for continuous integration with open source, written in Java. It was forked from the Hudson project after an argument with Oracle. Providing functions of continuous integration, it allows automating a part of the development process, which does not require the involvement of human developers. The abilities of Jenkins can be extended through plugins. At the moment the project is actively developed and supported by the developers and the community.
Although this article may look like a review in the "CCNet vs. Jenkins" style, the choice is already made for a server with Jenkins. The main reason why we changed the tool for continuous integration is that the CruiseControl.NET project is no longer developing. There were also other issues when working with CCNet that we are going to cover later.
Our project has quite a long history - recently we had a 10-year anniversary, you may read the story in the article "PVS-Studio project - 10 years of failures and successes". We were using CCNet for more than 5 years. We got so used to its interface, settings and functions, that Jenkins seemed to be really uncomfortable. First, we started using it with the release of PVS-Studio for Linux. The switch to the Jenkins platform was preceded by a long study of the tool. Some time was spent on looking for the similar functions from CCNet. Further on, I will describe the most interesting moments from our work.
The settings of the CCNet projects (configuration of the server) were stored in one xml file, and various passwords in a different one. Although the settings file reached the size of ~4500 lines, it was still quite convenient to use it. By pressing Alt+2 in Notepad++, the whole list of projects can be minimized and you can edit the one you need (figure 1).
Figure 1 - Editing the CCNet settings in Notepad++
Although the file had some duplicate code, there were no difficulties with the server support.
Here is how the SCM block was configured:
<svn>
<username>&SVN_USERNAME;</username>
<password>&SVN_PASSWORD;</password>
<trunkUrl>&SVN_ROOT;...</trunkUrl>
<workingDirectory>&PROJECT_ROOT;...</workingDirectory>
<executable>&SVN_FOLDER;</executable>
<deleteObstructions>true</deleteObstructions>
<cleanUp>true</cleanUp>
<revert>true</revert>
<timeout units="minutes">30</timeout>
</svn>Here is how the MSBuild block was configured:
<msbuild>
<description>PVS-Studio 2015</description>
<workingDirectory>&PROJECT_ROOT;...</workingDirectory>
<projectFile>...\PVS-Studio-vs2015.sln</projectFile>
<buildArgs>/p:Configuration=Release</buildArgs>
<targets>Build</targets>
<timeout>600</timeout>
<executable>&MSBUILD14_PATH;</executable>
</msbuild>Here is how the block for common tasks was configured
<exec>
<description>PVS-Studio 2015 sign</description>
<executable>&PROJECT_ROOT;...\SignToolWrapper.exe</executable>
<baseDirectory>&PROJECT_ROOT;...</baseDirectory>
<buildArgs>"&SIGNTOOL;" ... \PVS-Studio-vs2015.dll"</buildArgs>
<buildTimeoutSeconds>600</buildTimeoutSeconds>
</exec>On the basis of such a project file, CCNet displays very conveniently all the executable steps (albeit only in the desktop tray utility, for some reason the web-interface didn't support this). In Jenkins we had some troubles with the "high-level" display of stages of completing an integration project, but we will speak about that later.
In Jenkins you have to store quite a number of settings files: a server configuration, settings files for some plugins, each project has its own configuration file. Although all of these files exist in the xml format, they aren't very convenient to view (at least in comparison with CCNet), because all the commands in the tags are written as plain text. The truth is, this is largely connected with the ideology of the use of the instrument. In CCNet the config is written manually, so it can be "nicely" formatted. Jenkins suggests editing project settings using its web interface and automatically generates configs.
Here is an example of how the commands look like in the Jenkins configs:
<hudson.tasks.BatchFile>
<command>CD "%BUILD_FOLDERS%\Builder"
PVS-Studio_setup.exe /VERYSILENT /SUPPRESSMSGBOXES ...
Publisher_setup.exe /VERYSILENT /SUPPRESSMSGBOXES</command>
</hudson.tasks.BatchFile>And this is quite a small example.
As I have written before, in CCNet the projects are configured with Task blocks. Here is how a successfully completed task looks with all the steps (figure 2).
Figure 2 - viewing the status of tasks in the CCTray (a desktop client for CCNet)
An error in any of the blocks can be clearly seen in the hierarchy of subtasks. It is very convenient and illustrative visualization of the integration process. We almost never had to search for logs, it was clear looking at the description what requires checking on the local computer. We could not find such a feature in Jenkins, that's why we had to study this question really thoroughly before moving to Jenkins.
We can draw such analogy between CCNet and Jenkins: there is a project in CCNet (Project), the 'step' in this project is called a Task (as you can see on the figure above). In Jenkins there is also a project (Job), but its 'steps' are called the same - Steps (figure 3).
Figure 3 - Corresponding names in CCNet and Jenkins
Unfortunately, the web interface of Jenkins cannot visualize individual steps - the Job has only a full console log of all steps together. A big disadvantage is that you cannot see in Jenkins which of the steps failed - you need to watch the full log of the Job build. And as people get used to good things very quickly, we didn't want to abandon the old usage scenario. At that time we discovered Multijob Plugin
This plugin allowed us making the following improvements:
1. Using Jobs as Steps in other Jobs. Thus we got universal Jobs that allowed separating the log of certain subtasks and the main thing to view statuses of separate subtasks. The Jenkins web interface can visualize quite well the completion of individual Jobs within a Multijob - it's exactly what we were looking for. Figure 4 shows an example of a completed Multijob.
Figure 4 - viewing a completed Multijob.
2. We managed to get rid of duplicate code using universal Jobs. For example, there is a compilation of some utility: for the distribution, for tests, for the code analysis. In CCNet these were same Task blocks in 3 different projects. To compile this utility in Jenkins there is a Job that is used by several Multijobs.
3. During the creation of projects in Jenkins, we have the following workflow. We divide all the Jobs into Multijob "projects" and universal "steps". The names of the universal Jobs have a prefix "job_" and do not suggest using as a separate (stand-alone) project. They also don't have loading the source code from the repository. The names of Multijobs have the "proj_" prefix and include loading of the source code and running only other Jobs. (We try to avoid Steps, because they are not visualized).
The universal Jobs are run with the following parameter:
WORKSPACE=$WORKSPACEIt means that the Job will be run in the working directory of the Multijob.
Thus we can get a log of the source files updates and logs of all the steps of the build separately. It would be hard and meaningless to follow this ideology for all the projects. It is done only for several biggest and most important projects that should be studied in detail in case of problems.
4. You can configure conditional and parallel launch of the Jobs in Multijob. Multijobs can run Multijobs using the same ways. Thus you can combine launches of different projects: to start the build of all installers or all the tests.
It was extremely inconvenient to view the build logs of the projects, because they were mixed with the server output and had a special markup. There is no such a problem in Jenkins and there is a possibility to separate the logs for subtasks in some projects.
In Jenkins a separate revision of the version is defined for every added link to the SVN repository. I.e. if we add several directories, the numbers can vary greatly, but we need only one maximum number.
According to the documentation, we should work with this as follows:
If you have multiple modules checked out, use the svnversion command. If you have multiple modules checked out, you can use the svnversion command to get the revision information, or you can use the SVN_REVISION_<n> environment variables, where <n> is a 1-based index matching the locations configured.
It was done in this very way: of all the obtained values SVN_REVISION_<n> , the maximum value was chosen and added to the built programs.
The enhancement of the tool with the help of the plugins allows flexible configuration of the server. The plugins to run the tests in Visual Studio are perhaps the only ones that we refused to use. They had additional mandatory launch parameters, that we didn't use that's why it was easier to create a universal Job that would just start the tests from a command line. Unfortunately, Jenkins, being "out of the box" couldn't do much of what we got so used to in CCNet. However, we manage "to get back" all the necessary functionality.
Here is a list of those plugins that we found useful, with a small description:
To receive notifications from the CCNet we used a client for Windows - CCTray.
Here are variants that are now available to work with Jenkins:
1. CCTray - this program can be used for a Jenkins server as well. The projects will look approximately the same as they were before (figure 5).
Figure 5 - CCTray screenshot
The description of CCTray as a Jenkins client:
2. CatLight (figure 6)
Figure 6 - a screenshot of CatLight
Client description:
3. Kato (figure 7)
Figure 7 - screenshot of Kato
Client description:
4. CCMenu - a client only for Mac, open source. It is not relevant for us, but perhaps, some may need it.
Using CI is useful in any project. There is a great free tool Jenkins that was reviewed in the article, and also there are a lot of other free and paid Cl. It is rather pleasant to use a developing the project: a large number of upgrades are released for Jenkins and plugins. New solutions get created, as for example, Blue Ocean project that is still in Beta. You may find it on the main page of the Jenkins site.
Although, the clients for monitoring Jenkins projects weren't much impressive to me. A lot of obvious features are missing. Perhaps desktop clients are particularly in demand and it is more correct to use only some web interface.
When moving to a new server, we couldn't use Jenkins as a Windows service, because in this mode the UI tests do not run. We found a way out by setting up a server as a console application with a hidden window.
If you have any comments to this material of some interesting solutions of the problems we've mentioned here, we'll be glad to get them in the comment section of via a feedback form.