Every year, Microsoft releases a new version of .NET. This is a major event that prompts us to update PVS-Studio to support new features. Today, we'll talk about the PVS-Studio team's pain when updating Roslyn, an integral part of .NET.
For your convenience, I've divided the article in two parts: a short one for those who want to quickly find reasons for the changes made in PVS-Studio 7.34, and a long one for those who want to know the whole story.
The PVS-Studio 7.34 release brings support for .NET 9 and all new C# language features. To make it possible, we had to update Roslyn, which PVS-Studio uses for syntax analysis of the C# language, as well as to update the interdependent MSBuild libraries used for model building and parsing of C# and C++ projects (.csproj, .vcxproj).
Starting with PVS-Studio 7.34, the analysis of C# .NET, .NET Standard, and .NET Framework projects also requires the .NET 9 SDK to be installed on the system running the SDK-style analysis.
Classic .NET Framework projects remain unchanged if the system running the analysis has MSBuild or Visual Studio versions 2017 and later installed. If you only have MSBuild or Visual Studio versions 2015 and earlier installed on your system, you also need the .NET 9 SDK to analyze classic .NET Framework projects.
This change was prompted by the loss of backward compatibility in the Roslyn libraries used to analyze projects from Visual Studio\MSBuild versions 2015 and earlier. Roslyn now uses its own .NET SDK-based server component for this, which requires the .NET SDK to be installed on the system.
The short version is over.
PVS-Studio 7.34 introduced support for .NET 9 and all new C# language features.
By the way, I invite you to read our articles about the changes:
To support .NET 9, we had to update the Roslyn and MSBuild libraries. PVS-Studio and Roslyn use MSBuild to build a project model (the design-time build). This is a fairly complex process that requires some knowledge from either an employee or their mentor. One might think, "What's so hard about just clicking update on the right NuGet packages in the IDE?" Before we supported Visual Studio 2017, this was true. I could start telling you what's changed and why it's not enough to update the packages, but I'd rather share a link to an article that covers this topic.
Although, I still have to give you a brief introduction. What does one have to do to support the new .NET and why? Starting with the 15 (VS 2017) release, MSBuild can't find its own installation directory. I think this is related to MSBuild 15 and newer not being written to the registry. The following solution has been proposed to work around this issue:
Given this, to support the new .NET we needed to:
This is a very simplified scheme, and it would be better if you read the article linked above. We also have articles on Visual Studio 2019 and Visual Studio 2022 support, which cover the issues as well. After reading all that, you might be thinking, "Mmmm, yummy legacy."
Sure, legacy... Now we have a solution for MSBuild, which is MSBuildLocator. Although, it also has its limitations: it has separate builds for the .NET Framework and .NET Core. The .NET Framework version can search for and register MSBuild only for the .NET Framework while the .NET Core version is limited to do the same only for .NET Core. I guess this is because MSBuild is also split into MSBuild for .NET Framework and MSBuild for .NET Core.
Now that you have a little better understanding of what we're dealing with, let's get started.
.NET 9 RC 1 has been released and we have started implementing support. We updated the libraries and our BuildTools first. During the update, we noticed two new folders in the Microsoft.CodeAnalysis (Roslyn) package: BuildHost-net472 and BuildHost-netcore. At first, we were excited, thinking that the issues would be fewer now. But it wasn't the case.
Updating the libraries and BuildTools went pretty quickly, thanks to our prior experience. After this step, developers run a local test suite to ensure that the analysis is correct. These tests involve a large number of open-source projects that the analyzer runs on. After we ran the tests, issues emerged: in some code fragments, the analyzer started reporting (via internal logs) compilation errors in fully compiled code, and exceptions were being thrown here and there. We started fixing and managed to get the local tests to full completion.
The next step is to run the updated version on tests that run the analyzer in template projects for different versions of Visual Studio and .NET. These tests are run in containers with different environments. So, here we can see that the analysis breaks on machines with only VS 2010, 2012, 2013, 2015 installed. It also fails on SDK-style .NET Framework projects with VS 2017, 2019, 2022 but without the .NET SDK.
Why have previous tests been successful and revealed no issues? They were run locally on the developer's machine, which essentially has all versions of Visual Studio and many versions of the .NET SDK installed.
We looked into it and found the issue: the updated Roslyn made things even worse.
Roslyn used to expect users to use MSBuildLocator or something similar. We had our own BuildTools and our own toolsets that Roslyn used. This approach enabled us to support C# projects even for Visual Studio 2010. The only remaining challenge was the complexity of supporting the new .NET. As I mentioned earlier, Microsoft.CodeAnalysis now includes two additional folders. Roslyn has moved to a new architecture with Build servers for independent search and design-time build projects. Now it brings along two BuildHosts: one for the .NET Framework and one for .NET Core. Depending on the project type, Roslyn calls the appropriate BuildHost process to build the project. MSBuildLocator is used to search for MSBuild. And there's a few issues here.
BuildHost for .NET Core requires at least .NET 6 Runtime to run. Since PVS-Studio uses Roslyn, users must have the .NET 6+ Runtime installed. While this requirement is understandable and acceptable, some might wonder, "Why not release the BuildHost for .NET Core as a self-contained application or even use NativeAOT?" Unfortunately, doing so would make the .exe file significantly larger. However, the main issue lies elsewhere: under the hood, Assembly.Load is used to register MSBuild, which just doesn't work in applications like this.
MSBuildLocator for .NET Framework can only search for MSBuild 15, 16, 17 (Visual Studio 2017, 2019, 2022). So, if a user has a fairly old project and is using Visual Studio 2015, Roslyn won't be able to find a suitable MSBuild. Even if the project is fully built on the system, Roslyn simply won't work. We've reported this issue on GitHub. To cut a long story short, the devs don't prioritize it, citing that VS 2015 and earlier versions are just outdated. However, it turns out that if the user has the .NET SDK, Roslyn starts using a backup plan. If you have a .NET Framework legacy project, Roslyn will try to use BuildHost for .NET Core. Most of the time, this works fine, but issues can occur if there's something that isn't supported by MSBuild for .NET Core. You'll see this in the description of the third issue.
All of this leads us to the conclusion that if we want to maintain support for legacy projects, we need to ship the .NET SDK. For the product integrity, we decided to ship the .NET 9 SDK with it on Windows, as we already do on Linux and macOS. These operating systems require the .NET SDK of the version it was built for. Why we need the .NET 9 SDK on Linux and macOS is another discussion, but believe me, we need it.
For any SDK-style projects, Roslyn automatically uses BuildHost for .NET Core, and it doesn't matter whether MSBuild for .NET Core can actually build the project. MSBuild for .NET Core doesn't support all the features that MSBuild for .NET Framework does. For example, MSBuild for .NET Core doesn't support COMReference. We've also created an issue for this. How do we handle this on our end? Well, we don't. Such rare errors result in small flaws in the semantic model obtained from Roslyn. It doesn't affect the analysis quality in any way.
These are just the most memorable issues. During the update, we encountered a bunch of other little challenges.
So, what's the result of Microsoft's enhancements to Roslyn? A more complicated distribution process for the analyzer, requiring users to install unnecessary .NET SDKs, and dozens of days spent to make everything work as intended.
What's this article for, then? I wanted to share a little story about how updating and improving a tool can sometimes complicate life rather than simplify it. In fact, much of it could've been avoided if Roslyn had chosen BuildHost more carefully. I also wanted to provide a concise explanation for customers asking, "Why is this the case?" For this purpose, I've written the short version.
If it felt like I was being overly critical of Roslyn, you're not wrong—though only a little, and mainly because I had a lot of work added to my plate :) I still think Roslyn is a great tool that makes code review and analysis easier in many ways.
Moreover, in the future, we'd like to get rid of our BuildTools and move to a Roslyn-like architecture. As usual, this is quite a challenge as a lot of legacy code has to be rewritten or scrapped and rewritten while maintaining compatibility with old projects. It's no easy feat, really.
Thank you very much for reading :)