Episode 300 of CppCast with guest Marshall Clow recorded May 18th, 2021. In this episode, Rob Irving and Jason Turner are joined by Marshall Clow who talks about ABI stability. They also discuss Visual C++ and ASAN, Meeting C++ 2021 and the new RmlUI update.
Rob: Welcome to episode 300 of CppCast! Joining us today is Marshall Clow. He is a longtime LLVM and Boost participant. He was the code owner for libc++ the LLVM standard library implementation until last year. He was also the chairman of the Library Working Group of the C++ standard committee. He is the author of the Boost algorithm library and maintains several other Boost libraries.
Jason: I am curious, since you said that for a long time you were the co-owner of libc++. When did you get involved with libc++?
Marshall: 2012-ish. The principal author of libc++ at the time was Howard Hinnant, who was working at Apple. And Howard has been doing C++ standard library work for a long time. He too was a standard library chair. And he's the author of move semantics in C++. Then he left Apple and went to work for Ripple and didn't have time to do libc++ anymore. And I stepped up and basically led that project for five or six years. And you know, Howard is still around, and he doesn't contribute much anymore, but he asks questions about "why is this thing done this way?" And then I kind of burned out towards the end of the C++20 and needed to shed some of these responsibilities.
Rob: Well, Marshall, we've got a couple of news articles to discuss, feel free to comment on any of these and we'll start talking more about C++ ABI.
Rob: The first one is a blog post on the Visual C++ blog, and it says, "Finding bugs with AddressSanitizer: patterns from open-source projects". And we've talked a lot about how Microsoft now has ASAN as a built-in analysis tool in Visual C++. It is really great that they were able to bring this in from Clang. And they just ran it against a couple open-source libraries, found a few bugs and were able to show those to the library owners. And I think all of these have been fixed.
Marshall: I'm happy that Microsoft has this in libc++, that's a great thing. I'm a little amused that they're talking about it as if it's new in 2021. I went back and researched, found an old blog post I wrote basically running the libc++ test suite under ASAN to find bugs in ASAN. That was March 2013.
Jason: This article made me a little sad because these bugs that they're finding in open-source projects... I'm like, "wait a minute. Are you telling me that open SSL does not currently run their full test suite with an AddressSanitizer enabled? That's like world-ends kind of problem if open SSL has an unknown security flaw or a security flaw known by a small group of people!"
Marshall: An out-of-balance reader, right?
Jason: Yeah. Now to be fair, that, that bug that they found was a bug in the test suite specifically. But still it should have been caught.
Marshall: When I ran ASAN for the first time, I found a couple of bugs in the libc++ test suite. There was one real bug in libc++, and it had to do with something deep in iostreams. When we are starting up a stream, it would allocate a zero-byte buffer on the heap and then write one byte to it and then resize it. Nobody ever saw any problems with this because, you know, on macOS heap allocations always get rounded up to a multiple of 16 bytes, even as zero byte. And it never did any harm, but it was wrong on that. macOS would never do any harm. You would never see any ill effects from it, but it was still wrong. And if you took it to some other operating system with another allocator, that did not have that behavior, you could get bugs, you could get incorrect behaviors. And I was quite impressed when I did this, because it was like, whoa, I did never find that bug.
Jason: It's a fascinating one too, because the first time you run a tool against your project, you're thinking, oh, my code's perfect, surely it won't find anything. But on the other hand, you hope that it finds something. Because otherwise if you don't already know the tool and don't already know that you can trust it, then you're like, well, did it actually work? If it didn't find anything, did it run on my code?
Marshall: Well, the good news is that at the time the libc++ test suite took about 20 minutes to run. And when I enabled ASAN, it took 90 minutes to run. So clearly, it's doing something. The way ASAN works, the idea of something being a false positive is just not in its vocabulary. Every time ASAN goes off, there's a bug there.
Rob: All right. The next thing we have is an update of RmlUI. I don't think we've talked about RmlUI before, but it's an HTML CSS UI library for C++.
Jason: I've just been like five minutes during it and don't even understand it. I'm like, "wait a minute, why am I writing HTML with C++, what is this?"
Rob: It's their own UI kit, and they have a couple like little examples on the GitHub page. So it's mostly for video games. If you want to have a menu page with your game, this is maybe a really easy way to put that together.
Jason: Or if you just like HTML and CSS. You can use it as your interface description language or one of these markup languages for interfaces.
Rob: It looks like it's really easy to do data binding between your HTML code and C++. So that seems like it's pretty powerful.
Jason: The sprite sheets, it looks pretty crazy.
Marshall: I've never used anything like this, but the data binding looks pretty slick. Although, Model View Controller kind of stuff is really hard to get exactly right.
Rob: And then the other library we have here is called "Not enough standards", and this is little C++ 17 and 20 header-only utility library. And there were a couple neat things in here, like process management and shared library loading.
Jason: The process tool really caught my eye to just be able to very easily launch something and then stand there and send it out from its cross platform. And I know there's the Boost process and QProcess, but for the project I'm currently working on, I don't want Boost or QT in there. And what's funnier is just like last week I was Googling around, like, I know there are other process libraries out there and I couldn't find any until I stumbled upon this one.
Marshall: Yeah. Although what you said is actually a fairly common thing. "I don't want Boost in there" - I understand that. But you know, Boost is a collection of libraries, some of which are very large and some of which are small. They are not completely interdependent. You can use little bits of Boost without having all the Boost around.
Rob: Okay. And then the last thing we have is an announcement for Meeting C++ 2021. And it will be held online from November 10th to 12th.
Jason: I think it's probably worth pointing out too that NDC TechTown, which is in October, is officially planned to be an in-person conference and there hasn't been any official press release or anything that I've seen from CppCon. But the website is officially updated to say that CppCon at the end of October is going to be both virtual and in-person.
Rob: I'm sure we'll be seeing more announcements like this. And I have a feeling we'll see a lot of mixed virtual and in-person ones this year.
Rob: Well, Marshall. We we've talked a lot about the C++ ABI on some recent episodes. But one thing I'm still kind of surprised by is whenever I see a discussion on Reddit or something like that, I still see a lot of comments about people who just don't know what the ABI is. So maybe to start off the discussion, we could get your explanation of what the C++ ABI is and why it's important.
Marshall: Let's just start with, ABI stands for Application Binary Interface. And it's a way of defining how parameters are passed and how values are returned and so on. And usually that's a function of the platform or maybe the compiler on the platform. Windows has an ABI. Linux has an ABI. macOS has an ABI. Android, iOS have an ABI. And for Linux, for example, and for macOS there's this nice document called the Itanium ABI Specification, which you can Google for and find. But that's not what people want to talk about when they talk about standard libraries and ABI breaks.
That's what really the discussion that people are having right now is. "We want to be able to change things in the standard library in a way that is incompatible with existing compile code".
I gave a talk about this in March on the very late CppCon and talked about what this all means. And I spent probably 35 minutes talking about the One Definition Rule in C++. For those who are not familiar with the One Definition Rule, basically, C++ says that if there's a place where you have two different definitions of the same type or class or struct, and they differ, and there's some place that you can see both of them in your program then your program issues... the delightful acronym IFNDR, which stands for ill-formed, no diagnostic required. IFNDR means that your tool chain is allowed to put out an executable that can do anything. It's undefined behavior when you launch it.
Jason: And there's just no warnings or feedback from your tool.
Marshall: And there's a reason for that. I mean, let me run through three scenarios and I'll show you.
Suppose you have two definitions of a struct. Different layouts. One of them has three fields. One has two. They're in the same translation unit. The compiler can notice that and can give a warning or an error. And most compilers do.
Second scenario. There are different translation units, and the linker puts them together. The linker creates an executable from these two object files. It's theoretically possible that the linker could tell you that if all that information was embedded in the object file. Which is not.
Third scenario. Okay. You have two different translation units. One that gets linked into an executable. One that gets linked to a shared library. They differ. And they get passed back and forth. There's no way your linker can tell you this. There's no way that a library compiler can tell you this because the program is not assembled until you launch it. And this is why it's IFNDR - there isn't any place, any one place that you can in fact catch this until the program is launched.
So, I went around and gave a bunch of examples of ODR violations, some of which are obvious. Imagine a struct with two fields: first and second and second and first. They are different types. If you pass these back and forth, what happens? One of them says, I need the access first. It says, great. It's an offset six in this struct and it's four bytes long. And this one says, no, it's an offset zero and it's five bytes long. Yeah. There, you're going to get the wrong answers. You're going to be confused if they're different sizes. And you have an array of them or a vector of them, and you try to access the elements of the vector. They're going to be in different places and you're going to get confused. There's lots of examples of that. And there's no way to catch this.
There were some papers at the last in-person standard library meeting, or a standard committee meeting in Prague a year ago or something like that, talking about changing the ABI of the standard library. A whole bunch of things that we could do if we could change the standard library ABI and some of them are minor. Some of them could lead to major performance improvements for certain classes of programs. I mean, they're all good improvements, don't get me wrong.
But we don't want to just break every C++ application in the world. Actually, there are some people who do want to do that. I have seen people arguing that users who won't rebuild their software every three years are holding the C++ community back. Really. My easy response to that, of course, is that users have their own deadlines and own timetables. And, you know, my daughter uses a bunch of software written in C++, and you tell her she has to rebuild it. She'll say: "What rebuild? I didn't build it". You know, the assumption goes along with people who say that they should just rebuild their software every three years, that basically everything you have, you can build from source. And there are people who live in that world. And for them a stable ABI is of much less value.
Jason: I just want to clarify or ask. If the C++20 ABI was completely destroyed, why would that affect your daughter's software at all? She still has binaries with libraries that are already installed on her computer. Why would that affect her?
Marshall: What happens when she gets a system update from Apple that includes a new stair library?
Jason: I mean, I have like 18 of them from Visual Studio installed on my computer right now.
Marshall: Yeah. Apple has one.
Jason: Okay. Apple doesn't have any way of versioning their standard libraries.
Marshall: They have chosen not to do so. That's correct.
Jason: I see. Okay.
Marshall: So let me give us a very specific example about libc++. It has two different versions of standard basic_string in it. And they're ABI incompatible. And the reason for this is, after various people, including Apple, shipped libc++ for several years, some people at Google discovered that you could make changes to the way standard basic_string is laid out in memory to take advantage of cache alignment. And this was a surprisingly large win. Interestingly enough, they discovered that this one change in basic_string got them a 2% gain on their JavaScript benchmarks, which is a really big number. And so libc++ now has two subtly different versions of a basic_string unit. And they're controlled by a set of ifdef and people who ship the library can choose which version of basic_string they ship.
I can tell you that Apple has continued to ship the original one in the name of compatibility. But every time they create a new platform and define a new ABI, they switch over to the new and improved version of basic_string. The first one of those was when they introduced the 64-bit iOS devices. The second one was when they introduced the ARM-based Macs. And, you know, there are people like Chrome folks who embedded their own version of libc++ inside the Chrome binary, they turn them all on because they don't care about a stable ABI.
Jason: So just out of curiosity though. If I build libc++ and compile it right now, does it default to binary compatibility, or does it default to no undefined behaviors?
Marshall: It defaults to binary compatibility. For a couple of goals back in the day for libstdc++ on compatibility in particular, there was the goal that you should be able to build code with libc++, build code with libstdc++, link them together, and have it work. And in particular, you should be able to throw exceptions from one side and catch them on the other. But that's about the only part really there that where there's compatibility, you can't pass an std basic string, for example, from libc++ to libstdc++.
Jason: You know, like most things, right, we get stuck in our habits and I heard once that libc++ is compatible with libstdc++. I am sure there are people listening to this podcast right now that are linking both into their application, either intentionally or unintentionally, without realizing that they probably have some sort of ODR violations going on.
Marshall: Well, except that they really don't. And the reason for that is, libstdc++ puts all their symbols in namespace std, and libc++ puts them all in an in-line namespace called std::__1, right? So you have linked them together and they all have different names. Except for the exception types, which all live in namespace std. So that's how you can do that. You can mix code with both of them and link against both of them. It's fine because they, the std::basic_string will be the libstdc++ one and std::__1::basic_string will be the libc++ one. And the linker knows that they're different.
Jason: It looks like a duck, quacks like a duck, but it's not a duck in this case.
Marshall: Sometimes an ABI break is really, really subtle and it's very annoying. In C++03, we had a pair, right? It's used in an ordered map. The pair has two fields: first and second. And the copy constructor of the pair was defined in C++03 as follows:
pair::pair(const pair& rhs)
{
first = rhs.first;
second = rhs.second;
}That's the copy constructor. Done. Somebody got the idea for C++11, because we had this spiffy new language feature called =default, and we should redefine pairs' copy constructor to just say =default. And we're done. It's shorter. And it's obvious that that's what's going on. That there's no games going on here. And in fact, the compiler will generate the exact same code. It's all good, except that this opens up a new possibility that in C++11, when you say =default, some specializations of the standard pair are now trivially copyable.
Jason: Like a pair of pants.
Marshall: Right, or a pair of shorts or something like that. And some on some platforms...ahem, Itanium, a trivially copyable data type, which can fit into a register is passed as a parameter in a register instead of on the stack. And if you had a pair of short-short, and it had a non-trivial copy constructor, which is what C++03 had, it would get passed on the stack. And if you have one with a trivial copy constructor and in C++11 it gets passed into register. And if you had some code compiled with C++03 and some code compiled with C++11, they would get horribly confused because of looking on the stack for something that was in the register or vice versa.
Jason: I just found myself wondering while you're describing this problem. Why did the C++98 standard pair have a user-defined copy constructor at all?
Marshall: What should it have been then?
Jason: That should have been left out. The compiler generates the copy constructor for you if you don't define any other special member functions.
Marshall: I'm not sure that's true for C++98. But anyway, I don't know, off the top of my head. The aggregate rules keep changing. So, I don't know. But in C++11 =default is definitely the right thing. But libc++ does some work to make sure (unless you flip a particular compile time thing) that no matter what pairs are not passed, they are not trivially copyable.
Rob: You mentioned how at the Prague meeting there were a bunch of proposals mentioned. Were we able to do an ABI break? We could get these performance improvements such and such. Has there been any discussion about how ABI breaks could be handled in a safer manner? Cause you're talking about some of these bugs that are really difficult, and they only happen at runtime. Is there any way we could be better about catching these sorts of things that are not just awful runtime bugs, but that would crash your application?
Marshall: I wish, I really wish. And to me that is like the key to the whole matter. If we had such a way to do that, I think a lot of the objections to changing the ABI, - I'm going to continue to use that term, even though it's the wrong term - changing the binary layout of things in the standard library is going to be really hard. The idea of "people's programs will crash and that's their problem" is just a non-starter to me. Obviously, this can't be done at the tool chain level because the tools aren't involved when it needs to be detected. One of the suggestions that somebody has made, is to change the name mangling for things compiled with, say, C++26. But it's just a suggestion — I haven't seen anybody actually try it to see how well it would work.
Jason: I mean, it's basically the solution that the standard libraries do that you were just talking about. You're hiding it in a different symbol, basically.
Marshall: Yes. But then you've basically bifurcated the C++ community, you have old binaries and new binaries, and your people have to actually make a choice for compatibility or, you know, whatever the benefits are. And if this is going to happen, there needs to be a good solution here, and there also needs to be a lot of people involved in discussing this. People on the committee can say things and write things into the standard. But if standard library implementers don't implement it, it doesn't matter. And even better if system vendors don't ship.
Jason: I'm curious if there was like a historical change here, because I mean, there was the era before 2013 or whatever that Visual Studio broke ABI with literally every release. And GCC also used to break ABI with every release.
Marshall: Okay. The second one of those, I have some knowledge about, if you Google up GCC ABI breaks, you will discover that there's a list of like a dozen of them. Only one of them had to do with the standard library, all the rest of them...Ah, yeah, there's like five or six of them where we changed the name mangling for null pointer because we had a wrong last time, but most of them were changes to name mangling of very specific things.
But we do have an example from the libstdc++, and that comes from C++11. In C++11, the stairs committee knowingly changed the specification of basic_string to make it, so that copy-on-write strings were not actually standards compliant. They didn't come and say: "You can't implement a copy-on-write strings", but they specified it in such a way that basically you couldn't do it. You couldn't implement copy-on-write strings in a standard's conforming matter. And there were good reasons for this, right? This was right about the time I joined the standards committee. The multithreading was here in C++11. And copy-on-write strings do not play well in a multithreaded environment.
Anyway, libstdc++. The people who work on this, sat down and thought really hard about how to do this. And they implemented a second version of basic_string. It was C++11 compliant, and then they modified their compiler and did a bunch of very strange things in this narrow library. They did a lot of very strange and clever things, both in the compiler, in the library to allow people to use either one of those string implementations or both, even in the same process. And then told people, this is how you get the whole behavior, and this is how you get the new behavior. And let their users choose whether to upgrade. And this was still kind of a disaster. Okay. I still see posts on Stack Overflow where someone says: "I wrote this program on libstdc++ and it crashes all over the place". And then it turns out, oh yeah, you have this standard library that's built against the copy-on-write strings and this, and your program is built against the non-copy-on-write strings and you're passing them back and forth. The last time I saw a question like that on stack overflow was May of 2020. I know two organizations that steadfastly refused to turn on the non-copy-on-write strings.
Jason: Are they using C++17 right now or are they still using C++98 or...
Marshall: They're using a whole bunch of different versions, different. There they're working on several different systems. Some of the stuff gets compiled as C++98. Some is 03, some is 20, you know, some is 11, 17, 14. But yes. But those, those systems that they're using libstdc++, they're still using copy-on-write strings. They're planning on switching over eventually. This is bad. I mean, it's kind of down to a dull roar at this point. But this has been going on, you know, there, there have been this more or less steady stream of people finally running into this problem. For the better part of a decade.
Jason: If you don't mind, I'd like to jump back to the story. What was the conclusion of that? Does that mean on some platforms, a pair of trivial objects is not trivially copyable still?
Marshall: Yes. Like our macOS. And the reason for that is specifically to defeat the ABI break problem. In libc++, unless you set the particular ABI break macro, it inherits from an empty base class. With a non-trivial but empty copy constructor.
Jason: Right. Just to give back to that behavior.
Marshall: I want to give you a scenario that I think is interesting. And that is a thought experiment. I talked about this in my C++ now talk. Suppose Apple, the standards committee says, "we're going to make an ABI break for C++23". And it's an incompatible change. And Apple says, okay, fine, we'll ship that. And we go to somebody who is a graphic artist, they use Photoshop every day. And they're working long, and they get the notice that there's a new version of macOS 11.3, whatever it is. macOS Weed, since Apple always uses it at their announcements. And then they say, okay, well fine, I'll go upgrade this because it has a compelling list of features that I want to use. And there's a note there that says, oh yeah, you know, hey, we made some changes to the C++ standard library, you'll need to update all your programs. Okay, fine. They update their system and then they go to Adobe and say, I need a new version of Photoshop for this. And Adobe is right on top of it, it says absolutely, because you have a subscription. Here's the new version. We're all good.
And they open up one of their Photoshop files to start working on it. If they're really lucky, none of their plugins will load. If they're moderately unlucky, Photoshop crashes, because it's trying to load all the plugins that this person uses. And they have the old ABI. If they're really unlucky, Photoshop will compute the work just fine. And eventually, they'll do something, and it will crash or corrupt their document or something.
And they say: "Oh, I have to upgrade all my plugins. Great. How many do I have? About 40?" I checked with people at Adobe and that's a perfectly reasonable number of plugins for people who use Photoshop every day. 40 plugins from, say, 15 different vendors. Okay, I have to go contact all 15 of these vendors and get upgrades for every single one of them. And some of them will say: "Oh, sure. Here". And some of them will say: "Yeah, I have a new version. That'll be some upgrade fee". Some of them will say: "Oh yeah, I'll probably do that. I'll put that on my to-do list". Some of them will say nothing because they don't answer. A surprising number of Photoshop actions, Photoshop, plugins come as a result of somebody's master's thesis. But that's not an experience that Apple wants. That's not an experience that Adobe wants. That's not the experience I want either.
We don't want people thinking that software written is C++ is inherently unstable and it's labeled to break anytime you change anything on your system.
Rob: Adobe comes out with new versions. So, if they come out with a new version, they might be making API changes or adding new APIs. Could that be the time to upgrade to the latest change in the ABI?
Marshall: It could be. But the question is, if Adobe has traditionally been very careful not to make incompatible changes to their plugin API, if existing plugins continue to work. Now I would like to see some way forward of evolving things in the standard library. And some of those are binary changes. Some of them are source changes and so on, there are a lot of people who have stakes in this. I hate the word 'stakeholders', but there's a lot of people who basically are between the standards committee and the users, and they all have opinions, and they all have their own motivations. And so, to get something from the committee to the users requires cooperation of those people, all of those organizations. And they all need to be onboard.
Jason: So, do you say, until we have a solution, is there any way at all to break ABI in the standard library to move forward? Like would you say no, there's no option here until we have a good solution in place, or did you say sure. You know, in 2035 we can do it. Okay.
Marshall: It depends on the situation. It depends on the people involved or organizations involved. I mean, obviously, like I said earlier, when somebody defines a new ABI. You have a tabula rasa; you can do whatever you want. When you have a limited user base, that can be responsive to changes in an ABI. Go for it. The Linux people can do that. Although they still run into problems with things built for, say, Red Hat 6 and trying to run them on Red Hat 8 because right, you have, you know, you have pre-C++11 std:: strings say but you know, the Linux where you build stuff for a major release, you build everything from source. You can do that. For Google, for example, where famously every single build of their software is everything from scratch. A stable ABI is irrelevant, has no benefit, right? So, they can change it every single build.
Jason: Does Boost have a stable ABI?
Marshall: Well, it's a little more nuanced than that. Boost does not promise a stable ABI. In general, it has a stable ABI, unless there's a good reason to change it. So the short answer is no.
Jason: Okay. How is that different from the standard library?
Marshall: I'll give you one easy answer and that is that you can rebuild. You have the sources to Boost, you can build it. As for libc++, unless, you know how it was built at Apple, you you're going to have a kind of a detective job to figure out exactly what options were used.
Jason: You'll never find out because well, it's Apple.
Marshall: You can do it by inspection. And you can eliminate many of them off the tick.
Jason: I can still choose to use the older version of the library as long as I want to.
Marshall: Yes, you can. You can build old versions of library. And if you get your standard library from your system vendor, that's what you're going to use. When libc++ was new, people were like, oh, look, I can replace the standard library implementation on my Mac with something that has newer features. And Howard wrote a nice article about it, basically saying, yeah, that's a great way to make your Mac not loading. Replace the standard library with something you just built. If it's exactly the same – great. But if it's exactly the same, why are you replacing it? And if it's different, you know, have you investigated all the places that use the standard library in macOS and determined that your change isn't going to break it? I wish we had a way to evolve the standard library that was better than the Java way, which is basically "give things new names". That's the only one that I can think of off the top of my head, that it isn't just "change things and if stuff crashes – it's not my fault".
Jason: I mean, lots of libraries do that in general. They decide they're going to make a major break. They'll change not just the version number; they'll change the library name entirely.
Marshall: Yes. That's one way to do it. I know, Apple has spent a lot of effort over the years, shipping various things, they call fat binaries that contains the versions of an object code. And I suspect there's a solution there, but now that's a germ of an idea. That's not a solution. Right. And you know, there's a proposal called C++ Epochs. That looks like it would also solve this, but again, at the cost of basically fracturing the C++ community, this would fracture it like six ways: 98, 03, 11, 14, 17, 20. You know, when you build something with C ++17, it lives in an Epochs, and it only links against code that's built with C++17. If you need a shared library, and you had code that was built with C++11 and 14 and 17 and 20 - you need four copies of the library. This space is cheap, but it's not that cheap. Everybody's moving to SSDs and those 16 terabyte SSDs are still too expensive. I am sympathetic to the idea of improving things in the standard library, I like to do that. But the idea of just changing the behavior of things or the layout of things and saying "ah, if it crashes, it's your fault"- I am very much opposed to it.
The idea that you can just rebuild your software is pretty much a non-starter for everybody who doesn't live in the open-source world.
Rob: Well, I appreciate you bringing that perspective and I certainly think there's some things you went into that we haven't really talked about in our past discussions on ABI. We need to figure out some way to evolve. I'm just not sure what that's going to be, but hopefully the standards members are all thinking about it and trying to figure something out.
Marshall: I don't have any good ideas either. I mean I have a couple of suggestions, which could evolve into proposals at some point, but they are just suggestions. You know, maybe we should look over there and think about doing things this way, but that's not a solution. That's an idea — maybe this would work.
Rob: It does sound like we do need some sort of a standard proposal that will solve this problem. Like the standard has to define something that will handle this.
Rob: Thanks so much for listening in, as we chat about C++, we'd love to hear what you think of the podcast. Please let us know if we are discussing the stuff you're interested in, or if you have a suggestion for a topic. We'd love to hear about that. You can email all your thoughts to feedback@cppcast.com. We'd also appreciate it if you can like and follow CppCast on Twitter. You can also follow me @robwirving and Jason @lefticus on Twitter. We'd also like to thank all our patrons who help support the show through Patreon. If you'd like to support us on Patreon, you can do so at patreon.com/cppcast. And of course, you can find all that info in the show notes on the podcast website at cppcast.com. Theme music for this episode was provided by podcastthemes.com.
Podcast
News
Links
Sponsors