The Hacker Mind Podcast: Fuzzing Message Brokers

Fuzzing makes it possible to locate vulnerabilities even in “safe” environments like Erlang, a language designed for high availability and robust services.

Jonathan Knudsen from Synopsys joins The Hacker Mind to discuss his presentation at SecTor 2021 on fuzzing message brokers such as RabbitMQ and VerneMQ, both written in Erlang, demonstrating that any type of software in any environment can still be vulnerable. 

As I produce this episode, there's a dangerous new vulnerability known informally as Log4Shell, it’s a flaw in an open source Java logging library developed by the Apache Foundation and, in the hands of a malicious actor, could allow for remote code injection.  The flaw was discovered and reported directly to the administrators of Log4j,  widely used open source tool, who then patched it. The trouble is, details of this leaked prematurely.  Log4j is used in many applications or is present, with dependencies in enterprise applications as well as numerous cloud services, all of which makes updating all the possible uses for it hard, even if coordinated in advance. So it’s a race against time to patch every affected system before the bad actors find a way to exploit it for their purposes. It some ways it is similar to other high-profile vulnerabilities such as Heartbleed and Shellshock in that we’re going to be finding instances ot it all over the place and we will be likely to hear about Log4Shell exploits in vulnerable products for the near future. 

The solution is to patch Log4j immediately with the latest update, and while you’re at it maybe look at updating all your open source software as well. 

While this episode of The Hacker Mind isn’t specifically about Log4Shell, it is about finding flaws in open source tools like Log4j that we use, that are perhaps more ubiquitous than most of us even realize. It’s about a researcher, his fuzzer, and a mission to recreate a known vulnerability in the MQTT protocol, and what resulted in him finding several vulnerabilities in other related open source projects. 

[Music]

Vamosi: Welcome to The Hacker Mind, an original podcast from ForAllSecure. It’s about challenging our expectations about the people who hack for a living. 

I’m Robert Vamosi, and in this episode I’m going to be talking about fuzzing erlang, a programming language developed 35 years ago as a fast network system for the communications company Ericsson, and how brokers associated with it have some vulnerabilities, and how these vulnerability wouldn’t have been found unless someone hadn’t explored them.  

[Music]

Vamosi: The idea behind Open Source is great. Why code something that has already been coded? As a developer you can go to GitHub, see the license agreements, and then add the code to your next build. The problem comes with updating all that code over time. The longer you go without updates, the more likely someone has found a vulnerability-- either reported or unreported. And then there’s the additional problem of maintain thal, someone acting as the administrator for each open source project. And there’s the researchers, the one that come along and find things.

Knudsen: My name is Jonathan Knutson, and I am a security researcher at synopsis.

Vamosi: Jonathan is one of the good guys. Before Synopsys, he worked at  Codenomicon, the company that found Heartbleed back in the day. So as a researcher, Jonathan is used to finding vulnerabilities. For that he uses a technique known as fuzz testing. 

Knudsen: So in a nutshell fuzzing is delivering badly formed or unexpected inputs to a software to any piece of software and looking to see if something fails, looking to see if something breaks, I can go into great more detail if you would like a little bit more detail would be good. Okay, if you think about normal software testing, you give reasonable or valid input to a piece of software, and then you check and see if the right thing happened. So you're testing for functionality. If I push the button does the light come on. If I give it a name does it give me the right output and fuzzing inverts that, so it's. Let's give the software badly formed or invalid inputs. Essentially, garbage or noise. And let's see if the software is able to handle that or fails in some way. and if it fails in some way, then you know you found some kind of a bug, and from an attacker viewpoint, maybe it could be exploited. From a defense viewpoint. You found a bug and now you can fix it.

Vamosi: So fuzzing injects malformed data and then monitors the result. One could argue that what it really comes down to is proper error handling within the target software. We spend a lot of time testing for how the software should work, for performance, for reliability. Perhaps we should spend equal --if not more time-- testing how the software handles unexpected input. 

Knudsen: Absolutely. So a lot of it has to do with just how programmers naturally work which is they're trying to build something that functions, they have a list of things that it needs to do, and they try to write code to meet each of the things on that list. But along the way they tend to make assumptions like, oh I assume that the user is going to do something that makes sense, or I assume that this other system I'm talking to is going to give me correctly foreign messages or or or messages in the right order or messages that make sense at all. And so it's easy to make those assumptions. And if you are just doing normal functional software testing. You're also looking at sort of the happy path. So you're looking at, you know, if we give it this good input does it do the right thing with it, which is exactly how the programmer programs in the first place and so even if you get all of those things pass. You put this thing out in the real world and and strange stuff happens and it likely will fail for you. So fuzzing is a way during software development where you sort of proactively. Give it, stuff that's hard to deal with, and then if it breaks, you can fix it, and it's more robust and more secure by the time you actually release it.

Vamosi: Fuzz testing is a powerful tool -- even the bad actors have started to use the open source versions out there. Jonathan uses a commercial fuzzer that fuzzes protocols that his company maintains. And, as a researcher, he can aim that fuzzer at any protocol target that he chooses.

Knudsen:  That was my dream, I, I use the Defensics to fuzz Bitcoin, a little bit. So Bitcoin has its own network protocol. And I thought, well, wouldn't that be like the ultimate attack right because it's this big peer to peer network, And they're all talking to each other with the Bitcoin protocol so if you could get remote code execution, exploit, you could just spread throughout the entire network, almost instantaneously, but it didn't work out. Yeah, I mean, I think it could have like part of the challenge was just modeling the protocol and it's a pretty complicated protocol and I, you know, I just wanted to write an article about it. I didn't get too deep into it. Right. But it was, it was fun for sure.

[MUSIC]

Vamosi: To better understand this episode, it’s perhaps important that we agree on the use of the word broker. A broker, simply, is someone or some thing that negotiates between two parties. So in stock, you can hire a broker who acts on your behalf to buy and sell stocks in your best intersts. In software security, a broker is something that gathers and reports information. 

Knudsen:  I described a Message Broker as slack for robots. So it's essentially, it's a piece of software that supports publishers and subscribers, and it contains internally, a list of message queues, which are kind of like channels and slack. So, you would use it in a large software system consisting of many components, and it would enable them to talk to each other in a orderly and reliable way. So publisher would be a piece of software that had some data it wanted other people to know about other processes, and so it could publish messages into a Message Broker and then subscribers could pull them out as needed. 

Vamosi: Given that, how would one use a broker? If you want to control data feeds. For example, the number of registrations in any system.  When the task is to send data to several applications and avoid direct use of their API. When you need to complete processes in a defined order, like a transactional system.

Knudsen: Where would you see it. I guess one example that is easy to understand would be, let's say you're a power company and you've got 10,000 smart meters on people's houses out in the field. You could imagine all of those being publishers and and talking to a Message Broker and saying, Oh, this house use this much electricity. And then, in the offices of the power company, they could have subscribers that are picking up those messages from the message queues and doing whatever they do with them. I think I don't know exactly where message brokers are used in detail but they seem pretty important, and, and you can see in your head that it's sort of like a central nervous system of a large system.

Vamosi:  Jonathan was looking at Mosquitto, a lightweight open source message broker written in C that Implements the MQTT protocol.  In particular Jonathan wanted to see if he could find a known vulnerability in Mosquito with the new feature in his fuzzing solution.

Knudsen:  So I have to explain a couple things about that first. So instrumentation is for after you've delivered a bad input to the software -- how do you know if something went wrong. So instrumentation is how what we call that. And in that context, it means, you're looking to see if a process crashed or you got a kernel panic or all the memory filled up or you know any of the things that can go wrong with software. You have to decide what you're going to check for and how you're going to check for it when you're fuzzing. 

Vamosi: to find memory leakage, you’ll want to use what’s known as a Address Santizer or Asan. It’s an open source programming tool that detects memory corruption bugs such as dangling pointers or buffer overflows. 

Knudsen: So, we had a new feature in the agent instrumentation where it would use Address Sanitizer, as a way of looking for memory leaks. So Address Sanitizer is a project that's now built into si Lang, I think. I don't know. Anyway, you can you can build a piece of software with Address Sanitizer enabled and then when you run it, if that piece of software handles memory badly. It'll tell you about it. So we had this part of defense six, that would work with Address Sanitizer to find out about bad things happening with memory, and I kind of wanted to see how it worked. And I had heard that somebody used this feature to find a bug in mosquito, 

Vamosi: MQTT is a standard messaging protocol for the Internet of Things (IoT) , a publish-subscribe network protocol that transports messages between devices  In particular, he was looking at Mosquitto, a lightweight open source message broker written in C that Implements the MQTT protocol.  Jonathan wanted to see if he could find a known vulnerability in Mosquito, but right away he encountered a few challenges..

Knudsen:  And so, the steps I needed to do were get the old version of mosquito source code, build it with Address Sanitizer enabled, run it, and then I can run against it, and reproduce this bug. At the time I was working with, with containers as a way of making a reliable way of reproducing software environments, and, and getting things to work. So I made a container that got the source code from mosquito and built it and so on so forth and ran it. And after I was done, so that all happened, and I was able to reproduce this 2017 bug and mosquito. And I wrote an article about it, and, and that was fine. 

Vamosi:  Containers are a solution to the problem of how to get software to run reliably when moving it from one computing environment to another. Contains are a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another.

Knudsen:  And then after that I'm like wow that's, that's pretty cool how I was able to use a container to hold my test target mosquito, for two reasons. The one reason was, it's very, very reproducible. So once you get your Docker container set up the right way. You can, you can just build it and it happens the same way every time. And, you know all those steps that you had to do to get the source code and configure it and build it with Address Sanitizer and all that stuff is all encapsulated in a Docker file, so it's really easy to reproduce. 

Vamosi: A Docker container then is an image that is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.

Knudsen: What was cool was that containers are so much lighter than virtual machines. 

Vamosi: Virtual Machines are what just that; they’re software representations of hardware machines. They contain an operating system and they have confirmation for RAM and other hardware you may want to simulate. This is how malware, for example, is analyzed; if the virtual machine gets infected and crashes, the physical machine isn’t impacted. In fact, you can just spin up another VM.

Knudsen:Normally I would use virtual machines for this, I go install Linux and install the other stuff and work away at the command line until I got to where I wanted it to be. But then, if something got messed up, and I wanted to start over from the beginning it was a slow process. So containers, really helped with that as well. 

Vamosi: So that is a problem. With an OS and hardware requirements, a VM can be slow to spin up, and can tax your physical system if you have more than one VM running. 

Knudsen: Right so, so, in, under normal circumstances like if you just write an application and see it gets compiled to processor instructions machine instructions, and there's an operating system that knows how to run your program, and it's just running instructions directly on the processor. I would say it's closer to the metal with a virtual machine. It's sort of a made up processor that runs in software, above the real processor. And so, for example in Java, you write Java source code you compile it, it ends up as Java byte code which is essentially machine instructions for the made up processor of the Java Virtual Machine. So, it's just like an extra layer .y it's closer to the metal with a virtual machine. It's sort of a made up processor that runs in software, above the real processor. And so, for example in Java, you write Java source code you compile it, it ends up as Java byte code which is essentially machine instructions for the made up processor of the Java Virtual Machine. So, it's just like an extra layer .

Vamosi: So a better solution is to test on a container.

Knudsen: So I reproduce this bug and then I did more fuzzing on mosquito just to see if I could find any bugs and I couldn't. And then I thought well, there are other message brokers out there, why don't I go try some of those and see what happens. So that's how this started.

[MUSIC]

Vamosi: Given his success with Mosquito, his learnings from GitHub, Jonathan started looking at other message brokers.

Knudsen:  So I went to GitHub, I looked for message brokers, pick the ones with the most stars or the most forks, and got those running in containers in the same way.  So I was focusing on MQ TT. Network protocol I think it’s been around for a while.But then I think the most popular Message Broker on GitHub is rabid M q. And I can't remember. I don't know, maybe it needs a plug in for MQ TT or something but anyway it seems to do another protocol called am Kewpie better or more natively I think there's a plugin for it that I had enabled but. So that was the other protocol that I found, but both MQTT and SMTP have test suites and defenses so it was easy to just run the appropriate test suite and get it going.

Vamosi: Using the GitHub containers that are already created, that saved him time and effort. So Jonathan’s now got his fuzzing tool up and running, but wait … something wasn’t right. 

Knudsen: somewhere along the way. I was getting failures but it was kind of, was kind of slow, but they were out of memory failures. So I would do some fuzzing and somehow the Message Broker would get into a state where it would gobble up all of the memory on my laptop and then it would swap the disk for a while and thrash around for a while, and eventually the operating system, kill it because it was using too much memory. And then I realized that, since they're already running in containers, I can just constrain the memory on the container, so I can just say okay, you get a half a gig, and no swap space. So once it uses up all the memory it just fails immediately. And that's essentially what happened with all three of these message brokers. 

[music]

Vamosi: Maybe just using containers from gitHub wasn’t that easy. So Jonathan returned to tinkering and hacking these message brokers.

Knudsen:  The hardest part really was getting these things running so I did a combination of things with the containerization. So, in some cases I would just go and some of these projects have like official container images up on Docker Hub and that makes it really easy to just go grab that and run it. Some of them I wanted to build from source. I don't remember exactly why. Maybe I was, it was is because as because with MQ with mosquito. I wanted to include Address Sanitizer and in order to do that you have to rebuild the software. With the flag for Address Sanitizer or whatever because it inserts an extra stuff in there. And so my original plan was, I was going to do exactly that same thing with each other message brokers, but it turns out that all three of the message brokers i i broke rabbit MQ MQ X, and Vern MQ, they're all implemented in Erlang, and I had never really heard of erlang before. I thought, well, that's fine, you know, whatever it is, I can insert Address Sanitizer into it and boy was I wrong.

Vamosi: Erlang is a programming language that used by Ericsson and others to build massively scalable soft real-time systems with requirements on high availability. Some of its uses are in telecoms, banking, e-commerce, computer telephony and instant messaging. Erlang's runtime system has built-in support for concurrency, distribution and fault tolerance, but it was also first built in 1986.

Knudsen:  So Erlang, I think was originally developed by Ericsson maybe like in the 80s it's just outrageously old I had never heard of it. And it's, it's specifically designed for like high availability robust servers services. And so, I think it makes, well obviously it makes sense for message brokers, but it's kind of VM architecture like Java does so, you know, there's your application and there's some VM and then there's the OS below that  There are some mistakes you can't make an Erlang. So everyone knows C is dangerous right you've got pointers you've got buffer overruns and all this stuff. And so, some environments, you know, like there's some mistakes you can't make in Java, you can't have a stack buffer overrun. 

Vamosi: What Jonathan’s talking about is that there are memory safe languages such as Java  versus non memory safe languages such as C. What that means is that the language will stop you from putting something that might corrupt the memory. So it’s considered safe..

Knudsen: I mean, I'm putting it in air quotes which you can't see in the podcast because it. You know it's not really that safe, so everyone knows and see that it's trivial to write code that writes fast the end of the buffer or your reading or your writing memory that doesn't belong to you in an environment like Java, you can't do that. So if you write past the end of a buffer you the runtime system gives you index out of bounds exception instead. And so, you can't. You can't have a buffer overrun. Which means, unless the Java system itself has a bug. You can't exploit a Java application to do an exploit the way you would with a C application and Erlang also has a virtual machine architecture and so it's, it's sort of got this extra layer of insulation between bad things happening in the application, and actually enabling an attacker to run arbitrary code on on the base OH S.

Vamosi: But erLang was far from perfect as far as languages go.

Knudsen: There are some mistakes you can't make an Erlang. Likewise, but one of the things you can do is eat up all your available memory. And so for these message brokers, running in containers made sense because I could constrain the memory and really highlight when they ran out of memory.

Vamosi: Which is why he wanted to put in some address sanitizers.

Knudsen: And. Address Sanitizer is really designed to go into a C application. So you can compile it in. When you're building a C application. And so I think it might theoretically be possible to build the Erlang VM with Address Sanitizer enabled but anyway I got nowhere with it. I was able to build some of these from source, even as Erlang applications but that was pretty complicated as well so some of my container, Docker files were pretty gnarly for that. But part of it is part of the talk is that Erlang is one of these environments that sort of considered safe. So I think part of the point of this research was that sure it's written in Erlang and some things that can't go wrong, but there are still plenty of things that can go wrong. And so, with the fuzzing we were sending badly formed network protocol messages, and each one of these message brokers was somehow not handling it well in such a way that they consumed all their available memory and got killed

Vamosi:  What were some of the vulnerabilities that Jonathan identified in his talk at SecTor? What were some of the commonalities among them?

Knudsen: I guess if you had to sum it up, you would you would say it was like, improper input validation. So, again going back to what we were talking about the beginning. Developers sort of assume that the stuff they're getting is going to conform to what it's supposed to be. And, and if it doesn't, they might not have thought about that and they might not have coded for it. And this is a general problem across any language, you know any software is that you're, you're getting input and you probably can't trust it. And so you need to look at it carefully and make sure you're handling it properly and if it's not right, you need to take appropriate action either ignore it, or send back an error, or whatever that you do. 

Vamosi: Comparably, though, erlang is one of the safer languages in use.

Knudsen:  But, but it can still fail. And that's why the safest in air quotes, is that if you think of Java being safe as compared to C, it just means that you can't have a buffer overrun. You can't have it fail that way. But there's, there's so many ways that software can fail so that there's process crashes, but there's also, you can go into an end of endless loop, or you can write corrupted data into your database, or you can lose functionality that should be working or, or the thing can just lock up entirely and it's not available anymore. 

Vamosi: Some of that is by design. 

Knudsen: Erlang has a philosophy called   fast, where it's, you sort of expect things to fail. Normally when you're writing like a network protocol parser in C or some other language you're trying to think of every possible thing that could happen, or go wrong and you're trying to write code to handle every possible eventuality, and you're always going to miss something and you know, so early and kind of inverts that where you write code for kind of what you're expecting to happen, and then you expect it to fail. Otherwise, and then there's a layer of like supervisors that are responsible for seeing when things fail and relaunching them and, and all this stuff. And so it for a network service, it makes it very robust. 

[music]

Vamosi:  With the internet of Things, it seems that we just keep bringing old protocols like MQTT and languages like erlang forward because we have new uses for them but meantime they weren't designed for those things.

Knudsen: that's, that's especially true in industrial control systems, where the protocols, they are they're just all unauthenticated and they're all plaintext and you know they were, they, they come from an environment where you think you can trust everything that's on your network and now they're extended to IP networking and then there’s trouble.

Vamosi:  When testing things like HTTP, it comes out pretty well; in other words, when testing HTTP, it took a really long time to find a first crash. That's HTTP, we use it everywhere, and then things like to pick on MQ TT. I remember in one fuzzing report that I wrote it seemed like it crashed after a couple minutes of fuzzing because it was less mature less looked at.

Knudsen: Yeah, and again it depends on, not the protocol so much as the individual software so, you know, somebody could write a new HTTP server. Next week, and you could fuzz it and knock it over right away because it's, it's so new and hasn't been around for a while so So, some of it has to do with how long it's been around like, if you're going to go fuzz apache or nginx or, you know, one of the big ones that's hold and grizzled. It's going to be harder to find stuff. And I, I maintain that that's because it has been first, just by being out in the real world for so long. So you, you go and you write a piece of software and you put it out there it's gonna encounter malformed weird inputs, either by accident or by malice. And so, projects like those that been around for a long time is, is likely sort of organically fuzz and has fixed a lot of that stuff and so it's going to be more robust.

Vamosi: So the, the vulnerabilities that you discover Do you think it's because Erlang was first being used, it was spoon fed the inputs and now it's open to a lot more inputs, hence the vulnerabilities that you tripped upon.

Knudsen:  No, and I don't think it's anything about Erling in particular. I think the message brokers that I tested happened to be all implemented in Erlang because of it. I think it is a good choice for, like, network services that need to be robust, but I think it's more a reflection on. Nobody had done any network protocol fuzzing on these things before. So, you know, presumably each of these Message Broker projects has a set of internal tests that they run, but most likely those are focused on functionality and fuzzing is complementary to that, you know, the functional testing is great, let's make sure it works the way it's supposed to work, but you also need the fuzzing which is. Let's make sure it doesn't fail when we throw weird stuff at

Vamosi: it. So these brokers could have been written in other languages and still had the vulnerabilities that you found and it was more of attention that nobody had really tested these before.

Knudsen: right or not this particular type of testing. So, you know one thing we've learned about security testing is that you can't just get one thing and be done.

{MUSIC]

Vamosi: Over many years of fuzzing like it's an it's an in group it's a small group of us who actually know about it and understand it, why, why is that

Knudsen: it is. I always interview a guy to join a synopsis. About a year ago, and he came from. Well, one of our competitors, I don't remember exactly which one, and, and it was, it was just such fun for me to, to sit with this guy and talk for an hour about fuzzing because he was excited about it, I was excited about it and like you say there's just not that many people in the world who really know it at a deep level. So, with a lot of fun. You know, why don't more people. The other thing I'll tell you is, I joined Codenomicon in 2011. And I didn't really know, I didn't really know anything about fuzzing. When I joined. So, you know, hats off to Maiko for recognizing potential. And so my first week on the job they flew me out to California and they sat me down and they showed me talked about fuzzing and showed me the Sensex and showed how it broke some things. And I was, I was, what's the word ecstatic because I thought, We're gonna be rich. Everyone needs to do this, and I was right about everyone needs to do it, but it's, it's, it's a little complicated, I guess it's, it's a little hard, and so not everybody realizes or not everybody's you know at that level and in the security journey. So it's a little hard, it's some. I don't know quite why but it's sort of over the line, I guess,

Vamosi: That has been the perception over the years, that fuzzing is too hard. But advancements have been made. 

Knudsen:  I guess people want simpler tools like they want to, you know, I think they want security to be somebody else's problem. And I think it was in like the first generation of application security. So, so the first gen was like companies would say, Okay, well we've heard about this cybersecurity thing, and starting to look like it's important, so we'll do what companies do and we'll make a group for it, and we'll put somebody in charge of it, and there'll be responsible for it. And you can't, you can't do that with security, it's like if you were manufacturing airplanes, and you're like, Okay, you guys figure out how to build the airplane and build it. And then we'll have this other separate group that's responsible for keeping it safe. that way. And so I suppose the same way. So, I think. I think it's a little harder to use and some other security tools and so if you're just starting out, you're probably going to pick something else to begin with. And then once you get to a certain level of maturity you might pull in fuzzing.

Vamosi:  I guess I'm also still stuck on this idea of where are other Heartbleeds and shellshocks, both found in 2014. Until this year’s Log4Shell, I'm kind of surprised that we haven't had more incidents like that.

Knudsen:  Yeah, I mean I think it's, it's easy to sort of construct doomsday scenarios like well you know like you see in a TV show like oh we got this magic code that shuts down all electronic devices and whatever. And some of that reminds me I'm old enough to remember, y2k, and all the, like doomsday scenarios for that. And I think. And so, I would say doomsday scenarios probably not realistic. And part of the reason is just that there are so many different kinds of software running everywhere. And if you want to compromise something it's usually a very specific version of a specific piece of software. So, so that's one of the reasons Heartbleed was such a big thing was that so many of the web servers everywhere were running using this software component. Open to open SSL. And so there was some homogeneity there, and most of the time that's not true. So for example, that these three bugs I found were in three different pieces of software. The root cause was essentially similar like not handling input correctly, but the exact way to trigger that was completely different for each of them so I just found similar kinds of bugs because I was doing testing in a similar kind of way, right,

Vamosi: Why no new Heartbleeds or ShellShock. They both were found in 2014. Why haven’t we heard anything since on that same level, I would think if there's a lot of broken code out there that fuzzing would be saving the world.

Knudsen:  Heartbleed was really a perfect storm in a lot of different ways. So one way was, it was an open SSL which was an open source component that was used in like two thirds of the world servers at the time. So, so that was, I mean that was one thing like where the bug was, it was a very important software component, and another thing was just the nature of the bug was that it would just spew out server memory to anyone you didn't even have to be authenticated you didn't even have to be speaking encrypted stuff and then the TLS conversation. So it was really easy to exploit, and it was really, really dangerous and it was really really widespread. And, I guess, you know, people have done follow up since then, like, let's, let's survey the internet and see how much of it is still vulnerable to Heartbleed and that's pretty discouraging. You know, clearly there's a lot of stuff out there that nobody ever updates, why hasn't there been something that big, you know, I think there have been plenty of big ones, they just didn't quite reach that scale. And also I think there's a certain amount of fatigue going on like, oh, you know, another day another unsecured database or another bug or another compromise or whatever. And so, I think. I think it's hard to keep being interested in it, keep them waiting, it's hard to, to keep paying attention. On the other hand, there have been some incidents like I'm trying to remember when Maersk was and it might have been that long ago but, but you know Colonial Pipeline right. That was a big thing and lots of hospital systems being hit with ransomware and and so on and, and usually the roots of those are, are, are user errors like somebody opens up an email that they shouldn't have or something like that. But sometimes that has to do with with software vulnerabilities and so I don't know I, 

Vamosi: As I mentioned, Jonathan’s been looking at software vulnerabilities for more than a decade. I would imagine it is hard not to get too cynical about things. Jonathan is not.

Knudsen: I try to be optimistic like my other title is technical evangelist, and so I try to. I keep saying, oh, you know, as an industry we're realizing that software security is really important, and I figure if I keep saying it maybe it's going to become true. And, you know, everybody's at a different level in the journey so. So for some people that they, they get it, and they're, they're really building security into the way that they create software and consume it, but it's, it's a continual struggle.

I’d like to thank Jonathan Knudsen for talking about his presentation at SecTor 2021. And I’d also like to thank the thousands of people who are currently supporting open source software. It’s a hard job as the software ages, as the software get used in new and different ways from which it was originally created. Fortunately, because of research such as Jonathan’s, we’re finding those new vulnerabilities perhaps before the bad actors do. Again, it’s the responsiveness of these administrators of open source software that is keeping the internet safe. 

Let's keep the conversation going. DM me @Robert Vamosi on Twitter or join me on Reddit or discord. The deets are available at The Hacker Mind 

The Hacker Mind is brought to you every two weeks commercial free by ForAllSecure. 

For The Hacker Mind, I remain the always fuzzing Robert Vamosi