FuzzCon TV Tackles Federal Fuzz Testing

Continuing the discussions started at our successful FuzzCon event held earlier this year, ForAllSecure is hosting a series of follow-up sessions online called FuzzCon TV (formerly A Fuzzing Affair). Our second episode is hosted by Matt Venditto, VP for Federal Sales at ForAllSecure, and covers topics related to federal software systems. Guests include Jonathan Butts, Founder at QED Secure Solutions; Joe Jarzombek, Director for Government, Aerospace & Defense Programs at Synopsys; and Zach Walker, DIU Texas Lead at Defense Innovation Unit. A recorded version of this second episode can be found here

Joe Jarzombek started by discussing how he sees the adoption of fuzzing in the federal space today.  “I will say that the fundamental nature of fuzzing has changed -- and actually even on how we use it -- because 30 years ago we were using it primarily for reliability and resilience testing, and it was only subsequent to that that we started focusing more on security, so it's actually been done. I will tell you back in the 90s it was actually expressed as a best practice, and many people would say, well, then, why haven't we been doing it more? Maybe some of the other guys could address that as well, but I would say part of this comes down to that you have to have a mature supplier base. Understand that because, unless the customer has been asking for fuzzing, you're probably not going to get it, necessarily.”

Jonathan Butts piggybacked on that. “Typically when we think of fuzzing or sending inputs in software, we tend to think at the application layer. And I think what we're seeing is a shift, primarily in the military and the DoD, today there's actually further extensive in-use cases. It's not just an IoT or software-based solution, or a focus on our weapons systems.”

“My favorite anecdote,” said Zach Walker, “is from the major program office in DC that I went to as the government person and found out that not a single person in that office actually had responsibility for finding previously unknown software vulnerabilities, because it wasn't in the contract in 2000. Or with the systems integrator, the person responsible for delivering, in this case, a very expensive ship. They didn’t think that they needed to do it. And because we didn't specify anything in the government contract either, we just had to sign off and agree to that.  But I think we're moving in the right direction today.”

A question from the audience asked what’s done about it once we find all these vulnerabilities through fuzzing?

“First of all,” Jarzombek said, “I'm going to be a little bit specific about how we name this because we have vulnerabilities and we have weaknesses and there's a relationship but they're different. And the reason I say that is because we also have different tooling capabilities to look for these, and once you find them. Your mitigation strategy is also different with vulnerabilities and CVEs. CVEs, that's kind of the granddaddy of them all within the Department of Defense and federal government. We use CVEs. The information assurance vulnerability alerts, that have been pushed out, are 100% based on CVEs, but somebody has selected a subset for the Department of Defense and said here's what they are. The mitigation for that is that you patch it. Now, this is normally against operational systems and if you look at a typical sysadmin or network admin, their day job is to keep things up and running. Somebody puts out a security alert, there's a CVE that comes out, and here's an associated patch. So you patch the system. What does a patch do? It changes the configuration. More specifically, it breaks the configuration. That means the guy has more work to do. It's not just a simple matter of installing a patch. And by the way, we're talking about a lot of time and effort that's associated with this. We've seen instances within the Department of Defense because they just didn't have time to do all this but they were compliance oriented. They would install the patch, take a screenshot that they installed the patch, and then they would uninstall the patch to keep the system up and running. That's an unfortunate reality. Now, the other is understanding the exploitable weaknesses. That's why we started the common weakness enumeration (CWEs). These represent source vectors. As long as they exist within your network, they are potential source vectors for somebody to exploit the system and a new CVE will be generated. In fact when it's first known about it's called a zero day vulnerability.”

“Our focus is not on the network side or your traditional IT,” said Butts, “so I have a different perspective on CVEs as applied to weapon systems and operating systems trying to be mad. We've done a lot of work, and I can speak to medical devices and things of that nature. And we actually have done examples where if you apply a CVE to a vulnerability we found where we could actually steal medicine. I once had a very high CVE because it impacted integrity, availability, and confidentiality. We had another vulnerability against an infusion pump where you could kill somebody if that was exploited, but it didn't have a loss of confidentiality so you didn't know who you killed, and so that one actually ranked lower on a CVE score. So, if somebody takes those numbers they will think you got to go fix the one that lets you steal something versus the one where you can kill somebody. So, CVEs work very well for traditional IT, it just does not port over very well into the operational side or talking about weapon systems.”

Another question asked what is the most interesting federal vulnerability you found using fuzzing that you can share?

“There's definitely potential out there to impact systems through fuzzing,” Butts said. “We've done a lot of assessments against weapon systems and aircraft and satellites and space vehicles. We almost without fail use fuzzing as an initial kickoff platform, just because it scales much quicker  we can sit there and write code. Sometimes it's a simple fuzzer we just throw some scripts together with the protocol and fuzz against it sometimes we've got to build a more elaborate script. So, I can't get into specifics of identified vulnerabilities, but I can say without fail we use fuzzing on almost all of our systems.”

“Within the defense community,” Jarzombek said, “when you identify a vulnerability against a very specific platform, it is classified. You can't go public with it and Jonathan can say that's why I can't talk about it within our company. If you think back, years ago, there's this little thing called Heartbleed. It was first reported out by using Defensics. Well, that was rather interesting because of the wide scale implications that it had. It was simply finding unpatched open SSL, you know, on web application servers that really affected the world. And that's now over six years old, but it's amazing how they're still instances of unpatched open SSL. So, this is how the community is responding. You know, it's problematic; sometimes we make it easy for the guys to exploit things. And I say we, the community, simply because it's not about the ingenuity of the attackers, it's more about the vulnerability of the exploit target.”

You can hear the entire discussion here.