Using Mayhem for API to Fuzz allthebots.com

I recently wrote about using Mayhem for API to fuzz etcd server. In spite of being unfamiliar with etcd, I was able to find and fix a couple of crashing issues.

I thought it’d be interesting to see what Mayhem produces in the hands of someone who knows the target API in and out. And as it happens, I have a little hobby web app of my own, over at allthebots.com, which I know extremely well indeed (having written the entire thing myself!)

Spoilers: bugs were found. Lots of bugs.

Dive In

Time to get started. Why not just point Mayhem for API at my API spec and see what happens?

$ mapi run allthebots 30 ./allthebots.yaml \
--url https://internal_api/

At this point, Mayhem can only probe unauthenticated endpoints (of which there are exactly two!) It’s also only running for 30 seconds. Even so, it immediately finds several issues.

🚨 missing default exception handler

In the first run, Mayhem reported a reproducible server crash. I wasn't surprised that there were exceptions happening, but I was very surprised that they were causing a full-blown crash instead of a 500 Internal Server Error.

On top of being awful user experience, this a major reliability issue and even a potential security issue, in that a process restart is sufficiently costly that it makes a very juicy denial of service target.

It turns out that the extremely minimal API framework (hug) I use for allthebots didn't come with a default exception handler to turn all unhandled exceptions into HTTP 500 returns. Adding a default exception handler stopped the server crashes.

🤷 misconfigured keepalives

Also in the first run, Mayhem reported a number of non-reproducible server crashes. Luckily, the issue description included a hint about keepalive behavior possibly causing false positives. I did some googling, and quickly found that I needed to restart uwsgi with the  --http-keepaliveflag, to make it comply with the standard.

This was a real misconfiguration, but it wasn't user-facing. In fuzzing, I pointed Mayhem directly at uwsgi. In a real deployment, uwsgi is behind nginx, which has its own (compliant) keepalive implementation.

🐞 input validation

In my second run the same

mapi

command, but with the first two issues fixed) Mayhem found a handful of internal server errors… the ones which had previously shown up as server crashes.

These first couple of bugs were just egregious failure to validate input, in particular when given string inputs containing NULL bytes. The details aren't very interesting.

Make API Testing Ridiculously Easy.

Find out how ForAllSecure can ensure the quality of your APIs with autonomous fuzz testing.

Free Trial Learn More

Configure Authentication

Now it's time to fuzz the authenticated endpoints. For allthebots, that looks like:

$ mapi run allthebots 30 ./allthebots.yaml \
--url https://internal_api/ \
--header-auth "Authorization: Bearer ${TOKEN}"

Once again, I'm only fuzzing for 30 seconds; I’m looking for the easy stuff. And once again, Mayhem delivers:

🚨 extremely slow endpoint when there's non-trivial data

This was alarming: in 30 seconds, Mayhem successfully created enough of one resource type to find a serious (worse than linear) performance issue in the endpoint that enumerates them all!

This was probably the most serious user-facing issue Mayhem uncovered, and I'm extremely glad that it was Mayhem, not a user, who found it!

🐞 more input validation

The payloads of the authenticated endpoints are more complex, and so are the places where I've failed to validate my assumptions about those payloads. For example, Mayhem has reminded me (for the Nth time of my career) that not all valid json payloads are objects!

CI Integration

I love the idea of adding Mayhem to my build pipeline, because I already believe in keeping my API free of 500 errors, and I’ve seen what sort of issues Mayhem can uncover. So with the most blatant issues resolved, I’ve done that, giving it just 20 seconds per commit of fuzzing time.

It still occasionally finds things. Most recently:

🤷 semi-spurious "authentication bypass"

In a recent run, Mayhem reported an authentication bypass bug. This one was a real bug, although not a real authentication bypass: my API was treating 'Authentication: Bearer TOKEN' and 'Authentication: TOKEN' identically (because of lazy use of python's

removeprefix

method!

Even though this is a bit of a silly bug, I did fix it, to keep Mayhem free of false positives.

🐞 even more input validation

Mayhem tries all kinds of bizarre inputs, and finds all kinds of validation problems. None of them have been particularly interesting yet. But it impresses me that after I fixed the first ten or so issues, Mayhem is still uncovering bad behavior.

Takeaway

In a few hours using Mayhem for API on my hobby project, I was able to find (and fix) a very serious performance issue, a moderately serious security issue, an obscure configuration problem, and a dozen or more input validation defects of various degrees of severity.

I’m strongly on board with keeping my API free of 500 errors. Because fuzzing is random, having Mayhem in my CI pipeline means occasionally having a build fail when it hits an unrelated error; this will stop being an issue in time.

In the end, I’m happy with what I’ve gained from integrating Mayhem for API into my work: an enormous amount of “free” test coverage, and confidence that my service is well-behaved under all kinds of conditions.

What about you? If you work on an API, what kind of results do you think you’d get from fuzzing it with Mayhem? Why not give it a try, and let us know how it goes?