How We Make It Easy to Deploy Mayhem for Code on Your Premises

Mayhem for Code can help you find and fix vulnerabilities in your code.

Mayhem for Code can run on the cloud, but, when you're testing critical/sensitive/confidential code, you can make it more difficult for malicious actors to access Mayhem's findings: you can deploy Mayhem on-prem.

In earlier days, to install Mayhem for Code on-prem, we co-ordinated with the IT staff of our customers to ensure that they have the hardware, OS, and networking set up for Mayhem.

These days, to get Mayhem for Code on your premises:

• we can provision hardware for you, if you'd like.
• we can provision OS for you, if you'd like.

In our experience, RedHat Enterprise Linux - and its derivatives like Oracle Linux - are quite welcome in the IT environment of our customers. So, we looked into bundling Oracle Linux (OL) as we distribute Mayhem. (Okay, we initially started with CentOS, but have since moved on to Oracle).

This "OS family" can be installed using DVD ISOs that include the Anaconda-installer.
Anaconda-installer can be automated using kickstart.
We can kickstart the OS and networking set-up.
We can also (after adding Mayhem bits into the ISO image) use kickstart to automatically install Mayhem for Code.

The resulting ISO image was 15 GB. 10 GB comes from the "full ISO" to install Oracle Linux (8.5 at the time of this writing), which we use as a base. This "full ISO" carries enough packages to install LibreOffice, video conferencing software, etc. While these are useful software, we don't need to install these on machines set apart for Mayhem.

Can we trim the size of the ISO image?

In the "full ISO" for OL 8.5, we see that the AppStream directory takes up 8.4 GB. If these were "normal" files & directories, we would "normally" try rm [-r]. Can we do that with ISO images?

We can, with xorriso ("X/Open Rock Ridge ISO" tool). We can tell it to -rm $PATH ... and/or -rm_r $PATH ....

At the time of this writing, we can install xorriso using yum in OL and apt-get in Debian/Ubuntu without adding additional package archives/repositories. We can also install it in macOS with Home-brew.
We can also use xorriso in a Docker container (using, for example, oraclelinux or ubuntu images), but we noticed that disk I/O performance is significantly slower than using xorriso outside the container.

A basic xorriso invocation would be:

xorriso \
-indev OracleLinux-R8-U5-x86_64-dvd.iso \
-outdev out.iso \
-boot_image any replay

This informs xorriso to use OracleLinux-R8-U5-x86_64-dvd.iso as "input drive" and sets "output drive" to out.iso.

• -boot_image any replay tells xorriso to also copy the bits that make the "full ISO" boot-able.

If you're following along, at this point, we should have an out.iso that is basically equal to the input.

The Needful

Before telling xorriso to start removing files/folders, we first need to figure out what we need.

We can do this by manually going through the installation process. We should, however, hold off from setting up the network connection: going through an "off-line" installation will force Anaconda-installer to only use RPM files from the DVD (instead of fetching new/updated ones from the internet).

After a successful installation of the OS, Anaconda-installer creates a /root/anaconda-ks.cfg file containing kickstart configuration. We can use this file to automate what we just manually installed. More about this file in a bit.

We'll also need the list of RPM packages that we have just installed. We can get this list from yum list --installed. Here's a sample output:

Installed Packages
. . .
kernel.x86_64 4.18.0-348.el8 @anaconda
. . .
rsyslog.x86_64 8.2102.0-5.el8 @AppStream
. . .

In this example, we can find the RPM file for rsyslog package from the @AppStream repository in the ISO image at /AppStream/Packages/rsyslog-8.2102.0-5.el8.x86_64.rpm.

• The @anaconda repository in the above output points to /BaseOS/Packages/ in the ISO image, so the kernel package can be found at /BaseOS/Packages/kernel-core-4.18.0-348.el8.x86_64.rpm.

If, after installing the OS, we need to install additional packages from the DVD, we can have yum install packages from the DVD, instead of fetching new/updated packages from the internet:

# assuming we:
mount /dev/cdrom /mnt
# we can invoke `yum` like so:
yum \
--disablerepo=* \
--repofrompath BaseOS,/mnt/BaseOS \
--repofrompath AppStream,/mnt/AppStream \
install $PACKAGE ...

Now we know what we need. Following our previous example:

• /root/anaconda-ks.cfg
• . . .
• /mnt/BaseOS/Packages/kernel-core-4.18.0-348.el8.x86_64.rpm
• . . .
• /mnt/AppStream/Packages/rsyslog-8.2102.0-5.el8.x86_64.rpm
• . . .

We can copy the RPM files from the DVD to separate folders for BaseOS and AppStream, for example local_BaseOS_Packages and local_AppStream_Packages.

We can then proceed with the fun stuff: removing files/folders.

Add/Remove

It first appears that we don't need /BaseOS/repodata/ and /AppStream/repodata/ from the ISO image. But when we tried removing them, it broke the "Software Selection" section of Anaconda-installer. 😅

So, we settled to only remove /BaseOS/Packages/ and /AppStream/Packages/ from the ISO image, and add only the necessary RPM files from our local folders back in to the ISO image:

rm -f out.iso \
&& xorriso \
-indev OracleLinux-R8-U5-x86_64-dvd.iso \
-outdev out.iso \
-boot_image any replay \
-rm_r /BaseOS/Packages /AppStream/Packages -- \
-map local_BaseOS_Packages /BaseOS/Packages \
-map local_AppStream_Packages /AppStream/Packages

Some notes about the snippet above:

• we first rm -f out.iso to remove any existing out.iso, or else xorriso will "append" to the ISO image, instead of creating a new ISO image.
• -rm_r takes a list of folders. We tell xorriso that we're done with the list with --, before giving additional command (-map in the snippet above).
• we use -map to copy local files/folders into the ISO image.

At this point, we should have a much smaller ISO image, but we have yet to automate the installation. We have /root/anaconda-ks.cfg. We can edit this file to further adjust how we want to automate Anaconda.

We then need to:

• copy this kickstart config file to the ISO image, and
• edit the boot menus, to tell Anaconda installer where it can find the kickstart config file.

The "full ISO" can be used to boot (older) BIOS and (newer) EFI systems. We'll need to edit the boot menu for both. Assuming we add the kickstart config file into the ISO image at /anaconda-ks.cfg, we'll need append inst.ks=hd:LABEL=OL-8-5-0-BaseOS-x86_64:/anaconda-ks.cfg to the "kernel command line" (which the Linux kernel will eventually pass to Anaconda-installer).

The edited BIOS boot menu at /isolinux/isolinux.cfg should end up looking like:

. . .
label linux
menu label ^Install Oracle Linux 8.5.0
kernel vmlinuz
append initrd=initrd.img inst.stage2=hd:LABEL=OL-8-5-0-BaseOS-x86_64 quiet inst.ks=hd:LABEL=OL-8-5-0-BaseOS-x86_64:/anaconda-ks.cfg
. . .

. . .
menuentry 'Install Oracle Linux 8.5.0' . . . {
linuxefi /images/pxeboot/vmlinuz inst.stage2=hd:LABEL=OL-8-5-0-BaseOS-x86_64 quiet inst.ks=hd:LABEL=OL-8-5-0-BaseOS-x86_64:/anaconda-ks.cfg
. . .

rm -f out.iso \
&& xorriso \
-indev OracleLinux-R8-U5-x86_64-dvd.iso \
-outdev out.iso \
-boot_image any replay \
-rm_r /BaseOS/Packages /AppStream/Packages -- \
-map local_BaseOS_Packages /BaseOS/Packages \
-map local_AppStream_Packages /AppStream/Packages \
-map /root/anaconda-ks.cfg /anaconda-ks.cfg \
-update edited_isolinux.cfg /isolinux/isolinux.cfg \
-update edited_grub.cfg /EFI/BOOT/grub.cfg \
-volid MAYHEMDRIVE

• -update edited_isolinux.cfg /isolinux/isolinux.cfg will replace/overwrite /isolinux/isolinux.cfg in the ISO image with the edited_isolinux.cfg we supplied. We similarly update grub.cfg.
• While we're at it, we also changed the volume ID (-volid) of the ISO image from OL-8-5-0-BaseOS-x86_64
  to MAYHEMDRIVE. And, yes, we'll have to update the BIOS and EFI boot menu *.cfg files accordingly.

We use xorriso to do a few more things for us:

• We use a few more packages that are not included in the "full ISO". So we used createrepo to create additional yum repositories and had xorriso -map it to the ISO image.
• We also added Mayhem to the ISO image.

Similar tools

Some readers might notice that some of what we've done so far is quite similar to mkksiso. And that's partly true: mkksiso uses xorriso under the hood (in combination with other tools) to:

• edit boot menus to append kernel command-lines with kickstart config, and
• add more content into the ISO image.

But mkksiso doesn't trim the ISO image. Also, mkksiso (part of lorax) is not as easy to install as xorriso in Debian/Ubuntu or macOS.

There are still other ways to create Anaconda-kickstart bootable-DVDs. Notably, Fedora (on which RHEL is based) uses pungi.

Results

We would understand if some readers consider what we're doing with xorriso as a "hack". But it works for us. We started with a 15 GB ISO image. We were able to trim it down to ~4 GB for Mayhem for Code 1.17. So, these days, we've got an ISO image that can automatically install and configure OS and Mayhem, so our customers can start finding and fixing vulnerabilities on-prem, making it that more difficult for malicious actors.

Links

• xorriso
• Oracle Linux
• Anaconda installer
• kickstart for OL 8