In my previous post, I mentioned that some errors in the kernel will freeze your machine and likely force you to reboot. You can certainly agree that whenever one learns a new technology, mistakes are inevitable, and rebooting after every error we cause could become a bit time-consuming. Needless to say, crashing the live kernel on a machine with unsaved work can be very disheartening + there’s also the risk of messing up your filesystem.

The plan

To speed up our development process, we’re going to build ourselves a bare bones virtual machine using an emulator called QEMU. This VM will then boot the kernel and give us a simple shell with a bunch of tools. Using a virtual machine also stands together with easy debugging and fine-grained control over the system’s virtual hardware.

Before starting this blog, I tried to build something similar, but it didn’t turn out that great. Let’s make things right this time, shall we?

Prerequisites

Even though we’re not writing any C code yet, some preparations for the development environment are necessary. Here’s what we’re gonna use:

  • Linux - Duh!
  • BusyBox - An application suite meant as a GNU coreutils alternative; typically built as a single size-optimized binary, which uses specially named symlinks to act as different command-line tools like ls, rm, cp, tar etc.
  • QEMU - The hypervisor for our VM; when installing QEMU, most Linux distributions and macOS should have you covered, though I don’t recommend following this tutorial on Windows (but if you really want, precompiled binaries and Cygwin could be of help)
  • Basic grasp on GNU Make - The classic Makefiles still remain the kernel’s build system of choice, a little manual compilation-fu shouldn’t hurt.
  • Working git knowledge - Git is the industry’s leading version control system at the moment; in fact, git’s initial purpose was to track changes in Linux(!) after the previously used proprietary solution was discontinued. With both Linux and BusyBox using git, sticking to the same VCS seems like a good idea

Enter the submodules!

For downloading Linux and BusyBox code, we’ll use git’s submodules. Submodules are nothing more than nested repositories checked out at a given commit in some other top-level repo. Let’s put that into perspective:

Suppose you had a git repository foo. Inside, you could define a submodule bar pulled from a remote at https://www.example.com/bar.git, checked out at abcd1234 in a directory bar_dir/. When you do that, repository foo stores information only about those 4 things - the submodule’s name ("bar"), relative path (bar_dir/), remote URL (https://www.example.com/bar.git) and the hash of it’s desired revision (abcd1234). That way, foo doesn’t need to store the whole of bar, while referencing a known state of its files at the same time. When you pull the submodule, bar_dir/ will reflect the contents of bar at abcd1234.

Repository setup

Let’s start by creating an empty repository and pulling two submodules for Linux and BusyBox (optionally, you could go and grab something to eat as both pulls may take some time to complete):

git init
git submodule add https://github.com/torvalds/linux.git
git submodule add https://github.com/mirror/busybox.git
git commit -m "add linux and busybox submodules"

Hopefully, you should end up with two directories named busybox and linux

:information_source: Note:

At the time of writing this post, connections to the original kernel.org and busybox.net repos seem a bit flaky - the commands above fall back to the GitHub mirrors.

Building Linux

Linux, being a huge project, eventually developed a build configuration system called kbuild. You can expect me to explain it in a future post, but this time we’ll only nudge it a little. Let’s select a default config:

cd linux/
make defconfig

The output should look something like this:

*** Default configuration is based on 'x86_64_defconfig'
#
# configuration written to .config
#

Now, it is time to finally compile. Let’s build, already!

make -jN # N should be the number of CPU cores in your system

A correct build’s output should end with a message about the location of your warm, freshly built kernel - something similar to this:

[...]
Setup is 15836 bytes (padded to 15872 bytes).
System is 6657 kB
CRC d8c5cae9
Kernel: arch/x86/boot/bzImage is ready  (#1)

:+1: Tip:

When in doubt, check the $? variable in your shell. It stores the return value of your last command, which is always equal to 0 after a successful execution.

Lift-off! (?)

To execute your kernel in QEMU, run this in your terminal:

qemu-system-$(uname -m) -kernel arch/$(uname -m)/boot/bzImage -append console=ttyS0 -nographic

The $(uname -m) parts resolve to your architecture’s name; to exit the VM, press Ctrl+a and then x

If you got QEMU correctly installed, you should see a bunch of log entries storm through your terminal. It should then stop after a couple seconds with…

[    4.000081] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[    4.003700] Please append a correct "root=" boot option; here are the available partitions:
[    4.005934] 0b00         1048575 sr0
[    4.006006]  driver: sr
[    4.006772] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    4.007717] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-rc1+ #1
[    4.008084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-20170228_101828-anatol 04/01/2014
[    4.008799] Call Trace:
[    4.010123]  dump_stack+0x4d/0x65
[    4.010384]  panic+0xca/0x203
[    4.010682]  mount_block_root+0x175/0x229
[    4.010951]  mount_root+0x101/0x10a
[    4.011186]  prepare_namespace+0x13a/0x172
[    4.011420]  kernel_init_freeable+0x1c0/0x1d5
[    4.011698]  ? rest_init+0x80/0x80
[    4.011931]  kernel_init+0x9/0x100
[    4.012146]  ret_from_fork+0x29/0x40
[    4.013082] Kernel Offset: disabled
[    4.013712] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

…a kernel panic.

“Uh-oh!”

Let’s peek into the log excerpt from above to figure out what it means and why we crashed. Have a look at the first couple of lines:

[    4.000081] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[    4.003700] Please append a correct "root=" boot option; here are the available partitions:
[    4.005934] 0b00         1048575 sr0

In the leftmost column we can see the time since the kernel’s start in a <seconds>.<microseconds> format. Then, we’ve got a message suggesting that an entity called VFS (Virtual File System, a filesystem abstraction used in Linux) has problems with accessing some device. Looks like we forgot to define something, as the next line follows up by requesting us to set a root boot argument.

:information_source: Note:

Kernels, like any other program, can take arguments. These arguments are usually specified by the bootloader on kernel load and tell your kernel about stuff like whether to turn on debugging, network configuration or which device to use as the filesystem root.

It looks like the kernel behaved exactly the way it should - we didn’t name any device to mount as /, so Linux just shrugs and crashes, as it doesn’t even know where to look for an init program.

To fix that, we’re going to create a ramdisk, which QEMU will then load to act as a filesystem image for our kernel. There, we will form a simple environment that will let us access a shell with some command-line tools. Let’s start by building BusyBox.

Getting your box busy (Ha, ha.)

Let’s head over to the BusyBox directory and create the default configuration:

cd ../busybox
make defconfig

Looks familiar, huh? BusyBox is using kbuild too!

This time, we’ll change one option in our config before starting the build. It turns out that the default build links BusyBox dynamically against your host system’s shared libraries. Since our virtual machine can’t provide BusyBox with any of its dependencies, we’ll need to link it statically - that is, to embed the necessary libraries directly in the program’s binary. To do that, start a configuration editing script:

make menuconfig # or gconfig, xconfig

menuconfig is one of many frontends available for editing Linux-like configs. It’s the most popular, so that’s what I recommend to use.

After the configuration window appears, select Busybox Settings and hit enter. Then, use your arrows to go down and select Build BusyBox as a static binary (no shared libs) (hit space and an asterisk should show to the left). Exit, choose to save your changes and build!

make -jN # Again, N is your system's CPU core count

To make sure that the resulting executable is statically linked, let’s see what ldd has to say about it:

ldd ./busybox

ldd is a tool for viewing shared library dependencies in a program

If you got the config right, ldd should fend you off with a not a dynamic executable message. Looks like we’re almost there! Let’s go and see about that filesystem image.

Creating a simple environment in an initramfs image

initramfs stands for initial RAM filesystem, which is basically a small filesystem that gets loaded into RAM at boot time. It provides your kernel with the tools it needs to proceed with bringing up the rest of your OS.

Linux sources offer many useful tools, and usr/gen_init_cpio is one of them. gen_init_cpio is a program for generating filesystem images exactly like the one we’re after. One way to use it is to have it run during the Linux build and append the resulting initramfs image to your kernel. Alternatively, you can keep the image separate, and that’s what we’re going to do for now.

Let’s start by going back to the top level of our repository. There, we’ll create a special initramfs_desc (pick a different name if you want) file which will describe the files that we want in our image and where to put them. Here’s what I came up with:

# type          dest            src             perm    usr     grp

# dirs
dir             /bin                            775     0       0
dir             /proc                           775     0       0
dir             /sys                            775     0       0
dir             /dev/pts                        775     0       0

# files
file            /init           init.sh         775     0       0
file            /bin/busybox    busybox/busybox 775     0       0

# symlinks
slink           /bin/sh         /bin/busybox    775     0       0

# devnodes
nod             /dev/zero                       666     0       0       c 1 5
nod             /dev/null                       666     0       0       c 1 3
nod             /dev/console                    666     0       0       c 5 1

As you can see, each item has a type, source/destination paths (i.e. where to find the file on your computer and where to put it in the image), permissions, an owner user and a group (currently set to what you know as root). The three entries in the low right signify special properties of character device files, about which we’ll surely talk in the future.

If you look closely, you’ll also notice that in my initramfs_desc I listed a /init file to be copied from init.sh shell script. You may have heard about a thing called the init process - it’s basically a program that runs first thing after a UNIX-like kernel boots. This init program is the first userland process and gets assigned PID 1. init is the parent of all processes and is vital for a UNIX-like OS to work - in Linux, attempting to kill init results in a kernel panic. A popular approach is to make init a service that runs programs according to a file called inittab (a practice most widely known from System V) - BusyBox provides such functionality, but for the sake of simplicity, we’re not going to get into that.

We’ll create our own init as a simple shell script that will bootstrap BusyBox, mount some filesystems and fire up a shell. This is what my init.sh looks like:

#!/bin/sh

# BusyBox bootstrap
/bin/busybox --install -s /bin

# Mount pseudofilesystems
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devpts devpts /dev/pts

# Execute shell
while true; do
  /bin/sh
done

Thanks to the /bin/busybox :arrow_right: /bin/sh symlink we’ve defined earlier, BusyBox is able to provide us with a shell for executing the script.

Let’s create the image!

linux/usr/gen_init_cpio initramfs_desc > initramfs.img

Lift-off! :rocket:

We are ready. Run the final QEMU command:

qemu-system-$(uname -m) -kernel linux/arch/$(uname -m)/boot/bzImage -nographic -append console=ttyS0 -initrd initramfs.img

Your output should look similar to mine, with a / # shell prompt hidden somewhere in the kernel logs:

[    1.625412 ] netconsole: network logging started
[    1.627415 ] ALSA device list:
[    1.627503 ]   No soundcards found.
[    1.669467 ] Freeing unused kernel memory: 1224K
[    1.669726 ] Write protecting the kernel read-only data: 14336k
[    1.670954 ] Freeing unused kernel memory: 100K
[    1.686908 ] Freeing unused kernel memory: 1232K
/bin/sh: can't access tty; job control turned off
/ # [    1.831247 ] tsc: Refined TSC clocksource calibration: 3199.092 MHz
[    1.831527 ] clocksource: tsc: mask: 0xffffffffffffffff max_cycles:
0x2e1cf059357, max_idle_ns: 440795283803 ns
[    2.192547 ] input: ImExPS/2 Generic Explorer Mouse as
/devices/platform/i8042/serio1/input/input3
[    2.839558 ] clocksource: Switched to clocksource tsc

Feel free to snoop around and try out the different utilities provided by BusyBox. To see all of the available choices, just call busybox without arguments.

It’s alive!

*Frankenstein yelling "It's alive!"*

Me, about a year ago, when I brought up my first Linux OS

Congratulations! :confetti_ball: Believe it or not, but what you just did in this tutorial was bringing up a tiny Linux-based operating system. Even though you didn’t write any code, you managed to create something useful that will later on help you safely test your stuff.

Okay, what now?

In the next part of this course, We’re going to cover kernel modules, what they are and how to write one. There’s finally gonna be actual C code involved, so it might be a good idea to brush up on the language, if you don’t know it well.

Don’t hesitate to ask any questions or share your thoughts below. Constructive criticism will always be the most appreciated.

Known problems

QEMU complains about permissions

QEMU commands shown above might likely fail due to insufficient permissions of your user’s account. Many distributions only allow QEMU VMs to be run by members of the kvm group. Check your account with the groups command and if kvm is not there, add it with usermod:

sudo usermod -a -G kvm <your_username>

After getting into the group, simply log out and come back in for the change to take effect.

<some_package> is not installed

It’s possible that during the compilation process you’ll get error messages about missing dependencies of the build environment - this is quite normal if you’ve never built any C code on your computer, just look for your distribution’s instructions on getting the necessary packages.

Further reading