In my previous post, I mentioned that some errors in the kernel will freeze your machine and likely force you to reboot. You can certainly agree that whenever one learns a new technology, mistakes are inevitable, and rebooting after every error we cause could become a bit time-consuming. Needless to say, crashing the live kernel on a machine with unsaved work can be very disheartening + there’s also the risk of messing up your filesystem.
To speed up our development process, we’re going to build ourselves a bare bones virtual machine using an emulator called QEMU. This VM will then boot the kernel and give us a simple shell with a bunch of tools. Using a virtual machine also stands together with easy debugging and fine-grained control over the system’s virtual hardware.
Before starting this blog, I tried to build something similar, but it didn’t turn out that great. Let’s make things right this time, shall we?
Even though we’re not writing any C code yet, some preparations for the development environment are necessary. Here’s what we’re gonna use:
- Linux - Duh!
BusyBox - An application suite meant as a GNU coreutils
alternative; typically built as a single size-optimized binary, which uses
specially named symlinks to act as different command-line tools like
- QEMU - The hypervisor for our VM; when installing QEMU, most Linux distributions and macOS should have you covered, though I don’t recommend following this tutorial on Windows (but if you really want, precompiled binaries and Cygwin could be of help)
- Basic grasp on GNU Make - The classic Makefiles still remain the kernel’s build system of choice, a little manual compilation-fu shouldn’t hurt.
- Working git knowledge - Git is the industry’s leading version control system at the moment; in fact, git’s initial purpose was to track changes in Linux(!) after the previously used proprietary solution was discontinued. With both Linux and BusyBox using git, sticking to the same VCS seems like a good idea
Enter the submodules!
For downloading Linux and BusyBox code, we’ll use git’s submodules. Submodules are nothing more than nested repositories checked out at a given commit in some other top-level repo. Let’s put that into perspective:
Suppose you had a git repository
foo. Inside, you could define a submodule
bar pulled from a remote at
https://www.example.com/bar.git, checked out at
abcd1234 in a directory
bar_dir/. When you do that, repository
information only about those 4 things - the submodule’s name (
bar_dir/), remote URL (
https://www.example.com/bar.git) and the hash
of it’s desired revision (
abcd1234). That way,
foo doesn’t need to store the
bar, while referencing a known state of its files at the same time.
When you pull the submodule,
bar_dir/ will reflect the contents of
Let’s start by creating an empty repository and pulling two submodules for Linux and BusyBox (optionally, you could go and grab something to eat as both pulls may take some time to complete):
git init git submodule add https://github.com/torvalds/linux.git git submodule add https://github.com/mirror/busybox.git git commit -m "add linux and busybox submodules"
Hopefully, you should end up with two directories named
Linux, being a huge project, eventually developed a build configuration system
kbuild. You can expect me to explain it in a future post, but this
time we’ll only nudge it a little. Let’s select a default config:
cd linux/ make defconfig
The output should look something like this:
*** Default configuration is based on 'x86_64_defconfig' # # configuration written to .config #
Now, it is time to finally compile. Let’s build, already!
make -jN # N should be the number of CPU cores in your system
A correct build’s output should end with a message about the location of your warm, freshly built kernel - something similar to this:
[...] Setup is 15836 bytes (padded to 15872 bytes). System is 6657 kB CRC d8c5cae9 Kernel: arch/x86/boot/bzImage is ready (#1)
When in doubt, check the
$?variable in your shell. It stores the return value of your last command, which is always equal to
0after a successful execution.
To execute your kernel in QEMU, run this in your terminal:
qemu-system-$(uname -m) -kernel arch/$(uname -m)/boot/bzImage -append console=ttyS0 -nographic
$(uname -m) parts resolve to your architecture’s name; to exit the
Ctrl+a and then
If you got QEMU correctly installed, you should see a bunch of log entries storm through your terminal. It should then stop after a couple seconds with…
[ 4.000081] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6 [ 4.003700] Please append a correct "root=" boot option; here are the available partitions: [ 4.005934] 0b00 1048575 sr0 [ 4.006006] driver: sr [ 4.006772] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) [ 4.007717] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-rc1+ #1 [ 4.008084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-20170228_101828-anatol 04/01/2014 [ 4.008799] Call Trace: [ 4.010123] dump_stack+0x4d/0x65 [ 4.010384] panic+0xca/0x203 [ 4.010682] mount_block_root+0x175/0x229 [ 4.010951] mount_root+0x101/0x10a [ 4.011186] prepare_namespace+0x13a/0x172 [ 4.011420] kernel_init_freeable+0x1c0/0x1d5 [ 4.011698] ? rest_init+0x80/0x80 [ 4.011931] kernel_init+0x9/0x100 [ 4.012146] ret_from_fork+0x29/0x40 [ 4.013082] Kernel Offset: disabled [ 4.013712] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
…a kernel panic.
Let’s peek into the log excerpt from above to figure out what it means and why we crashed. Have a look at the first couple of lines:
[ 4.000081] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6 [ 4.003700] Please append a correct "root=" boot option; here are the available partitions: [ 4.005934] 0b00 1048575 sr0
In the leftmost column we can see the time since the kernel’s start in a
<seconds>.<microseconds> format. Then, we’ve got a message suggesting that an
VFS (Virtual File System, a filesystem abstraction used in
Linux) has problems with accessing some device. Looks like we forgot to define
something, as the next line follows up by requesting us to set a
Kernels, like any other program, can take arguments. These arguments are usually specified by the bootloader on kernel load and tell your kernel about stuff like whether to turn on debugging, network configuration or which device to use as the filesystem root.
It looks like the kernel behaved exactly the way it should - we didn’t name any
device to mount as
/, so Linux just shrugs and crashes, as it doesn’t even
know where to look for an
To fix that, we’re going to create a ramdisk, which QEMU will then load to act as a filesystem image for our kernel. There, we will form a simple environment that will let us access a shell with some command-line tools. Let’s start by building BusyBox.
Getting your box busy (Ha, ha.)
Let’s head over to the BusyBox directory and create the default configuration:
cd ../busybox make defconfig
Looks familiar, huh? BusyBox is using kbuild too!
This time, we’ll change one option in our config before starting the build. It turns out that the default build links BusyBox dynamically against your host system’s shared libraries. Since our virtual machine can’t provide BusyBox with any of its dependencies, we’ll need to link it statically - that is, to embed the necessary libraries directly in the program’s binary. To do that, start a configuration editing script:
make menuconfig # or gconfig, xconfig
menuconfig is one of many frontends available for editing Linux-like configs.
It’s the most popular, so that’s what I recommend to use.
After the configuration window appears, select
Busybox Settings and hit enter.
Then, use your arrows to go down and select
Build BusyBox as a static
binary (no shared libs) (hit
space and an asterisk should show to the left).
Exit, choose to save your changes and build!
make -jN # Again, N is your system's CPU core count
To make sure that the resulting executable is statically linked, let’s see what
ldd has to say about it:
ldd is a tool for viewing shared library dependencies in a program
If you got the config right, ldd should fend you off with a
not a dynamic
executable message. Looks like we’re almost there! Let’s go and see about that
Creating a simple environment in an initramfs image
initramfs stands for
system, which is basically a
small filesystem that gets loaded into RAM at boot time. It provides your kernel
with the tools it needs to proceed with bringing up the rest of your OS.
Linux sources offer many useful tools, and
usr/gen_init_cpio is one of them.
gen_init_cpio is a program for generating filesystem images exactly like the
one we’re after. One way to use it is to have it run during the Linux build and
append the resulting initramfs image to your kernel. Alternatively, you can keep
the image separate, and that’s what we’re going to do for now.
Let’s start by going back to the top level of our repository. There, we’ll
create a special
initramfs_desc (pick a different name if you want) file which
will describe the files that we want in our image and where to put them. Here’s
what I came up with:
# type dest src perm usr grp # dirs dir /bin 775 0 0 dir /proc 775 0 0 dir /sys 775 0 0 dir /dev/pts 775 0 0 # files file /init init.sh 775 0 0 file /bin/busybox busybox/busybox 775 0 0 # symlinks slink /bin/sh /bin/busybox 775 0 0 # devnodes nod /dev/zero 666 0 0 c 1 5 nod /dev/null 666 0 0 c 1 3 nod /dev/console 666 0 0 c 5 1
As you can see, each item has a type, source/destination paths (i.e. where to
find the file on your computer and where to put it in the image), permissions,
an owner user and a group (currently set to what you know as
root). The three
entries in the low right signify special properties of character device files,
about which we’ll surely talk in the future.
If you look closely, you’ll also notice that in my
initramfs_desc I listed a
/init file to be copied from
init.sh shell script. You may have heard about
a thing called the
init process - it’s basically a program that runs first
thing after a UNIX-like kernel boots. This
init program is the first userland
process and gets assigned PID 1.
init is the parent of all processes and is
vital for a UNIX-like OS to work - in Linux, attempting to kill init results in
a kernel panic. A popular approach is to make init a service that runs programs
according to a file called inittab (a practice most widely known
from System V) - BusyBox provides such functionality, but for the
sake of simplicity, we’re not going to get into that.
We’ll create our own
init as a simple shell script that will bootstrap
BusyBox, mount some filesystems and fire up a shell. This is what my
#!/bin/sh # BusyBox bootstrap /bin/busybox --install -s /bin # Mount pseudofilesystems mount -t proc proc /proc mount -t sysfs sysfs /sys mount -t devpts devpts /dev/pts # Execute shell while true; do /bin/sh done
Thanks to the
/bin/sh symlink we’ve defined
earlier, BusyBox is able to provide us with a shell for executing the script.
Let’s create the image!
linux/usr/gen_init_cpio initramfs_desc > initramfs.img
We are ready. Run the final QEMU command:
qemu-system-$(uname -m) -kernel linux/arch/$(uname -m)/boot/bzImage -nographic -append console=ttyS0 -initrd initramfs.img
Your output should look similar to mine, with a
/ # shell prompt hidden
somewhere in the kernel logs:
[ 1.625412 ] netconsole: network logging started [ 1.627415 ] ALSA device list: [ 1.627503 ] No soundcards found. [ 1.669467 ] Freeing unused kernel memory: 1224K [ 1.669726 ] Write protecting the kernel read-only data: 14336k [ 1.670954 ] Freeing unused kernel memory: 100K [ 1.686908 ] Freeing unused kernel memory: 1232K /bin/sh: can't access tty; job control turned off / # [ 1.831247 ] tsc: Refined TSC clocksource calibration: 3199.092 MHz [ 1.831527 ] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2e1cf059357, max_idle_ns: 440795283803 ns [ 2.192547 ] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 [ 2.839558 ] clocksource: Switched to clocksource tsc
Feel free to snoop around and try out the different utilities provided by
BusyBox. To see all of the available choices, just call
Me, about a year ago, when I brought up my first Linux OS
Congratulations! Believe it or not, but what you just did in this tutorial was bringing up a tiny Linux-based operating system. Even though you didn’t write any code, you managed to create something useful that will later on help you safely test your stuff.
Okay, what now?
In the next part of this course, We’re going to cover kernel modules, what they are and how to write one. There’s finally gonna be actual C code involved, so it might be a good idea to brush up on the language, if you don’t know it well.
Don’t hesitate to ask any questions or share your thoughts below. Constructive criticism will always be the most appreciated.
QEMU complains about permissions
QEMU commands shown above might likely fail due to insufficient permissions of
your user’s account. Many distributions only allow QEMU VMs to be run by members
kvm group. Check your account with the
groups command and if
not there, add it with
sudo usermod -a -G kvm <your_username>
After getting into the group, simply log out and come back in for the change to take effect.
<some_package> is not installed
It’s possible that during the compilation process you’ll get error messages about missing dependencies of the build environment - this is quite normal if you’ve never built any C code on your computer, just look for your distribution’s instructions on getting the necessary packages.
- Speeding up kernel development with QEMU - My main inspiration for writing this post, the article targets more experienced developers and proposes some more sophisticated solutions
- Kernel Newbies - A cool place for beginners to find some basic information about the kernel; personally, I was amazed by the human-readable descriptions available on the site, which cover the changes between different Linux versions
- The course’s GitHub repo - Feel free to peek in to see some of the code I use in this series