Skip to main content

11.3 Building the ARM Target System and Kernel

Don't rush to flash anything just yet.

In the previous section, we clarified the difference between vmlinux (the blueprint for engineers) and bzImage (the compressed package for the machine). Now, we need to put this theory into practice by building a tangible debugging environment capable of running KGDB.

This is trickier than it sounds—you can't just grab a production kernel and start debugging it. We need a custom-built target system. Any functional Linux system, no matter how minimal, needs these four pieces of the puzzle:

  • Bootloader: Responsible for waking up the kernel. In a QEMU virtual environment, QEMU itself acts as the Bootloader, saving us this step. But on real boards (like ARM embedded devices), we typically use Das U-Boot, while on x86, the familiar GRUB is the standard.
  • DTB (Device Tree Blob): A required "hardware manual" for architectures like ARM/ARM64/PPC, telling the kernel what devices are on the board and which bus they are connected to.
  • Kernel Image: This is the compressed package we discussed in the last section—the actual payload loaded during boot.
  • Rootfs: The filesystem mounted after the kernel boots, containing the init process and essential tools.

Only when you have all four do you have a runnable system.

11.3.1 Automated Building: The SEALS Project​

Building a rootfs from scratch is a tale of woe—BusyBox configuration, library dependencies, init scripts... These aren't problems solved by writing code; they are pure engineering grunt work.

To keep us from spending half our lives on "environment setup" in this chapter, we choose to cheat: by using the SEALS (Simple Embedded ARM Linux System) project.

This is an open-source automated build tool designed specifically to generate "minimal but functional" ARM Linux systems. It handles the following for you:

  1. Kernel configuration and compilation.
  2. DTB generation.
  3. A minimal BusyBox-based rootfs.

SEALS Default Configuration

We use the SEALS ARM Versatile Express (VExpress) platform configuration by default.

  • CPU: ARMv7 Cortex A9 multi-core.
  • Memory: 512 MB RAM.
  • Runtime: QEMU virtual machine.

The most elegant part of this approach is nested virtualization: you can run an ARM virtual machine on an x86_64 host, and debug the kernel inside that ARM VM. This means you don't need to buy any ARM development boards to complete all the experiments in this book.

Of course, Yocto and Buildroot are more powerful industrial-grade solutions, and you might even have a Raspberry Pi or BeagleBone on hand. But for the single goal of "learning KGDB," SEALS is simple enough, fast enough, and requires no specific hardware.

Prerequisites​

Before using SEALS, your host machine needs a few things:

  • QEMU ARM emulator (qemu-system-arm).
  • Cross-compilation toolchain (x86_64 to ARM32 gcc).
  • Some miscellaneous dependency libraries.

Since SEALS isn't the focus of this book, we'll leave the build details to its Wiki. You need to clone the SEALS repository and set up the environment following the Wiki. Here are some signposts:

Once configured and running, you should see an interface similar to Figures 11.4 and 11.5. If you see scrolling boot logs in the QEMU window, the foundation is laid.

11.3.2 Configuring the Kernel: Enabling KGDB​

With the environment ready, it's the kernel's turn.

When you enter make menuconfig, you need to think of yourself as a detective. You need to flip on those debugging switches you would normally never touch. We covered the basics of kernel debugging in Chapter 1; here, we perform some specific surgery for KGDB.

Mandatory Options​

Whether you're using ARM or x86, to get KGDB running, the following kernel configuration options (with the CONFIG_ prefix omitted) are mandatory:

  1. DEBUG_KERNEL=y

    • Location: Kernel hacking -> Kernel debugging
    • This is the master switch. Selecting it reveals many of the subsequent debugging options.

  2. DEBUG_INFO=y —— Most Important

    • Location: Kernel hacking -> Compile-time checks and compiler options -> Compile the kernel with debug info
    • Function: This passes the -g parameter to the compiler, burning debug symbols into vmlinux.
    • Why mandatory? Without it, GDB is just a mess of cryptic addresses, and you'll have no idea what's going on. Technically it isn't "required" (the kernel will still run), but in practice, don't even try debugging without it.

  3. KGDB=y

    • Location: Kernel hacking -> Generic Kernel Debugging Instruments -> KGDB: kernel debugger
    • Function: This formally embeds the GDB server code into the kernel. Only with this can the kernel understand commands sent from GDB.
    • Principle: A gdbserver now runs inside the kernel, sitting there waiting (usually over a serial port or network). Once connected, it executes the GDB commands you send locally and spits the results back.

  4. KGDB_SERIAL_CONSOLE=y

    • Function: Allows reusing the serial console for KGDB communication. This is the simplest and most stable connection method.
    • Details: This involves a kernel parameter called kgdboc (KGDB over console), which we'll cover in detail later when we bind it to a specific serial port (like ttyS0).

  5. KGDB_HONOUR_BLOCKLIST=y

    • Recommended to enable. It prevents you from setting breakpoints inside certain non-interruptible functions (like those on the kprobe blacklist), avoiding recursive traps that could deadlock the system.

  6. MAGIC_SYSRQ=y

    • Location: Kernel hacking -> Generic Kernel Debugging Instruments -> Magic SysRq key
    • Function: Allows you to intervene in the kernel at runtime via /proc/sysrq-trigger. Writing a g to it forces the kernel to stop and enter KGDB debugging mode.
    • Companion: You also need to set /proc/sys/kernel/sysrq to 1 at runtime to ensure all SysRq functions are enabled.

What's That Thing Called Kdb?​

In this menu, you might also see Kdb.

Kdb is a command-line-only debugger. It doesn't require two machines and can be used directly over a serial port. You can inspect memory, registers, and logs, but it does not support source-level debugging.

It's like this: KGDB is a professional IDE with a GUI, while Kdb is a console with only a hex editor. We focus on KGDB in this chapter; just understand Kdb as an alternative tool.

Options to Disable (Pitfall Avoidance)​

Some security features will clash with software debugging breakpoints. If the following options appear in your menu, turn them off:

  • CONFIG_STRICT_KERNEL_RWX
  • CONFIG_STRICT_MODULE_RWX

These two options force the kernel code segment to be read-only and the data segment to be non-executable. While intended as security protection, they prevent GDB from writing software breakpoint instructions (which essentially replace an instruction with INT 3).

Solution: Either turn them off, or use Hardware Breakpoints. Hardware breakpoints don't modify memory; they use the CPU's debug registers, so they aren't affected by this protection. This book recommends prioritizing hardware breakpoints.

If you don't mind the extra effort, the following options can improve the experience:

  • FRAME_POINTER=y (Kernel hacking -> Compile the kernel with frame pointers)
    • Although marked as Optional, this is very useful for stack unwinding. If your platform has CONFIG_UNWINDER_ARM enabled by default, this option might conflict—choose accordingly.
  • DEBUG_INFO_SPLIT=y: Splits the debug information into a separate .dwo file, reducing the vmlinux file size.
  • GDB_SCRIPTS=y: This is fantastic. When you load vmlinux, it automatically links some Python-based GDB helper scripts (like lx-symbols, lx-lsmod). We have a dedicated section on this later.
  • DEBUG_FS=y: Mounts the debugfs filesystem, where a lot of kernel debugging information resides.

⚠️ One Last Warning

When debugging the kernel, always disable the watchdog.

If a software or hardware watchdog is running and you pause at a breakpoint for too long, the watchdog feed will timeout and the system will reboot directly. The bug you just found will vanish without a single error message.

Enabling KGDB in SEALS​

If you're using SEALS, things are a bit simpler. The SEALS build scripts have a built-in KGDB switch.

Open your board's build.config file and find this line:

$ grep KGDB build.config
KGDB_MODE=0 # make '1' to have qemu run with the '-s -S' 
            # switch (waits for client GDB to 'connect')

Change KGDB_MODE to 1. This way, the QEMU boot script generated by SEALS will automatically add the -s -S parameter, causing QEMU to freeze the CPU at boot and wait patiently for GDB to connect.

Tip: GCC Plugins

If asked to Enable GCC plugins? during kernel compilation, we recommend selecting No. Plugins can sometimes interfere with debug information generation or introduce weird optimizations, actually making debugging harder. Keep it simple.

11.3.3 Test-Running the Target System​

Don't take my word for it—trust the facts.

After configuring and compiling, gather all the components generated by SEALS. In the SEALS staging directory, you should find these items:

  • Kernel Images:
    • .../linux-5.10.109/arch/arm/boot/zImage (the compressed package for the Bootloader, used for booting)
    • .../linux-5.10.109/vmlinux (the blueprint, with debug symbols, used by GDB)
  • Device Tree: .../linux-5.10.109/arch/arm/boot/dts/vexpress-v2p-ca9.dtb
  • Rootfs: .../seals_staging_vexpress/images/rfs.img

Now, string them together with QEMU.

If you're running QEMU manually, the command looks roughly like this (don't be intimidated—we'll break it down):

qemu-system-arm \
 -M vexpress-a9 \
 -m 512 \
 -smp 4,sockets=2 \
 -kernel <...>/seals_staging_vexpress/images/zImage \
 -drive file=/<...>/seals_staging_vexpress/images/rfs.img,if=sd,format=raw \
 -append "console=ttyAMA0 rootfstype=ext4 root=/dev/mmcblk0 init=/sbin/init" \
 -dtb /<...>/seals_staging_vexpress/images/vexpress-v2p-ca9.dtb \
 -nographic

Parameter Breakdown (don't skip this):

  • -M vexpress-a9: Specifies the emulated board model. Running qemu-system-arm -M help shows all boards supported by QEMU.
  • -m 512: Allocates 512MB of RAM to the virtual machine.
  • -kernel ...: Specifies the kernel image. Note that we provide the zImage here because we want to boot it.
  • -drive file=...,if=sd: Specifies the rootfs image and tells QEMU to emulate it as an SD card.
  • -append "...": These are the boot parameters passed to the kernel.
    • console=ttyAMA0: Redirects console output to this virtual serial port (the standard serial port for ARM).
    • root=/dev/mmcblk0: Tells the kernel that the rootfs is on the emulated SD card.
  • -dtb ...: You must specify the DTB file, otherwise the kernel won't know what hardware it's running on.

If everything goes well, you'll see scrolling boot logs in the terminal, finally stopping at the BusyBox shell prompt:

[    2.345678] Freeing unused kernel memory: 204K
/ #

See that / #?

Congratulations, your ARM virtual machine is alive.

Figures 11.4 and 11.5 show screenshots of this step (QEMU boot logs and shell login). If you can reproduce this interface, the "target system" piece is officially done.

Now we have a well-behaved virtual sheep in our hands. In the next chapter, we take a knife (GDB) to it and see how it runs on the inside.