Skip to main content

Chapter 2: A Torch in the Dark Forest

The core insight established in this chapter is that debugging the kernel and debugging user-space programs are entirely different beasts. In user space, you have the entire libc library at your disposal, isolated process spaces, and a GDB you can attach to at any time. In kernel space, a single bad pointer can paralyze the entire machine—and once it halts, all context evaporates with the power loss, leaving nothing behind but a pile of hex numbers.

Once you understand this, the "clunky," "complex," and even "primitive" design of kernel debugging toolchains no longer seems baffling. It is, in essence, dancing in chains—extracting information from an extremely constrained environment.

Environment Preparation​

Don't rush to write code—check your gear first.

This might sound like a cliché, but trust me: diving into kernel code without a properly configured environment is the most inefficient way to waste your time. If you already set up your environment following the "Setting up the workspace" section in Chapter 1, you can reuse that setup right now. If not, now is a great time to go back and do it—we need a stable Linux development environment (a host machine or VM is recommended), a cross-compilation toolchain, and a kernel source tree that actually builds and runs.

We won't repeat the tedious apt install list here, but you must ensure you have the following two things ready at hand:

  1. Build toolchain: A gcc/make toolset capable of compiling the kernel and modules.
  2. Target system: A Linux system you can tinker with (either a QEMU VM or a physical board), where you can see its output via serial or network, and even make it halt on a crash so you can inspect it.

Don't skip this step. Later on, when you're staring blankly at a screen full of meaningless registers, you'll be grateful for the extra ten minutes you spent here.

2.1 Technical Requirements and Stack​

Hardware and Software Prerequisites​

Just like the previous chapter, let's get on the same page.

Whether you're using a local VM or a remote development board, all the following demonstrations assume you have these capabilities:

  • Root privileges: Kernel debugging doesn't play nice. Loading modules, inspecting kernel memory (/dev/mem), and configuring debug interfaces all require root.
  • Kernel source tree: You need a source tree that is exactly identical to the version running on the target machine (or at least matches the version number). Analyzing a crash stack trace with mismatched source code is like using an English map to find Chinese street signs—you might get a rough idea, but you'll definitely get lost when it matters.

Workspace Checklist​

Let's do a quick run-through of the workspace checklist.

You don't need to type anything out, just mentally check these off:

  • Compiler: Is arm-linux-gnueabihf-gcc or x86_64-linux-gnu-gcc in your $PATH?
  • Kernel configuration: Are CONFIG_DEBUG_KERNEL and CONFIG_KGDB enabled in your .config file? (If these terms look unfamiliar, don't panic—we'll break them down shortly.)
  • Serial/Network: Can you connect to the target machine using minicom, picocom, or ssh?

Don't cut corners here.

If you skipped some steps while setting up the environment in the previous chapter, thinking you "won't need them for now," this is where it comes back to bite you. Many debugging tools—especially KGDB and kdump, which we'll cover later—are extremely sensitive to kernel versions and configurations. Being off by a single patch version, or missing a single .config option, might leave you with absolutely no output, or worse: a screen full of misleading garbage.

Don't rush to type code just yet.

If all of the above checks out, we'll officially enter the jungle of kernel debugging—but first, we need to know what kind of beast we're hunting.