Skip to main content

7.4 Precision Strike: Pinpointing the Culprit with objdump and GDB

In the previous section, we finished reading the autopsy report. We know the do_the_work function crashed, and we know the RIP register stopped at offset 0x124.

But that's not enough.

The challenge is "pinpointing." We want to know exactly which line of source code, which specific statement, caused the disaster. Was it that pr_info? Or the assignment that followed?

This requires mapping the RIP address (or offset) back to a source code line. That's exactly what we'll do in this section: go from "roughly knowing" to "knowing for sure."

Preparation Before Debugging: Bugging the Kernel​

Before diving into practice, there is one prerequisite: your binary must contain debug symbols.

A binary without symbols is like a pill bottle with the labels removed—you know there are pills inside, but you don't know which one is the painkiller and which one is poison.

To get the most complete debugging experience, you must recompile your buggy kernel or module with CONFIG_DEBUG_INFO=y. Simply put, this means booting into a debug kernel and building your module there.

Open our so-called "enhanced Makefile" and find this variable:

MYDEBUG := y

It defaults to n, and you need to change it to y.

Rebuild the module. You'll notice that the generated .ko file has grown larger—sometimes significantly so. This is perfectly normal, because it now contains not just machine code, but also a detailed "map" telling the debugger which chunk of machine code corresponds to which line of C code.

objdump: The Microscope of Disassembly​

objdump is a powerful tool that can tear ELF object files (including the uncompressed kernel image vmlinux and module .ko files) down to their bare bones, letting you see every single vein inside.

The most common combo is the -dS option:

  • -d (--disassemble): Disassemble executable sections.
  • -S (--source): Intermix source code with the assembly where possible (provided you compiled with -g).

Step 1: Find the Module's In-Memory Base Address​

If your module is still running in memory (or just crashed), you can first grab its load base address from /proc/modules.

When a normal user views this file, the kernel hides the addresses for security (to prevent information leaks):

$ grep oops_tryv2 /proc/modules 
oops_tryv2 16384 0 - Live 0x0000000000000000 (OE)

All zeros. That's useless.

You need root privileges to peek at the truth:

$ sudo grep oops_tryv2 /proc/modules 
oops_tryv2 16384 0 - Live 0xffffffffc0604000 (OE)

See that? 0xffffffffc0604000. That's the kernel virtual address (VMA) where our module was loaded.

Step 2: Generate a Disassembly File with Addresses​

With the base address in hand, we can put objdump to work. The key is the --adjust-vma parameter, which tells objdump: "please pretend this module starts at this address." This ensures the addresses on the left side of the output correspond one-to-one with the actual runtime kernel addresses.

$ objdump -dS --adjust-vma=0xffffffffc0604000 ./oops_tryv2.ko > oops_tryv2.disas

Open the generated oops_tryv2.disas, and you'll see output similar to this:

static void do_the_work(struct work_struct *work)
{
ffffffffc0604000:	e8 00 00 00 00 	callq  ffffffffc0604005 <do_the_work+0x5>
ffffffffc0604005:	55                   	push   %rbp
[...]

On the left is the absolute address, in the middle is the machine code, and on the right is the assembly intermixed with C source code.

Step 3: Calculate the Exact Crash Address​

Remember the RIP value we extracted from the Oops message in the previous section?

RIP: 0033:[<ffffffffc0604124>]

This is an absolute address. But since we already specified the module base address in objdump—or more simply, since we can just use the offset provided in the Oops message, which is 0x124 from <do_the_work+0x124>—the calculation is straightforward.

The formula is very simple:

模块基址 (从 /proc/modules 来) + 偏移量 (从 Oops RIP 来) = 精确崩溃地址
0xffffffffc0604000 + 0x124 = 0xffffffffc0604124

Step 4: Follow the Map​

Armed with the calculated 0xffffffffc0604124, search through the oops_tryv2.disas file we just generated.

You'll find this section (keep your eyes on the addresses on the left):

(This corresponds to the original Figure 7.15/7.16, showing objdump output)

Amidst that sea of hexadecimal code, you'll discover that right above the 0xffffffffc0604124 line lies the culprit C code:

61              pr_info("Generating Oops by attempting to write to an invalid kernel memory pointer\n");
62              oopsie->data = 'x';

Found it. It's line 62.

Through this method, we turned the cold, hard number in the RIP register into a specific line of C code. This is ten thousand times faster than guessing blindly.

⚠️ Warning If you aren't debugging a live run—meaning you don't have the base address from /proc/modules—you can still use objdump directly, just without the --adjust-vma parameter. In this case, you'll need to manually match the offset from the Oops (e.g., 0x124) against the relative addresses in the objdump output. The principle is exactly the same.

Aside: What if the Kernel Itself Crashed?​

If the culprit isn't a module but core kernel code, the method is the same. You just swap the input file for the uncompressed vmlinux image:

${CROSS_COMPILE}objdump -dS <path/to/kernel-src/>/vmlinux > vmlinux.disas

Of course, the prerequisite is that your vmlinux was compiled with debug information. This is a one-and-done effort—unless you update your kernel, you can keep reusing this disassembly file.

GDB: Querying Without a Disassembly File​

If you find manually calculating addresses or grepping through disassembly files too tedious, GDB offers a more "automated" way to query.

The prerequisite remains the same: your module must be compiled with debug symbols (i.e., MYDEBUG := y in the Makefile from earlier).

To make GDB debugging easier, our Makefile quietly adds a bunch of compiler flags when MYDEBUG is enabled:

ccflags-y += -DDEBUG -g -ggdb -gdwarf-4 -Og -Wall -fno-omit-frame-pointer -fvar-tracking-assignments

(-g is the key one; -Og provides optimization without excessive instruction reordering, making debugging easier)

Point GDB at your .ko file (note that this is .ko, not .o):

$ gdb -q ./oops_tryv2.ko
Reading symbols from ./oops_tryv2.ko...
(gdb) list *do_the_work+0x124

Note the syntax: list *函数名+偏移量. Here, 0x124 is the RIP offset we grabbed from the Oops message.

GDB will immediately spit out this section:

0x160 is in do_the_work (<...>/ch7/oops_tryv2/oops_tryv2.c:62).
  [...]
61              pr_info("Generating Oops by attempting to write to an invalid kernel memory pointer\n");
62              oopsie->data = 'x';
63         }
64         kfree(gctx);
(gdb)

A perfect hit. Line 62 again.

The beauty of GDB is that it understands file formats (ELF) and debug information (DWARF), saving you from doing hexadecimal addition and subtraction yourself. As long as you give it the correct module file and offset, it will take you straight to the scene of the crime.

addr2line: A Brute-Force Address Converter​

If you find GDB too heavy, or if you just want to write a script to batch-process addresses, addr2line is a lighter-weight alternative.

As the name implies, its sole function is: address -> file:line_number.

The usage is straightforward. You need to point it at the executable file (-e) and then give it an address (or multiple addresses):

$ addr2line -e ./oops_tryv2.o -p -f 0x124

Note: addr2line is typically used on .o files or symbolized ELF binaries. Sometimes .ko works too, depending on whether the symbols have been stripped. If .ko doesn't work, try the .o file generated during the build process.

Output:

do_the_work at <...>/ch7/oops_tryv2/oops_tryv2.c:62

Parameter explanation:

  • -e: Specifies the binary file.
  • -f: Displays function names.
  • -p: Pretty-prints the output format for better readability.

Still line 62. Different tool, same truth.

If the kernel crashed and you're holding a vmlinux along with a kernel virtual address, the usage is exactly the same:

addr2line -e </path/to/>vmlinux -p -f <faulting_kernel_address>

⚠️ Warning: The KASLR Pitfall There's a historical caveat here. If your system has KASLR enabled (Kernel Address Space Layout Randomization)—the security feature that randomizes the kernel base address on every boot—then addr2line goes blind.

Why? Because the address in the Oops is a "randomized absolute address," while the symbols in vmlinux are "compile-time relative addresses." addr2line has no idea what the random offset is for this particular boot.

When you run into this, you have two options:

  1. Disable KASLR: Add nokaslr to your boot parameters.
  2. Use faddr2line: This is a script specifically designed to handle KASLR, which we'll cover in the next section.

At this point, we've gone through the three classic tools for pinpointing an Oops location: objdump, GDB, and addr2line. Once you master these, you'll never have to stare blankly at hexadecimal addresses in an Oops message again.