3.4 Wielding the Kernel's Dynamic Debug Powerhouse

In the previous section, we saw that trace_printk() is a lifeline for high-frequency paths. But this creates a dilemma: for debugging, we need to print; for performance, we dare not print too much—especially in production environments.

This dilemma exists fundamentally because we have been using "compile-time" switches (like #ifdef DEBUG) to control "runtime" behavior.

It's like the airplane is already in the sky, but you have to go back to the hangar on the ground to adjust the wings. Too slow, too clunky.

If only we could freely toggle any pr_debug() statement at runtime.

This isn't just a wishful thought—it's reality. The kernel has a facility called Dynamic Debug built exactly for this purpose. But to truly appreciate its benefits, we need to first look at its predecessor: the module parameter approach.

3.4.1 Prelude to Dynamic Debug: Module Parameters

Before Dynamic Debug became widespread, the most common trick for kernel developers was to use a module parameter to control the debug switch.

This approach is very simple, and you can still see it in many existing driver codebases today.

The idea is straightforward: define a debug variable, defaulting to 0 (off). If it's 1 (on), print the logs.

static int debug;
module_param(debug, int, 0644);

Behind this single line of code, the kernel creates a file in sysfs for you: /sys/module/<模块名>/parameters/debug

Hands-on: See how i8042 does it

The classic i8042 keyboard/mouse controller driver uses this exact trick. Open its source code (drivers/input/serio/i8042.c), and you'll see:

// drivers/input/serio/i8042.c
static bool i8042_debug;
module_param_named(debug, i8042_debug, bool, 0600);
MODULE_PARM_DESC(debug, "Turn i8042 debugging mode on and off");

There's a small detail here: it uses module_param_named. This allows you to use the variable name i8042_debug in your code, but expose it as debug in sysfs. This is a good naming practice.

How do we use it?

First, you can use modinfo to see what parameters a module has:

$ modinfo -p /lib/modules/$(uname -r)/kernel/drivers/hid/hid.ko
debug:toggle HID debugging messages (int)
ignore_special_drivers:Ignore any special drivers and handle all devices by generic driver (int)

Once the module is loaded, you can directly manipulate that sysfs file (note the permissions are 0600, so only root can play with it):

$ ls -l /sys/module/i8042/parameters/debug
-rw------- 1 root root 4096 Oct  3 07:42 /sys/module/i8042/parameters/debug

# 查看当前状态（默认是 N，即关）
$ sudo cat /sys/module/i8042/parameters/debug
N

# 打开它
# echo "Y" > /sys/module/i8042/parameters/debug

# 关闭它
# echo "N" > /sys/module/i8042/parameters/debug

This looks perfect, right? No need to recompile, dynamically toggled.

Advanced trick: Multi-level verbosity

You can even turn that debug parameter into an integer and build a "verbosity" tier system:

0: Completely silent
1: Only print critical paths
2: Print all details
...

Then your code ends up filled with if (debug > 1) pr_debug(...).

⚠️ But there's a catch

Although this works and is flexible, it has a fatal flaw: performance overhead.

Think about it: even when debug=0 (off), every time the CPU reaches that if statement, it still has to make a conditional check. If this function is called a million times per second (which is common in the kernel), you're doing a million useless checks.

Furthermore, the code gets cluttered with all sorts of if (debug > x), making it ugly and a nightmare to maintain.

Is there a way to dynamically toggle like a module parameter, but with zero overhead when turned off?

Yes. This is why Dynamic Debug is called a "powerhouse."

3.4.2 Introducing Dynamic Debug

Dynamic Debug (Dyndbg) follows a completely different design philosophy:

Instead of adding if in the code, it directly modifies the code itself.

This sounds like black magic, but the principle is simply the kernel's "dynamic code patching" technique (the same underlying tech used by ftrace). When you disable a log point in Dyndbg, the kernel directly NOPs out the assembly instructions or jumps over them. Overhead? Nearly zero.

Prerequisites

To use this feature, your kernel configuration must have CONFIG_DYNAMIC_DEBUG enabled. Modern distribution kernels usually have this turned on. If you're working with a custom trimmed embedded kernel, remember to enable this option.

As a side note, this will increase the kernel size slightly (by about 2%). If you are extremely concerned about size, you can enable just the core functionality CONFIG_DYNAMIC_DEBUG_CORE, and then add ccflags-y += -DDYNAMIC_DEBUG_MODULE in the Makefile of the modules that need it.

What can it control?

Not just pr_debug(). As long as a macro uses the Dynamic Debug mechanism, Dyndbg can manage it:

pr_debug()
dev_dbg()
print_hex_dump_debug()
print_hex_dump_bytes()

3.4.3 Finding the Console: debugfs and procfs

The console for Dynamic Debug is a virtual file. Depending on your kernel configuration, it can appear in one of two places:

Ideal scenario (debug kernel): The debugfs filesystem is mounted. The control file is located at: /sys/kernel/debug/dynamic_debug/control
Production environment (hardened): For security reasons, production environments often prohibit mounting debugfs (CONFIG_DEBUG_FS_DISALLOW_MOUNT=y). In this case, the control file "falls back" to procfs: /proc/dynamic_debug/control

How to tell?

First, try mount | grep debugfs. If there's no output, use the /proc/... path. Don't panic—the two are functionally identical; only the path differs.

What does this file look like?

The contents of this file are usually huge. This is because it lists all print points in the kernel marked for dynamic debugging.

# wc -l /sys/kernel/debug/dynamic_debug/control
3217 /sys/kernel/debug/dynamic_debug/control

On my debug kernel, there are over 3,000 control points. Let's just look at the first few lines:

# head -n5 /sys/kernel/debug/dynamic_debug/control
# filename:lineno [module]function flags format
drivers/powercap/intel_rapl_msr.c:151 [intel_rapl_msr]rapl_msr_probe =_ "failed to register powercap control_type.\012"
drivers/powercap/intel_rapl_msr.c:94 [intel_rapl_msr]rapl_msr_read_raw =_ "failed to read msr 0x%x on cpu %d\012"
sound/pci/intel8x0.c:3160 [snd_intel8x0]check_default_spdif_aclink =_ "Using integrated SPDIF DMA for %s\012"

Decoding the format

The first line is the header: # filename:lineno [module]function flags format

Each subsequent line is a "print point." Let's break down the second line:

drivers/powercap/intel_rapl_msr.c:151 [intel_rapl_msr]rapl_msr_probe =_ "failed to register powercap control_type.\012"

Breaking it apart:

File name: drivers/powercap/intel_rapl_msr.c:151 This is the source code path and line number. You can go directly to this line to see the code.
Module name: [intel_rapl_msr] The text in square brackets is the name of the module containing this code. If it's core kernel code (vmlinuz), there might be no square brackets.
Function name: rapl_msr_probe The function where this print point resides.
Flags: =_ This is the most critical part. It determines whether the print point is on or off, and what additional information to print.
Format string: "failed to register powercap control_type.\012" This is the string passed to printk (\012 is the octal encoding for a newline character).

Verifying against the source

To prove this file isn't made up, let's look at line 151 in the source code:

// drivers/powercap/intel_rapl_msr.c
   rapl_msr_priv.control_type = powercap_register_control_type(NULL, "intel-rapl", NULL);
   if (IS_ERR(rapl_msr_priv.control_type)) {
       pr_debug("failed to register powercap control_type.\n");
       return PTR_ERR(rapl_msr_priv.control_type);
   }

A perfect match.

3.4.4 Mastering the Flags

What does that =_ mean?

= means "set," and _ means "no flags." So =_ means "no features enabled, i.e., disabled."

You can change it to =p (enable printing), or add more prefixes. This is the most powerful aspect of Dynamic Debug: it doesn't just control on/off; it controls the "verbosity" of the logs.

Flag	Meaning	Effect
p	Print	Enables the print point. If you don't add `p`, the other flags (`mftl`) won't take effect.
f	Function	Includes the function name in the log prefix.
l	Line number	Includes the line number in the log prefix.
m	Module	Includes the module name in the log prefix.
t	Thread ID	Includes the Process ID / Thread ID (PID/TID) in the log prefix.
_	No flags	Does nothing (default state).

Operators

You will use three symbols to combine these flags:

= : Set. =p means "only turn on p, turn off everything else."
+ : Add. +p means "add p to the current flags."
- : Remove. -p means "remove p from the current flags."

Examples

Assume the initial state is =_ (all off).

Enter +p -> state becomes =p (enabled).
Enter +t -> state becomes =pt (enabled, and showing thread ID).
Enter -p -> state becomes =t (showing thread ID, but printing is turned off—which is pointless).
Enter =pflt -> state directly becomes =pflt (printing enabled, showing function, line number, and thread).

3.4.5 Hands-on: Debugging a Module in Production

Theory is cheap without practice. Let's experiment with a simple misc driver.

Scenario setup: We have a miscdrv_rdwr driver. When compiling it, we intentionally do not define the DEBUG macro. We are running it on a production kernel (5.10.60-prod01). Now, a bug suddenly appears, and we need to see its logs.

Step 1: Prepare the environment and module

First, check whether debugfs is mounted.

$ mount | grep -w debugfs
# (没输出)

As we expected, debugfs isn't mounted on the production kernel. No problem, we'll use the /proc control file.

Let's load the module first (skipping the compilation process, assuming you have the compiled .ko file):

# insmod miscdrv_rdwr.ko
# dmesg | tail
[...] miscdrv_rdwr:miscdrv_rdwr_init(): LLKD misc driver (major # 10) registered

The module loaded successfully.

Step 2: Confirm the "DEBUG undefined" state

Let's write some data to the device and see if there are any debug logs.

# echo "test" > /dev/llkd_miscdrv_rdwr
# dmesg | tail
[...] misc llkd_miscdrv_miscdrv_rdwr:  filename: "/dev/llkd_miscdrv_rdwr"

We only see one pr_info level log (that filename is printed by pr_info). All those dev_dbg in the code have vanished. This is the expected behavior when DEBUG is undefined.

Step 3: Use Dynamic Debug to "see through" the code

Now, without recompiling or rebooting, we'll make those dev_dbg visible.

First, let's see what points our module has in this control file:

# grep "miscdrv_rdwr" /proc/dynamic_debug/control
<...>/miscdrv_rdwr.c:303 [miscdrv_rdwr]miscdrv_rdwr_init =_ "A sample print via the dev_dbg(): driver initialized\012"
<...>/miscdrv_rdwr.c:242 [miscdrv_rdwr]close_miscdrv_rdwr =_ "filename: \042%s\042\012"
[...]

See that? The code that was supposedly "dead" because DEBUG was undefined is actually alive, just in a dormant state (=_).

Step 4: Enable it

Time to act. We want all logs from the miscdrv_rdwr module to print.

# echo -n "module miscdrv_rdwr +p" > /proc/dynamic_debug/control

Note: Here we used the module keyword. This is one type of Match Spec. We'll go into details later.

Now let's look at the control file again:

# grep "miscdrv_rdwr" /proc/dynamic_debug/control | head -n1
<...>/miscdrv_rdwr.c:303 [miscdrv_rdwr]miscdrv_rdwr_init =p "A sample print via the dev_dbg(): driver initialized\012"

Look! =_ has changed to =p.

Step 5: Verify the effect

Operate the device again:

# echo "test" > /dev/llkd_miscdrv_rdwr
# dmesg | tail
[...] misc llkd_miscdrv_miscdrv_rdwr:  filename: "/dev/llkd_miscdrv_rdwr"
[...] miscdrv_rdwr:miscdrv_rdwr:open_miscdrv_rdwr(): 001)  bash :1080   |  ...0   /* open_miscdrv_rdwr() */
[...] miscdrv_rdwr: misc llkd_miscdrv_miscdrv_rdwr: opening "/dev/llkd_miscdrv_rdwr" now; wrt open file: f_flags = 0x8241

Wow!

Those silent dev_dbg from earlier are all firing now. You can even see the function name, file name, and some internal state (f_flags).

And we did this entirely on a running production kernel.

Step 6: Try something advanced (Match Spec + Flags)

Let's turn it off, then turn it back on with a more advanced approach. This time, we don't just want to see the logs; we also want to see which process is operating it (add the t flag) and the module name (add the m flag).

# 先关掉
# echo -n "module miscdrv_rdwr -p" > /proc/dynamic_debug/control

# 再开，加上 p, t, m
# echo -n "module miscdrv_rdwr +ptm" > /proc/dynamic_debug/control

Operate it one more time:

# echo "advanced test" > /dev/llkd_miscdrv_rdwr
# dmesg | tail
[...] [1080] miscdrv_rdwr: miscdrv_rdwr:open_miscdrv_rdwr(): 001)  bash :1080   |  ...0   /* open_miscdrv_rdwr() */

Notice the beginning of the log: [1080]. This is PID 1080 (our bash process) operating the device. This is an absolute lifesaver when debugging multi-threaded race conditions.

3.4.6 Dynamic Debug Syntax Cheat Sheet

That echo "module miscdrv_rdwr +ptm" line from earlier might look like magic. It actually follows a very strict syntax.

Command format:

echo -n "<match-spec> <flags>" > <control-file>

There are two parts here: Match Spec (who to match) and Flags (what to change).

1. Flags (what to change) We've already covered this: =pmflt, +p, -t, etc.

2. Match Spec (who to match) The Match Spec determines which print points your command will modify. You can snipe a single line of code with extreme precision, or wildly enable an entire module.

Commonly used Match Spec keywords:

Keyword	Syntax Example	Purpose
module	`module miscdrv`	Matches all print points in the module named `miscdrv` (supports wildcards like `*`).
file	`file drivers/tty/*`	Matches files whose path matches the pattern.
func	`func usb_submit_urb`	Matches the function name.
line	`line 100-150`	Matches a line number range (`100-150` means lines 100 to 150).
format	`format "DMA:"`	Matches format strings containing a specific substring.

Combination tricks

You can combine multiple specs together; their relationship is AND.

# 只打开 snd 驱动里，函数名包含 'ctl' 的，并且行号在 600 之前的日志
echo -n "module snd func *ctl* line 1-600 +p" > /proc/dynamic_debug/control

It's just like writing a database query, except you're querying kernel code.

3.4.7 The Production Roadblock: Kernel Lockdown

Now, there is one more potential issue.

If your production kernel has Kernel Lockdown enabled (CONFIG_LOCK_DOWN_KERNEL), you might find that you can't load modules or manipulate Dynamic Debug.

The purpose of Lockdown is to protect kernel security, preventing anyone (even root) from loading unsigned modules or modifying kernel memory. This is usually mandated by security teams.

How to bypass it?

If you are the system administrator, you have the right to temporarily disable it at boot.

Method: Modify Bootloader parameters

Reboot the machine, press e at the GRUB screen to edit the boot entry, and append the following to the end of the linux line:

lockdown=none

Then press Ctrl+x to boot.

Or modify it permanently in /etc/default/grub (add lockdown=none to GRUB_CMDLINE_LINUX_DEFAULT), and then run update-grub.

⚠️ Warning: This reduces security. Remember to change it back after debugging!

3.4.8 Debug at Boot? Boot Parameters

Dynamic Debug isn't just for modules loaded after insmod; it can also be used for core kernel code and built-in modules.

If you want to debug the kernel boot phase (like initcalls during initialization), you can't wait until after boot to write to the /proc/... file, because by then it's already too late.

You need to tell the kernel what you want at boot time.

Trick 1: Debug built-in modules / core kernel

Add the following to the kernel cmdline in GRUB:

dyndbg="module mydriver +p"

Or something more complex:

dyndbg="file drivers/usb/* +pflmt"

Trick 2: Debug loadable module initialization

If you want to debug the exact moment a module loads (like modprobe xxx), you can stuff the command into the /etc/modprobe.d/ configuration.

Create /etc/modprobe.d/mydriver.conf:

options mydriver dyndbg=+pmflt

This way, every time modprobe mydriver is called, systemd will automatically turn on Dyndbg for you.

You can even do it directly in the cmdline:

mydriver.dyndbg="file mydriver.c +pmft"

3.4.9 Bonus: Other Useful Kernel Parameters

Since we're on the topic of boot parameters, here are a few more lifesavers:

Parameter	Purpose
debug	Enables all kernel debug messages (beware of log floods).
ignore_loglevel	Ignores the loglevel and spits all messages directly to the console. Extremely noisy, but can save your life in critical moments.
initcall_debug	Prints the execution time and return value of every initcall. A must-have for diagnosing boot hangs.

If the kernel hangs at some point during boot, add initcall_debug, and you'll see exactly which initcall it stopped at. The culprit is right there.

3.4.10 Wrapping Up

By this point, we are fully armed.

From the simplest printk, to the tiered pr_debug, to the rate-limited ratelimited, and finally to the god's-eye view of Dynamic Debug.

You are no longer a repairman blindly swinging a hammer (printf) everywhere. You are an engineer equipped with precision instruments, capable of probing deep inside the kernel without stopping the machine.

But what about extreme cases where even printk is too slow, or when the kernel isn't far enough along to print? In the next chapter, we'll dive into deeper debugging mechanisms.

3.4.1 Prelude to Dynamic Debug: Module Parameters​

3.4.2 Introducing Dynamic Debug​

3.4.3 Finding the Console: debugfs and procfs​

3.4.4 Mastering the Flags​

3.4.5 Hands-on: Debugging a Module in Production​

3.4.6 Dynamic Debug Syntax Cheat Sheet​

3.4.7 The Production Roadblock: Kernel Lockdown​

3.4.8 Debug at Boot? Boot Parameters​

3.4.9 Bonus: Other Useful Kernel Parameters​

3.4.10 Wrapping Up​