Skip to main content

10.3 Writing a Custom Kernel Panic Handler

In the previous section, we dissected the standard procedure during a kernel panic like bomb disposal experts—from printing the last words to deciding whether to reboot. We even learned how to use the panic_print tuning knob to control how much information the kernel spits out before it dies.

This is a lot like observing a black box: you know it died, and you got the death report.

But as an engineer digging deep into the system, you can't help but wonder: Can I intervene the exact moment it dies?

For example, on an embedded device, a kernel panic means the system is out of control. At that point, you might need to immediately shut down a running motor, light up a red LED to tell the operator "the system is hung," or even send a final alert to a monitoring server via a backup channel. The kernel's own panic() function won't do any of this—it only cares about its own death, not your business logic.

To achieve this, we need a mechanism that lets us "hook" our own code into the kernel's death hooks.

The Linux kernel provides a powerful infrastructure for this: Notifier Chains.

A Step Back: Understanding the Notifier Chain Mechanism​

Don't rush to write code; let's first understand how this mechanism works. You're probably familiar with the publish-subscribe pattern—it's not a deeply complex concept, just like subscribing to a YouTube channel and getting notified when the creator uploads a new video.

Notifier chains in the kernel do exactly this.

  • Publisher: A place in the kernel where a critical event occurs (e.g., a network interface comes online, the system is about to reboot, or the kernel crashes).
  • Subscriber: Modules interested in these events (e.g., device drivers).
  • Subscription Action: Registering a callback function onto a linked list.

When the event occurs, the publisher traverses this list and calls all registered callback functions in order. You can even set priorities to decide whether you get notified first (VIP seat) or line up at the back.

But Reality Isn't That Simple—Four Types of Chains​

Although intuitively simple, the kernel is highly sensitive to execution context. What context is your callback running in? This determines the type of notifier chain.

The kernel defines four types of notifier chains, differing in whether they can sleep and what locking mechanism they use:

  1. Atomic: Callbacks run in atomic context. They absolutely must not block or sleep. Internally protected by a spinlock. If you try to call kmalloc(..., GFP_KERNEL) or use a semaphore here, the system will crash even harder immediately.
  2. Blocking: Callbacks run in process context and are allowed to sleep. Internally uses a read-write semaphore to implement blocking behavior.
  3. Sleepable RCU (SRCU): These also run in process context and allow sleeping. However, they use a more advanced RCU mechanism to achieve lock-free semantics. It's suited for scenarios where "callbacks are extremely frequent, but unregistration is very rare."
  4. Raw: Context is undefined—it might sleep, it might not. There are no safety guards; the caller is entirely responsible for safety.

⚠️ Analogy Time: The Bulletin Board You can think of a notifier chain as a bulletin board in an office building lobby (Publish/Subscribe).

  • Atomic bulletin board: You're a special forces soldier passing by; you can only glance at it while running. You can't stop, and you can't crouch down to tie your shoes.
  • Blocking bulletin board: You can stop, even pull up a chair and sit down to read slowly. No one will bother you even if you stare at it for an hour.
  • SRCU bulletin board: This is like a high-tech electronic board that only consumes power when you're actually looking at it. You can space out while reading, but unregistering is a hassle.

Back to Reality: The chain we're dealing with now is panic_notifier_list. Take a guess—when the kernel crashes, the system is in an extremely unstable state. Interrupts might be disabled, scheduling might be stopped—which type do you think this is?

The answer is Atomic. At a time like this, if you dare to sleep, the system won't even have a chance to report an error to you.

To Hook In, We Need to See What the "Hook" Looks Like​

We'll be using a predefined atomic notifier chain in the kernel: panic_notifier_list.

Here is how it's defined in the kernel source:

// kernel/panic.c
ATOMIC_NOTIFIER_HEAD(panic_notifier_list);
EXPORT_SYMBOL(panic_notifier_list);

The ATOMIC_NOTIFIER_HEAD macro generates a list head for you and exports the symbol, meaning our kernel module can directly manipulate it.

Registration and Unregistration: The Partner Principle​

To hook ourselves in, we use this API:

int atomic_notifier_chain_register(
    struct atomic_notifier_head *nh, 
    struct notifier_block *nb
);

It's actually a wrapper that protects your registration action internally with spin_lock_irqsave. The parameters are straightforward:

  1. nh: Points to the list head we want to register with—in this case, &panic_notifier_list.
  2. nb: Points to a notifier_block structure, which is your "ID card."

Of course, if there's registration, there must be unregistration. Although in a Panic scenario, the module's cleanup function is highly unlikely to be called (because the system has stopped), as a good coding practice, we still call the unregistration function when the module exits:

int atomic_notifier_chain_unregister(
    struct atomic_notifier_head *nh, 
    struct notifier_block *n
);

Core Data Structure: struct notifier_block​

This is the soul of the entire mechanism. Its definition is very clean:

// include/linux/notifier.h
struct notifier_block {
    notifier_fn_t notifier_call;
    struct notifier_block __rcu *next;
    int priority;
};
  1. notifier_call: Function pointer. This is the callback function you write. When a kernel panic occurs, execution jumps here.
  2. next: The next node in the linked list. Set to NULL during initialization; the kernel framework handles it for you.
  3. priority: Priority. A higher number means it gets called earlier. If you set it to INT_MAX, you get to be first in line. Usually, you can ignore this; the default of 0 is fine.

What Does the Callback Function Look Like?​

Your callback function must match this signature:

typedef int (*notifier_fn_t)(
    struct notifier_block *nb, 
    unsigned long action, 
    void *data
);

These three parameters are clues passed to you by the kernel:

  1. nb: Points to your own notifier_block. This is useful when your callback needs to handle multiple registrations, typically used to retrieve the parent structure containing this structure (using container_of).
  2. action (also called val): An unsigned long that tells us "why" it was triggered. For the Panic chain, it's usually a value from the architecture-specific enum die_val. 
    // arch/x86/include/asm/kdebug.h
    enum die_val {
        DIE_OOPS = 1,
        DIE_INT3, DIE_DEBUG, DIE_PANIC, DIE_NMI, 
        // ...
    };
    You can use this value to determine whether the Panic was caused by a regular Oops or by an NMI (Non-Maskable Interrupt).
  3. data: A pointer to a struct die_args. This is packed with useful info: 
    // include/linux/kdebug.h
    struct die_args {
        struct pt_regs *regs; // CPU 寄存器快照
        const char *str;      // 也就是 panic() 里的那个字符串！
        long err;
        int trapnr;
        int signr;
    };
    Usually, what people care about most is the str in data—the specific reason description that triggered the Panic.

Closing Note: How Do You Tell the Kernel You're Done?​

Your callback function must return a specific macro to tell the kernel framework your processing result:

  • NOTIFY_OK: Processing done, everything is fine (the most commonly used return value).
  • NOTIFY_DONE: Processing done, I don't care about subsequent notifications.
  • NOTIFY_STOP: Processing done, stop calling other callbacks on the chain (this is a bossy choice).
  • NOTIFY_BAD: An error occurred, don't continue notifying others (equivalent to a veto).

For a Panic handler, just obediently return NOTIFY_OK.

Hands-on: Writing Our Panic Monitor Module​

With the theory laid out, let's put the pieces together. We're going to write a module that screams when the kernel crashes.

1. Define the Module Header and notifier_block​

Pay attention to the license choice. Because the atomic_notifier_chain_register API is exported as GPL-only, if your module only declares MIT or Proprietary, the kernel will reject this "freeloading" behavior, resulting in an undefined symbol error. Using Dual MIT/GPL or GPL will solve the problem.

// ch10/panic_notifier/panic_notifier_lkm.c
/* 必须包含合适的 LICENSE，否则无法使用 GPL-only 的 API */
MODULE_LICENSE("Dual MIT/GPL");
MODULE_AUTHOR("Your Name");
MODULE_DESCRIPTION("A custom kernel panic handler demo");

/* 定义我们的通知块 */
static struct notifier_block mypanic_nb = {
    .notifier_call = mypanic_handler,
    /* .priority = INT_MAX  如果你想抢在所有人前面跑，解开这行 */
};

2. Write the Callback Function​

This is our core logic. Here we both print information and reserve space for actual hardware operations.

/* 模拟一个硬件动作：比如拉响警报 */
/* 在真实产品中，这里可能是写寄存器关电机、开继电器等 */
static void dev_ring_alarm(void)
{
    pr_emerg("!!! ALARM !!!\n");
}

/* 实际的回调处理函数 */
static int mypanic_handler(
    struct notifier_block *this, 
    unsigned long val, 
    void *data)
{
    struct die_args *args = (struct die_args *)data;

    /* pr_emerg 是最高优先级的 printk，通常能刷到控制台 */
    pr_emerg("\n************ Panic : SOUNDING ALARM ************\n"
             "val = %lu\n"
             "data(str) = \"%s\"\n", 
             val, 
             (args && args->str) ? args->str : "unknown");

    /* 执行我们的硬件保护逻辑 */
    dev_ring_alarm();

    return NOTIFY_OK;
}

There's a detail to note here: data might be NULL (though rare during a Panic), so it's best to do a defensive check before accessing args->str. For brevity, we used a ternary operator inside pr_emerg for this check.

3. Module Initialization and Exit​

Finally, hook it up.

static int __init panic_notifier_lkm_init(void)
{
    if (!atomic_notifier_chain_register(&panic_notifier_list, &mypanic_nb)) {
        pr_info("Custom panic handler registered.\n");
        return 0;
    }
    return -EFAULT; /* 注册失败极少见，但以防万一 */
}

static void __exit panic_notifier_lkm_exit(void)
{
    atomic_notifier_chain_unregister(&panic_notifier_list, &mypanic_nb);
    pr_info("Custom panic handler unregistered.\n");
}

module_init(panic_notifier_lkm_init);
module_exit(panic_notifier_lkm_exit);

Game On!: Verifying Our Hypothesis​

The code is written. Now we need to artificially trigger a Panic to see the effect.

⚠️ Warning The following operations will cause your virtual machine to crash immediately. Make sure you have saved all important work, and do not attempt this on a production machine.

Step 1: Prepare the Network Receiver​

Because the local screen might not have time to display anything during a system crash—or there might be no screen at all—we'll continue using the netconsole and netcat configured in the previous section to receive logs.

Make sure you are listening on your host machine (or another machine):

# 接收端命令
nc -d -u -l 6666

Step 2: Load Our Module in the Virtual Machine​

Compile the module and insert it into the virtual machine:

# 在虚拟机里
insmod ./panic_notifier_lkm.ko
# 检查日志确认注册成功
dmesg | tail

You should see the text Custom panic handler registered..

Step 3: Trigger the Disaster​

We wrote a small script to do this. Its logic is:

  1. Set panic_on_oops to 1 (to escalate an Oops to a Panic).
  2. Enable SysRq.
  3. Trigger a Crash via SysRq.
$ cat ../cause_oops_panic.sh
sudo sh -c "echo 1 > /proc/sys/kernel/panic_on_oops"
sudo sh -c "echo 1 > /proc/sys/kernel/sysrq"
sync; sleep .5
sudo sh -c "echo c > /proc/sysrq-trigger"

Run this script.

Result: Witnessing Death​

If everything goes well, your virtual machine will instantly freeze. Go back to your receiver terminal, and you should see a massive flood of logs from netcat pouring out.

In that ocean of familiar garbled text and stack traces, you should be able to clearly spot the mark we left behind:

[   ...] ************ Panic : SOUNDING ALARM ************
val = 1
data(str) = "SysRq : Trigger a crash"

[   ...] Kernel panic - not syncing: SysRq : Trigger a crash
[   ...] ---[ end Kernel panic - not syncing: SysRq : Trigger a crash ]---

See that? val = 1 corresponds exactly to DIE_OOPS (or DIE_PANIC, depending on the architecture's enum definition), and data(str) is precisely the command string we triggered.

This proves that before the kernel completely breathed its last breath, our mypanic_handler was successfully executed.

Final Words of Advice: Don't Screw It Up​

Although we can insert custom code, there is a very important constraint: keep it extremely minimal.

When a Panic occurs, the kernel is in an incredibly fragile state. Spinlocks might already be held, the scheduler has stopped working, and interrupts might be disabled. If your Panic handler tries to acquire a lock that's already held, or attempts to sleep, the system might hang inside your handler. This could even prevent kdump (crash dump) from being generated, leaving you unable to determine the cause of death.

The comments in the kernel source code have already warned about this:

// kernel/panic.c:panic()
/*
 * 注意：因为某些 panic_notifier 可能会让崩溃的内核更加不稳定，
 * 这也会增加 kdump 失败的风险。
 */

So, inside this callback, only do things that are absolutely necessary: log a few register values, toggle a GPIO state (like turning off a laser), or write a reboot command to a special hardware register. Don't do complex string processing, don't allocate memory, and don't wait on mutexes.

We have now achieved our goal: not only do we know the kernel died, but we also equipped it with a "dying words recorder." In the next section, we will face a more insidious form of death—the system didn't crash, but it hung.