Skip to main content

1.8 Seeing the Difference — Production vs. Debug Kernel Config Showdown

In the previous section, we turned on every debug switch we could find — which is great, but it raises a question: what exactly did we change? In other words, how much does all this y and n toggling actually divide this kernel from its production sibling?

This question matters more than you might think. If you don't know where the boundary lies, you might end up fixing a "debug-only bug" as if it were a universal issue, or worse — trying to load a module that only exists thanks to debug symbols in a production environment.

Let's make the invisible visible.

The Config Comparison Microscope: diffconfig​

Hidden in the kernel source tree is a handy script called scripts/diffconfig. It acts like a microscope, purpose-built to compare two .config files and magnify the differences for you.

Now, suppose we have two config file paths:

  • Production kernel config: ~/lkd_kernels/kconfig_prod01
  • Debug kernel config: ~/lkd_kernels/kconfig_dbg01

We run this script from within the debug kernel's source directory, piping the differences into a text file:

scripts/diffconfig ~/lkd_kernels/kconfig_prod01 ~/lkd_kernels/kconfig_dbg01 > ../../kconfig_diff_prod_to_debug.txt

Once that command finishes, you have a complete "health report."

On my system, this report exceeds 200 lines. That means the debug kernel isn't just "slightly" tweaked — it has undergone a genetic-level mutation.

Interpreting the Difference Report​

Open kconfig_diff_prod_to_debug.txt in an editor. You'll notice the lines fall into three categories, each telling a different story.

Category 1: Removed Features (Prefix -)​

Lines starting with - represent features that exist in the production kernel but were stripped out of our debug kernel.

-BPF_LSM y
-DEFAULT_SECURITY_APPARMOR y
-DEFAULT_SECURITY_SELINUX n
-DEFAULT_SECURITY_SMACK n
[ … ]

Seeing this list might give you pause: Wait, why would a debug kernel disable security modules?

It sounds counterintuitive, but there's a very practical trade-off at play. Certain security mechanisms (like some modules under the LSM framework) perform deep interception of system calls, which can severely slow down the system or even interfere with a debugger's normal tracing. On a kernel dedicated to hunting bugs, we sometimes have to temporarily lower these shields for the sake of performance and purity.

Category 2: Toggled Switches (n -> y or y -> n)​

This is the core section. It shows the options whose "state has flipped." You'll notice that the vast majority of arrows point toward y.

DEBUG_ATOMIC_SLEEP n -> y
DEBUG_BOOT_PARAMS n -> y
DEBUG_INFO n -> y
DEBUG_KMEMLEAK n -> y
DEBUG_LOCK_ALLOC n -> y
DEBUG_MUTEXES n -> y
DEBUG_PLIST n -> y
DEBUG_RT_MUTEXES n -> y
DEBUG_RWSEMS n -> y
DEBUG_SPINLOCK n -> y
[ … ]

Every single n -> y line means we've planted a pair of watching eyes in the debug kernel.

  • DEBUG_ATOMIC_SLEEP: Originally, sleeping in atomic context was forbidden; now it's not only forbidden, but the kernel will shout about it.
  • DEBUG_INFO: Originally, symbols were omitted to save space; now they're all included so gdb can understand what's going on.
  • DEBUG_LOCK_ALLOC, DEBUG_MUTEXES, DEBUG_SPINLOCK: Lock-related checks are fully enabled.

Here, you'll also see the name suffix we specifically modified earlier:

LKDTM n -> m
LOCALVERSION "-prod01" -> "-dbg01"
LOCK_STAT n -> y
MMIOTRACE n -> y
MODULE_SIG y -> n
[ … ]
  • LOCALVERSION changed from -prod01 to -dbg01. This is exactly why you see 5.10.17-dbg01 instead of 5.10.17-prod01 in the boot log.
  • MODULE_SIG changed from y to n. There's a subtle point here: production kernels typically require module signatures (y) to prevent loading malicious modules; but debug kernels usually turn forced signing off (n) to make it easier to tinker with test modules. Otherwise, loading a test module you wrote yourself would require wrestling with signature certificates — essentially asking for unnecessary pain.

Category 3: Added Features (Prefix +)​

Lines starting with + represent entirely new entries that don't exist in the production kernel config at all.

+ARCH_HAS_EARLY_DEBUG y
+BITFIELD_KUNIT n
[ … ]
+IKCONFIG m
+KASAN_GENERIC y
[ … ]

Pay attention to that last one, KASAN_GENERIC. This is Kernel Address SANitizer, a heavy-hitter in kernel debugging specifically designed to catch nasty memory bugs like out-of-bounds accesses and use-after-free. It drastically slows down the system (potentially by 2x or more), so it's absolutely forbidden in production kernels. In a debug kernel, however, it's priceless.

Back to Reality: Your Project Might Be Different​

Although we demonstrated a "standard" production vs. debug pairing approach here, I have to be honest:

Specific kernel configurations are merely representative.

Your project or product might have a bunch of peculiar requirements. Maybe your production kernel needs certain special real-time patches, or maybe your debug environment needs to run on a quirky virtualization platform.

Additionally, if you're working on modern embedded Linux, you're probably not typing make menuconfig on the command line — you're using a heavyweight build system like Yocto or Buildroot.

If that's the case, the "manual .config editing" skills you just learned will need to be applied elsewhere. In Yocto, you typically need to write a bbappend recipe file to inject your debug configuration. That's a deeper rabbit hole, but the core logic remains the same: one is a trimmed-down athlete, the other is a test subject covered in sensors.

Final Check​

At this point, I'm secretly hoping you've actually been following along — meaning you should now really have two compiled kernels in hand: one production, one debug.

If you don't, I strongly recommend going back and running the compilation step now. Don't just read without practicing.

If all went well, you should now have your custom 5.10 LTS production and debug kernels, ready for battle. In the upcoming chapters, we'll actually boot them and see how they behave differently when things go wrong.

Before that, let's use the final section to equip you with a few pieces of "close-fitting debugging gear."