Skip to main content

Chapter 1: A New Perspective on Kernel Programming — Building Your Kernel Workspace from Scratch

There is a class of problems that appear to be "engineering problems" on the surface, but are actually problems of "privilege and perception."

Imagine a scenario: you write a very simple piece of code, trying to read from or write to a specific memory address, or wanting to capture a signal the moment a key is pressed. In User Space, you can't do it—the operating system will ruthlessly reject you, telling you Segmentation Fault or Operation not permitted. The OS acts like a stubborn gatekeeper, blocking you outside the wall and telling you: "It's dangerous in there, go play in the sandbox."

But what if you don't just want to "play in the sandbox," but actually want to design the foundation, plumbing, and wiring of the house? At that point, you can no longer be a managed "user"; you must become the operating system itself. This is the essence of Linux kernel programming—transitioning from a rule follower to a rule maker.

Our mission in this chapter is to tear down that wall.

We will set up a kernel lab that belongs to you, configuring the environment from scratch, and step by step, we will dive into the internals of one of the most complex open-source projects on the planet. This isn't just about writing code; it's about understanding the soul of a machine.

1.1 Why Learn Kernel Programming?​

Let's take a step back and ask the most fundamental question: why do we need to write kernel code directly?

Honestly, most day-to-day software development never needs to touch the kernel. If you're writing web apps, mobile apps, or desktop tools, the rich libraries in user space are more than enough to last you a lifetime. But when you start dealing with hardware, extreme performance, or low-level mechanisms, the boundaries of user space become your ceiling.

This isn't just about efficiency; it's about capability.

The traditional development model is a "request-response" model: your program asks the OS to do something, the OS schedules it, does it for you, and returns the result. This model is safe, but for certain tasks, it's too slow—or simply impossible. We need a more direct approach—shoving our code straight into the kernel and letting it run in kernel mode.

This brings us to the star of the show: LKM (Loadable Kernel Module).

You can think of an LKM as a "plug-and-play expansion card" attached to the kernel—like inserting a cheat cartridge into a game console.

But the "expansion card" analogy is wrong in one respect: a real expansion card is physically plugged in, whereas an LKM is a piece of binary code that is dynamically loaded into the kernel's address space at runtime, possessing the exact same privileges as the kernel (Ring 0). It's not just a plugin; it becomes part of the kernel. If this code is written incorrectly, it won't just crash a program—it will crash the entire system.

At this level, we no longer rely on so-called "system calls"—we are the system calls.

1.2 What Path Does This Book Take?​

There are plenty of books on the market, but this one is a bit different. Our goal isn't to make you memorize APIs; it's to help you build kernel intuition.

To achieve this, we've divided this long journey into three stages, with each stage laying the foundation for the next:

Stage 1: Building the Environment from Scratch (Part 1)​

This is where you stand right now. In this part, we tackle the most practical problem: "A workman must first sharpen his tools." We'll start by setting up a safe kernel development environment, using a Virtual Machine to isolate risks (after all, a kernel crash during debugging will reboot the machine). Then, we'll download and compile the latest Linux 6.1 LTS kernel source code with our own hands. We'll write your first kernel module, watch it load via the insmod command, and leave your first "Hello World" in the kernel log. The key here is getting your hands dirty—don't be intimidated by compilation errors; that's just the beginning.

Stage 2: Dissecting the Kernel (Part 2 — Sister Volume)​

This part is the deep end of the cognitive pool. We're going to tear open the machine that is the kernel. We'll dive deep into the Linux kernel architecture, understanding how the Task Structure (task_struct) manages every lifecycle, and how the user-mode stack and kernel-mode stack switch during a system call. We'll spend three entire chapters chewing on the tough bone of Memory Management—not just understanding the theory, but learning how to efficiently allocate and free memory inside the kernel. After all, there's no garbage collector here to clean up after you. Finally, we'll touch the core of CPU Scheduling to see how the OS decides who runs next.

Note: This book is Part 1. If you're interested in advanced driver development, hardware interrupt handling, and complex kernel synchronization mechanisms, be sure to read its sister volume, Linux Kernel Programming Part 2. That is the necessary path to becoming a professional kernel developer.

Stage 3: Mastering the Art of Chaos (Part 3 — Sister Volume)​

The kernel is highly concurrent. If the code you write deadlocks inexplicably on a multi-core processor, or data gets mysteriously overwritten, you've encountered a concurrency problem. We will learn kernel synchronization techniques here—spinlocks, mutexes, and atomic operations. This is the watershed that separates "script kiddies" from "kernel engineers."

1.3 Why Linux 6.1 LTS?​

You might ask: Kernel versions update so fast, why choose 6.1? Why not 5.x or the latest 6.x?

This is a very practical choice.

Linus Torvalds released version 6.1 in December 2022 and marked it as Long-Term Support (LTS). This means this version will be maintained by the kernel community for a full 4 years—until December 2026. For the lifecycle of a book, that's plenty long.

But that's not the most critical reason.

What's more critical is that the Civil Infrastructure Platform (CIP), an ultra-longevity organization dedicated to open-source software for critical infrastructure (like power and water systems), has selected 6.1 as its SLTS (Super LTS) version and plans to maintain it for over 10 years, until August 2033.

What does this mean? It means what you learn today will still be running in industrial settings a decade from now. This isn't an experiment; this is the future standard.

Returning to the "expansion card" analogy: Linux 6.1 LTS is like a rigorously tested standard interface that promises to plug into mainstream motherboards for the next ten years. Mastering it offers an incredibly high return on investment.

1.4 The Hands-On Philosophy: Get Your Hands Dirty or Go Home​

I want to emphasize one thing here: kernel programming is not an armchair exercise.

If you just sit and watch me write code, you'll think you understand. But the moment you open a terminal and face hundreds of lines of errors from make, or the incomprehensible register stack dump after a kernel Panic, you'll realize you didn't understand anything at all.

To avoid this, we stick to one principle: Be Empirical!

This book is paired with a GitHub code repository (Linux-Kernel-Programming_2E), which contains about 40 kernel modules (double the first edition!), along with companion user-space test programs and scripts. You need to clone this code, modify it, compile it, load it, try to break it, and then fix it.

⚠️ Pitfall Warning Don't just read! Whatever you do, don't just read! I've seen too many people who think they understand it, only to get an Invalid module format error from insmod the moment they try it. The hands-on process will expose your blind spots—you might forget to specify the correct kernel version number in the Makefile, or forget to add MODULE_LICENSE("GPL") in your code (which will cause the kernel to refuse to load your module). You must get moving. Your error logs are your best teachers.

1.5 Preparing Your "Operating Table"​

Alright, enough theory. Now we're going to set up the workspace. If you cut corners here, you'll struggle every step of the way later.

Technical Prerequisites Checklist​

Before we begin, please verify your setup:

  • Hardware: You need a sufficiently powerful PC or laptop. Don't use an antique machine with only 4GB of RAM. Compiling the kernel is an extremely memory- and CPU-intensive process. If you have 16GB of RAM and a quad-core (or better) CPU, that's ideal. Leave at least 30GB of disk space, because the intermediate files generated during compilation and the VM images take up a lot of room.
  • Operating System:
    • First choice: Native Linux System. This is the top choice for all veteran kernel developers—best performance, most direct.
    • Second choice: Virtual Machine. If you must work on Windows or macOS, this isn't just the only option; it's actually a safer option. If we crash the kernel inside a VM, at worst we kill the VM. Your host machine can still play videos just fine. Most of the examples and outputs in this book were captured on an x86_64 Ubuntu 22.04 LTS Guest VM (running on Oracle VirtualBox).

Installing the Guest VM​

The specific steps for this part (like how to create a new VM in VirtualBox, mount the Ubuntu ISO, configure partitions, etc.) are undeniably important. However, if I wrote down every single click in this book, half of it would be screenshots.

To save space, I've placed this detailed and verified installation guide in the book's GitHub repository and in the companion online chapter PDF.

Be sure to download and read this PDF guide: Download Chapter 1 PDF

In this guide, I'll walk you through step by step:

  1. How to create an Ubuntu 22.04 LTS virtual machine.
  2. How to configure networking and shared folders (to make it easy to transfer code between the host and the VM).
  3. The most important step: How to install all the dependency packages required for kernel compilation (such as build-essential, libncurses-dev, bc, flex, bison, etc.). Miss even one, and the compilation will halt at some inexplicable stage.

1.6 Getting the Code Repository: Your Arsenal​

Once you have the VM set up and can type ls and pwd in the terminal, the first thing to do is pull down the code repository.

All the source code for this book is hosted on GitHub. Open a terminal and type:

git clone https://github.com/PacktPublishing/Linux-Kernel-Programming_2E

This command will pull the entire code tree to your local machine.

You'll notice the code is organized by chapter:

  • The ch1/ directory contains the code for this chapter.
  • The root directory contains some common files, such as convenient.h (a header file containing common macro definitions) and klib.c (some common kernel helper functions).

Sharpening Your Tools: ctags and cscope​

Now you have tens of thousands of lines of kernel code and this book's example code at your fingertips. If you use grep or your editor's search function to jump around, your efficiency will be frustratingly low.

If you want to survive in kernel development, you must learn to use code indexing tools. I strongly recommend building ctags or cscope indexes for your codebase.

It's like installing radar on your code. For example, you can place your cursor on the do_fork function, press a single key, and your editor jumps directly to the function's definition—even if it's in another directory 500 files away.

Building the index is very simple. Navigate to the root of your source tree and run:

ctags -R

Or use cscope (which is often better when dealing with the massive kernel source code):

cscope -Rbkq

🎯 Pace Shift Right now, you might think this step is trivial. Wait until you're tracing a complex struct pointer reference, jumping back and forth between 10 files—you'll come back and thank this section.

1.7 Stepping Out of Your Comfort Zone​

Alright, the tedious part of environment configuration is almost over.

There is one detail to note here: regarding output consistency.

Unless otherwise stated, the code output and logs in this book were captured on an x86_64 Ubuntu 22.04 LTS environment. The Linux world has many distributions and fast iteration cycles, so the output you see might differ slightly from mine—that's normal. Even if changes in the Linux kernel itself cause some function names to change, don't panic.

This is inherently part of kernel programming: adapting to change and solving new problems.

Before wrapping up this stage, please confirm you've completed the following checklist:

  • Successfully installed and booted your Linux VM (or native system).
  • Installed all dependencies required for kernel compilation.
  • git clone the book's code repository.
  • Tried generating ctags/cscope indexes.
  • Downloaded and read the detailed online Chapter 1 PDF.

If you've got all of that done, congratulations—you're no longer a "user" of Linux. You're standing at the doorstep of being a "developer."

Next Stop: Building Your Kernel​

With the environment ready, we'll stop sharpening the knife and start cutting.

In the next two chapters, we'll accomplish something incredibly rewarding: downloading the Linux kernel source code from scratch, configuring it, compiling it, and then booting your system with it.

This is a moment where, when you see the black-and-white scrolling boot log, you won't be able to help but say, "Woah!" Your kernel journey truly begins now.

Chapter Recap:

Remember the question we posed at the beginning of this chapter—why transition from a "user" to a "kernel developer"? Through the groundwork laid in this chapter, the answer is actually quite clear: it's not to look cool, but to gain control. When you've configured the environment yourself and compiled your first module, you're no longer standing outside the OS's wall. What you hold in your hand is no longer a request letter, but a key.

In the next chapter, we'll insert this key into the lock and turn it.

Exercises​

Exercise 1: Understanding​

Question: In Linux kernel development, why is the Loadable Kernel Module (LKM) framework generally recommended for feature development or driver writing, rather than directly modifying the core kernel source code and recompiling? Please list one primary reason and explain.

Answer and Analysis

Answer: Because LKM allows kernel code to be dynamically loaded or unloaded without recompiling and rebooting the entire kernel.

Analysis: LKM (Loadable Kernel Module) is a mechanism designed by the Linux kernel to improve flexibility and extensibility. Without using LKM, any code modification, addition, or deletion (such as device drivers) would require recompiling the entire kernel source code and rebooting the system. This is highly inefficient during the development and debugging phase, and it also causes service interruptions in actual production environments. With LKM, developers can load functionality on demand and iterate quickly during debugging, greatly improving development efficiency.

Exercise 2: Application​

Question: Suppose you need to maintain a highly critical industrial control system that is expected to run continuously for over 10 years. Based on this chapter's content, which version of the Linux kernel should you choose for development? Please explain your reasoning.

Answer and Analysis

Answer: Choose Linux Kernel 6.1 LTS (specifically the SLTS version).

Analysis: As introduced in the chapter, Linux Kernel 6.1 was designated as an LTS (Long-Term Support) version by the community, with support until 2026. More importantly, the Civil Infrastructure Project (CIP) has designated it as SLTS (Super LTS) and plans to maintain it until 2033 (over 10 years). For an industrial control system requiring long-term operation, choosing the SLTS version ensures security patches and bug fix support throughout its entire lifecycle, avoiding the risks associated with frequent kernel upgrades.

Exercise 3: Application​

Question: As a kernel developer, you've just downloaded this book's companion source code via git clone. To efficiently find function definitions and variable references within the massive source code tree, which two tools should you use to generate code indexes? Please write out the specific commands.

Answer and Analysis

Answer: You should use ctags and cscope. The command to generate a ctags index is: ctags -R (executed in the source code root directory).

Analysis: The Linux kernel source code is massive in scale, and relying on text searches to find definitions is extremely inefficient. ctags and cscope are code browsing tools specifically designed for C that can parse code structure and generate tag (index) files. Running ctags -R in the root directory of the source tree recursively generates the index files, allowing editors (like Vim) to quickly jump to function definitions or find variable references. This is an essential skill for reading and debugging kernel code.

Exercise 4: Thinking​

Question: The chapter mentions that in terms of performance, a native Linux system is typically about twice as fast as a guest VM; however, this book strongly recommends that beginners perform kernel development experiments in a VM. From the perspectives of "security" and "fault tolerance," analyze why a VM is a more recommended working environment for kernel programming beginners.

Answer and Analysis

Answer: Because kernel code runs directly at the hardware privilege level, erroneous code can cause system crashes, data corruption, or hardware failures. A VM provides a layer of isolation.

Analysis: Thinking (Deep Dive):

  1. Isolation/Security: Kernel programming operates at Ring 0 (highest privilege). Any arbitrary pointer error or infinite loop in the code could directly hang the host operating system, leading to data loss or even affecting physical hardware. When running in a VM, a Guest OS crash only affects the VM itself; the host machine and other applications remain safe.
  2. Debugging Convenience/Fault Tolerance: In a VM, if the kernel crashes, you can easily roll back to a previous state via snapshots, or simply reboot the VM without needing to reboot the physical machine. Although there is a performance penalty, for isolating the inevitable bugs during the learning process and maintaining system stability, a VM provides a safer, lower-cost "sandbox" environment.

Key Takeaways​

The essence of kernel programming lies in transitioning from a "rule follower" to a "rule maker." By developing LKMs (Loadable Kernel Modules), we inject code directly into the kernel with Ring 0 privileges, breaking through the ceiling of user space in hardware control and performance optimization. However, this also means that code errors will directly crash the entire system.

The hands-on practice in this book will be based on the Linux 6.1 LTS version. We chose this version not only because it has a 4-year official community maintenance cycle, but more critically because it was selected by the CIP project as Super LTS (Super Long-Term Support), expected to serve in industrial settings until 2033. This ensures that the techniques learned today will retain immense industrial value for the next decade.

Building a safe kernel development environment is the first step of hands-on practice. Because kernel debugging mistakes can easily crash the host machine, we recommend using Oracle VirtualBox to install an Ubuntu 22.04 LTS VM as a sandbox, and you must reserve 16GB of RAM and 30GB of disk space to handle the extreme resource consumption of kernel compilation.

Mastering source code navigation tools is an essential skill for dealing with the kernel's massive codebase. After obtaining the book's companion code via git clone, you should not rely on simple text searches. Instead, immediately use ctags or cscope to build indexes for the source code, enabling cross-file function and struct jumps in your editor. This is a crucial capability for efficiently tracing kernel logic.

Kernel programming is a discipline that relies heavily on empiricism. Developers must abandon the passive learning mode of just reading books, follow the "get your hands dirty or go home" principle, and verify theory to build true kernel intuition by personally writing, modifying, and loading modules, and solving problems through repeated errors (like Panics or Invalid module format).