Skip to main content

Chapter 10: A Tunnel to Trust

There is a class of problems that appear to be network configuration issues on the surface, but are actually problems of trust.

That is exactly the kind of problem we will tackle in this chapter.

Imagine sitting in a café teeming with hackers and monitoring probes. You open your laptop, ready to connect to your company's intranet to check an urgent email. Theoretically, every bit you send could be intercepted, tampered with, or simply dropped by someone in the middle. In this scenario, the physical cables are untrusted, the routers are untrusted, and even that guy next door constantly typing away is untrusted.

Yet, you need to build a "completely trusted" channel over this entirely untrusted network. This sounds like a paradox—how can you use dirty water to wash a clean shirt?

The answer isn't to purify the entire network (that's impossible), but to put your data into an "armored car" that no one else can open. No matter how chaotic the outside world gets, the contents inside the armored car remain intact and confidential.

This is the core mission of IPsec. It is not only the cornerstone of enterprise VPNs, but also one of the most complex subsystems in the modern Linux network stack.

This chapter will be hardcore. We will start with user-space key negotiation and drill all the way down to the XFRM framework deep within the kernel, watching step-by-step as a packet is encrypted, encapsulated, decrypted, and restored. This goes far beyond simply configuring a few parameters—we need to understand how this mechanism builds a defensive line for network communication without sacrificing performance.

Ready? Let's start dismantling this armored car.

10.1 Overview: The Two Faces of IPsec​

IPsec has become the de facto standard for IP VPNs (Virtual Private Networks) in today's world.

Of course, the world isn't monolithic. Besides IPsec, you've likely seen SSL-based VPNs, or that ancient PPTP approach that stuffs a PPP connection into a GRE tunnel. But honestly, if you talk about "orthodox" VPNs in a Linux environment, people default to IPsec. It isn't a small patch running at the application layer; it's muscle grafted directly onto the skeleton of the IP protocol.

Within this massive protocol family, there are two core protocols you must distinguish right away: AH (Authentication Header) and ESP (Encapsulating Security Payload).

  • AH is responsible for exactly one thing: proving "no one has touched this packet." It provides integrity and authentication, but does not encrypt data. It's like sending a letter in a transparent sleeve—others can read it, but if it's opened or tampered with, the recipient can tell at a glance.
  • ESP is the real main event: it encrypts data while ensuring integrity. This is the most commonly used protocol in IPsec, and the star of our chapter. It's like an opaque safe—not only can others not open it, they can't even see what's inside.

Choosing a Mode: Transport vs. Tunnel​

Knowing which protocol to use is only the first step. Next, you have to decide "how to use it." IPsec has two distinctly different operating modes, and this choice determines what your packets look like.

1. Transport Mode

You can think of transport mode as a "courier wearing a bulletproof vest."

In this mode, the IP header remains untouched, and only the IP packet's payload is encrypted and authenticated. Routers can still read the IP header information to forward the packet, but the actual content is protected. This method is typically used for end-to-end encrypted communication. It is efficient but does not support virtual network addresses (because the IP header is in plaintext and must be usable).

2. Tunnel Mode

Tunnel mode, on the other hand, is like "loading the armored car into a train car."

In this mode, the entire original IP packet (including its IP header) is treated as a payload, encrypted, and then stuffed into a brand-new IP packet. The new IP header contains the gateway addresses.

This is why most VPNs use tunnel mode. To external routers, it just looks like an ordinary packet sent from Gateway A to Gateway B. As for whose data is inside, they have no idea. This mode allows private IP addresses (like 192.168.x.x) to traverse the public internet, because they are hidden behind the new IP header.

Naturally, there are exceptions. Some scenarios (like L2TP/IPsec) are also VPNs but use transport mode—this is because they already have their own tunnel layer, and IPsec is only responsible for encrypting the data within that tunnel.

Chapter Roadmap​

IPsec is not just a kernel module; it is a massive system spanning both user space and kernel space. To explain it clearly, I've drawn a map for you:

  • IKE (Internet Key Exchange): This is the "diplomat" in user space. Before actually transmitting data, both communicating parties must negotiate what keys and algorithms to use—this is key exchange. This job is highly complex and is usually handled by user-space daemons (like strongSwan or Openswan), not the kernel.
  • XFRM Framework: This is the "executor" in the kernel. Regardless of what user space negotiates, the results must ultimately be translated into structures the kernel understands (policies and states), which the XFRM framework then uses to perform encryption, decryption, and encapsulation.
  • Tx/Rx Paths: How packets enter the IPsec processing pipeline and how they are restored.
  • NAT Traversal: A mechanism dedicated to handling "accidents"—how IPsec survives in the cracks when it encounters a NAT device.

Next, we will start with the user-space IKE protocol and see how this "diplomatic negotiation" plays out.