Skip to main content

10.7 NAT Traversal in IPsec

In the previous section, we were enjoying the sense of certainty brought by xfrm_lookup() — the policy matched, the state was found, the encryption completed, and the packet was sent out. Everything looked great.

But the real world loves to slap you in the face at moments like this.

Imagine this: you've set up an IPsec VPN at home, connecting to your company's server. At home, you're a machine behind NAT (e.g., 192.168.1.5), and your router has a public IP. When you eagerly try to bring up the tunnel, you find it either fails to establish or drops immediately after coming up.

Why? Because in the eyes of that NAT device in the middle — the one responsible for "translating" addresses — your ESP packets are complete gibberish.

Why NAT Devices Break ESP​

Let's look at this from the NAT device's perspective. The core job of NAT is simple: rewrite source/destination IP addresses (and sometimes ports), then fix up the checksums in the TCP or UDP headers.

The keyword here is fix up the checksums.

When the transport layer (TCP/UDP) calculates checksums, it includes the source and destination IP addresses from the IP header (known as the Pseudo-Header). So, once NAT changes the IP address, it must synchronously update the TCP/UDP checksum. Otherwise, the checksum calculated by the receiver won't match, and the packet gets dropped immediately.

In transport mode, ESP encrypts the entire TCP/UDP header.

This is awkward.

A NAT device looking at an ESP packet faces a dilemma:

  1. It wants to change the IP address.
  2. It wants to update the TCP/UDP checksum.
  3. But the TCP/UDP header is encrypted — it can't read it, let alone calculate the checksum.

Result: The NAT device can only do one of two things: either forward the packet as-is (which usually fails because private IPs can't route out), or — in the vast majority of cases — drop the packet outright. This causes IPsec to fail completely in NAT environments.

But not all protocols have this problem. SCTP, for example, has a checksum that doesn't cover the IP header, making it "immune" to NAT changes. Unfortunately, what we primarily use day-to-day are TCP and UDP.

To solve this problem, the IETF came up with a standard (RFC 3948) with a very straightforward name — UDP Encapsulation of IPsec ESP Packets. This is what we commonly call NAT-T (NAT Traversal).

Its core idea is simple: since you don't recognize ESP, I'll just disguise it as your favorite protocol — UDP.

The Rules of NAT-T​

Before diving into the code, let's highlight a few key points to save you from some pitfalls:

  1. ESP only, not AH: The AH protocol performs integrity checks on the IP header itself. Once NAT changes the IP address, the AH verification will fail. Therefore, NAT-T only works with ESP. If you're still using AH, there's basically no solution in a NAT environment.
  2. No manual keys: NAT-T cannot be used with manual keys. It must rely on the negotiation mechanism of IKEv1 or IKEv2. Why? Because enabling NAT-T requires a handshake confirmation between both parties, and this process is part of the IKE protocol.
  3. Software-level compromises: Although NAT-T is an IPsec matter, many P2P applications (especially VoIP) actually deal with similar problems. Protocols like STUN, TURN, and ICE all exist to solve the problem of "how to establish a connection from behind a NAT." strongSwan even implements a TURN-like relay service, allowing two clients both behind NAT to establish a connection through a third party.

⚙️ How NAT-T Works​

The goal of NAT-T is very clear: forcefully insert a UDP header between the IP header and the ESP header.

This way, the NAT device sees a standard UDP packet. It can change the IP, change the port, and calculate the UDP checksum — everyone is happy.

1. Enabling and Negotiation​

This step happens in user space, but the kernel must cooperate.

In older software (like Openswan), you had to explicitly write nat_traversal=yes in /etc/ipsec.conf to activate this feature. In strongSwan's IKEv2 daemon Charon, however, NAT-T is enabled by default and cannot be turned off — the author clearly believes that in today's network environment, NAT is the norm, and not supporting NAT-T is just shooting yourself in the foot.

The negotiation process goes roughly like this:

  • Step 1: Confirming Capability During the Main Mode phase of IKEv1, both parties exchange ISAKMP messages. If they support NAT-T, they include a specific marker in the Vendor ID field. In IKEv2, this is already part of the standard, so no special declaration is needed.

  • Step 2: NAT Detection Both parties send each other NAT-D (NAT Discovery) payloads. This is essentially a probe packet: I know my own IP, and I know the IP I see for you. If these values don't match, or if the IP addresses along the path have changed, we know there's a NAT device in the middle "messing things up."

Once it's confirmed that there's NAT on the path, both parties immediately switch to NAT-T mode.

2. UDP Encapsulation Structure​

After switching modes, an ESP packet that originally looked like this:

[ IP Header ] [ ESP Header ] [ Encrypted Data ] [ ESP Trailer ] [ ICV ]

Now looks like this:

[ IP Header ] [ UDP Header ] [ ESP Header ] [ Encrypted Data ] [ ESP Trailer ] [ ICV ]

Key details:

  • Port numbers: Both the source and destination ports in the UDP header are 4500. This is the NAT-T dedicated port assigned by IANA.
  • Keep-Alive mechanism: NAT mappings have a timeout. If there's no traffic for a long time, the NAT device will delete the mapping. To prevent this, NAT-T requires sending a Keep-Alive packet every 20 seconds.
    • What the Keep-Alive looks like: It's a packet sent to UDP port 4500 with a payload of exactly one byte, with the value 0xFF. When the kernel receives this packet, it knows this is just "keeping the connection alive" — no decryption is needed, and it doesn't need to be delivered to the upper layers.

3. Kernel-Space Receive Path (xfrm4_udp_encap_rcv)​

When this "disguised" UDP packet arrives at the receiver's kernel, what happens?

When we looked at the IPsec receive path before, the packet went directly from the IP layer (the callback registered by ip_protocol) into the ESP processing logic. But that won't work now, because the IP layer sees a UDP protocol, and the kernel will treat it as a normal UDP packet and hand it to the UDP stack.

After receiving the packet, the UDP stack checks whether this is an "encapsulated" packet.

In net/ipv4/xfrm4_input.c, there's a function specifically designed to handle this case: xfrm4_udp_encap_rcv().

Its logic is actually very straightforward (Mode B):

  1. Intercept: Before processing the data, the UDP stack calls this function.
  2. Strip the disguise: The function checks if the UDP port is 4500, then inspects the UDP payload.
    • If the payload is 0xFF (Keep-Alive), it drops it immediately. Job done.
    • If it's an encapsulated ESP packet, it calls xfrm4_rcv_encap().
  3. Hand off to ESP: xfrm4_rcv_encap() strips the UDP header and passes a pointer to the ESP header to the normal IPsec receive path (xfrm_input()).

To the kernel, after stripping the UDP header, the packet is indistinguishable from a normal ESP packet. The remaining decryption, verification, and reassembly processes all reuse the logic we covered in the previous section.

Returning to the "translator" analogy:

You can think of NAT-T as putting an encrypted document inside a standard shipping envelope. The post office (NAT) only looks at the address on the envelope (IP and UDP port). It doesn't care what's inside the envelope — as long as the address is correct, it handles the delivery. Only when the envelope reaches its destination does the recipient (the kernel) tear open the envelope and take out the encrypted document for processing.

The sole purpose of this envelope (the UDP header) is to fool the post offices along the way. While this might seem like a roundabout approach, given the reality of IPv4 address exhaustion and NAT being everywhere, it's the most practical solution.

At this point, we've connected all the dots of IPsec's core mechanisms — from the underlying XFRM framework, to the integration of cryptographic algorithms, to the interplay between policies and states, and finally the UDP encapsulation necessitated by traversing NAT.

This isn't just a bunch of protocols and code; it's a complex engineering system that evolved to establish secure channels over insecure networks. Even for a design like NAT that "breaks" the internet's end-to-end principle, IPsec can adapt by adding a layer of encapsulation. This kind of compromise and adaptation is, perhaps, the true norm of the engineering world.