Skip to main content

10.5 Receiving an IPsec Packet (Transport Mode)

In the previous section, we paused at the end of initialization at xfrm4_rcv()—the receive entry point that the ESP protocol registered with the kernel. The pipeline is laid out, so what kind of journey does a packet actually take when it flows in?

Let's assume this is an IPv4 ESP transport mode packet destined for the local machine.

10.5. Receiving an IPsec Packet (Transport Mode)​

As we mentioned in the previous section, xfrm4_rcv() is registered as the shared entry point for ESP, AH, and even IPCOMP protocols. This is a clever design: since the very first thing they all do after entering is table lookup and replay attack verification, it makes sense to consolidate this dirty work in one place.

So, when this ESP packet arrives at ip_local_deliver_finish(), the kernel sees that the protocol field in the IP header is 50 (ESP) and directly calls xfrm4_rcv(). This function does nothing else but hands the packet straight to the generic XFRM processing center—xfrm_input().

This is the place where the "encryption shell" truly gets cracked open.

Step 1: Look Up the Key in the SAD​

After xfrm_input() gets the packet, the first thing it does isn't rushing to decrypt. Instead, it verifies one thing: have we agreed with the peer on how to decrypt this packet?

This requires a lookup in the Security Association Database (SAD). The lookup keys are the SPI (Security Parameter Index), destination address, and protocol number. If the lookup fails, it means the packet was either misdirected or blindly sent, so it gets dropped immediately:

/* 在 state_byspi 哈希表中执行查找 */
x = xfrm_state_lookup(net, skb->mark, daddr, spi, nexthdr, family);

/* 如果查找失败，静默丢弃数据包 */
if (x == NULL) {
    XFRM_INC_STATS(net, LINUX_MIB_XFRMINNOSTATES);
    xfrm_audit_state_notfound(skb, family, spi, seq);
    goto drop;
}

If the corresponding SA (x == NULL) isn't found in this step, the kernel silently drops the packet and increments a counter, XFRMINNOSTATES. This kind of "silent failure" is common at the network layer, because you can't send an ICMP error message back for an unknown encrypted packet—you have no idea who sent it, and you can't decrypt its contents.

Step 2: Protocol-Specific Decryption Callback​

If we're lucky and the SA is found, the x pointer will point to a valid xfrm_state structure. Next, the kernel calls the .input callback function registered by the protocol associated with this SA (e.g., ESP).

For the IPv4 ESP scenario we're discussing, this x->type is the esp_type registered in the previous section, and its .input callback points to esp_input():

/* 调用对应协议的 input 方法（这里是 esp_input），
   返回值 nexthdr 是加密前原始数据的协议号 */
nexthdr = x->type->input(x, skb);

esp_input() calls the Crypto API to perform decryption and authentication (AEAD operations). If the cryptographic verification fails, the packet is dropped; if it succeeds, the ESP header and ESP trailer are stripped away, leaving behind the original IP payload.

This function returns something important: nexthdr. This is the protocol number of the original packet inside, after stripping off the ESP encapsulation (for example, TCP is 6, UDP is 17).

The kernel carefully stores this protocol number and tucks it into the control buffer (cb) of skb. Because the data area of skb is now decrypted, but the original IP header hasn't been updated yet, we need to remember it for when we fix the IP header later:

/* 将原始协议号保存在 SKB 的控制缓冲区中，
 * 稍后修改 IP 头时会用到 */
XFRM_MODE_SKB_CB(skb)->protocol = nexthdr;

Step 3: Cleanup and Disguise (Transport Finish)​

Decryption is done, and now we're holding a "restored" packet: its data portion is in plaintext, but its IP header still says "protocol=ESP" (50). This obviously won't work—the transport layer (L4) would be completely confused if it received this packet.

So, xfrm_input() finally calls xfrm4_transport_finish() to perform the final cosmetic surgery.

Let's see what this function does:

int xfrm4_transport_finish(struct sk_buff *skb, int async)
{
    struct iphdr *iph = ip_hdr(skb);

    /* 此时 iph->protocol 还是 50 (ESP)，
     * 我们要把它改成解密后原始包的协议号（如 TCP/UDP），
     * 这样 L4 协议栈才能认领这个包 */
    iph->protocol = XFRM_MODE_SKB_CB(skb)->protocol;
    
    /* 调整 skb 的指针，因为解密后包的长度可能变了，
     * 需要让 IP 头部的指针回到正确的位置 */
    __skb_push(skb, skb->data - skb_network_header(skb));
    
    /* 更新 IP 头部的总长度字段 */
    iph->tot_len = htons(skb->len);

    /* 重新计算 IP 头部校验和，因为我们改了 protocol 和 tot_len */
    ip_send_check(iph);

This section is critical. You can think of it like unboxing a package: esp_input is responsible for taking the items (data) out of the box, while xfrm4_transport_finish is responsible for changing the information on the shipping label (IP header) back to what the original product looked like.

Modifying the IP header is mandatory, because this packet is about to re-enter the subsequent stages of the network stack. If we don't change it, the kernel will think this is still an ESP packet and keep sending it to the encryption engine, resulting in an infinite loop.

Step 4: Reinject into the Protocol Stack​

After the disguise is complete, this packet is logically a normal, unencrypted IP packet. To let it finish the rest of its journey (reaching the TCP or UDP socket), the kernel throws it back into the receive path of the protocol stack.

This step is implemented via a Netfilter hook:

    /* 调用 netfilter 的 PRE_ROUTING hook，
     * 然后通过 xfrm4_rcv_encap_finish 继续流转 */
    NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, skb->dev, NULL,
            xfrm4_rcv_encap_finish);
    return 0;
}

xfrm4_rcv_encap_finish() ultimately calls ip_local_deliver(). At this point, the protocol field in the IP header is already TCP or UDP, and the kernel handles it just like a normal packet freshly arrived from the NIC, passing it to the upper-layer protocol.

With this, the receive journey of an IPsec transport mode packet truly comes to an end.

From the outside, it looks like an ESP packet; once inside xfrm_input(), it gets looked up and decrypted; when it comes out, it has changed its appearance and turned into a normal IP packet, silently blending into the regular data flow. This is the essence of transport mode—it doesn't alter the original routing path, it simply quietly hardens the transmission process.