The Authentication Header (AH) is a security protocol used within the IPsec suite. Its main purpose is to ensure that the message is not modified during transmission and that the data is coming from the expected source.
- It provides security by adding a header to the IP packet
- IP packet contains a cryptographic hash value and other information used for verification.
Functions
- Message Integrity: Ensures that the message is not modified during transmission from source to destination.
- Source Authentication: Confirms that the data is received from the correct and trusted sender.
- Protection Against Replay Attacks: Prevents attackers from capturing packets and sending them again later.
Authentication Header (AH) Format
The AH format consists of the following fields:

1. Next Header (8 bits)
- Identifies the type of protocol that comes after the AH header.
- Indicates whether the next part is TCP, UDP, ICMP or another protocol.
- Helps the receiving system correctly process the remaining packet.
2. Payload Length (8 bits)
- Specifies the length of the Authentication Header.
- Measured in 32-bit words.
- Helps the receiver determine where the AH ends and the payload begins.
3. Reserved (16 bits)
- Reserved for future use.
- Currently set to zero.
- Ensures compatibility with future protocol updates.
4. Security Parameters Index (SPI) (32 bits)
- Identifies the Security Association (SA) between sender and receiver.
- Used to determine which security settings and keys are applied to the packet.
5. Sequence Number (32 bits)
- Used to prevent replay attacks.
- Each packet is assigned a unique sequence number.
- If a packet is received with a repeated number, it is discarded.
6. Authentication Data (variable length, multiple of 32 bits)
- Contains the Integrity Check Value (ICV).
- Generated using a cryptographic hash function (HMAC).
- Ensures the packet has not been modified during transmission.
Working
Authentication Header works in IPsec to ensure the packet is authentic and not modified during transmission.
- Packet Creation: The sender creates a packet containing the IP header and data (payload).
- Addition of Authentication Header: Authentication Header (AH) is inserted between the IP header and the payload.
- Hash Value Generation: A cryptographic hash value (ICV) is generated using the payload and important fields of the IP header.
- Packet Transmission: The packet is transmitted from source to destination through the network.
- Change in Mutable Fields: Some IP header fields like TTL (Hop Count) may change during transmission, so these fields are not protected by AH.
- Verification at Receiver Side: The receiver recalculates the hash value using the same method.
- Integrity Check: If the calculated value matches the received value, the packet is accepted; otherwise, it is rejected.
- Replay Attack Protection: Sequence numbers are used to detect duplicate packets and prevent replay attacks.
Modes of Operation
1. Transport Mode
The Authentication Header is inserted between the original IP header and the transport layer header (TCP/UDP).
- Protects the payload and selected parts of the IP header
- Original IP header remains visible
- Used mainly for host-to-host communication
- Provides authentication and integrity, but no encryption
2. Tunnel Mode
The entire original IP packet (original IP header + data) is authenticated. A new outer IP header is added and the Authentication Header is placed between the new IP header and the original IP packet.
- Protects the entire original packet
- Inner IP header contains the actual source and destination IP addresses
- Outer IP header contains IP addresses of security gateways (firewalls or VPN routers)
- Commonly used in VPN communication
Handling Replay Attack
- Authentication Header in IPsec uses a Sequence Number to identify each packet.
- The sequence number starts from 0 and increases by 1 for every packet sent using the same Security Association (SA).
- The receiver maintains a sliding window of size 64 to keep track of valid packets.
- If a packet with a duplicate or old sequence number is received, it is rejected, which prevents replay attacks.