10 - 1
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
1
IP Security
IPsec Protocols
This course was originally authored by Jean Triquet, a senior communications specialist
with Public Works and Government Services of Canada. In the last two years, he
participated in a project for the implementation of secure remote access services for the
organization. The technology of choice was Virtual Private Network (VPN) through IPsec.
In this course, the student will learn how Internet Protocol (IP) (an inherently unsecure
protocol) can be secured using the IPsec suite of protocols which focus on security via the
network (IP) layer. The Internet Protocol is the layer that is responsible for networking and
has no built-in mechanisms for prevention of alteration or eavesdropping of data in transit.
IPsec can assist in ensuring the integrity and privacy of the data that is sent.
IPsec is one of three widely used VPN protocols. A VPN essentially offers private network
connectivity between two communicating hosts using shared public links, such as the
Internet. Before the use of VPN’s the only way that hosts could privately communicate
was by using hardwired dedicated links. But, now with protocols such as IPsec, these
communications don’t require dedicated private links. You’ll see how this is done using
IPsec.
10 - 2
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
2
Outline
• IPSEC - security services
• Security Association (SA)
• Internet Key Exchange (IKE)
• The Authentication Header (AH) protocol
• The Encapsulating Security Payload (ESP) protocol

-Integrity
-Access control
-Partial protection against traffic flow analysis
Public IP
Network
Security
Gateway
Host
Security
Gateway
Host
Private IP
Network
Private IP
Network
Look at slide “IPsec – Security Services” to see the network diagram representing the different type of
relationships that can be established with IPsec.
IPsec can be implemented between two hosts, two security gateways or between a host and a security
gateway. A host can be any network device, for example a computer or a router, even a firewall. A
firewall can also be a security gateway. The difference between a host and a security gateway, for the
purpose of IPsec, resides in how IPsec is used by the network device. If the device uses IPsec to secure
communications between itself (its own IP stack) and another IPsec node, then it’s considered a host. If
the network device uses IPsec to secure communications coming from and going to a network segment,
than it is a security gateway (for IPsec purposes).
IPsec is a set of protocols which, when combined, offer security over IP networks. The protocols work
at the IP layer, protecting all the upper layers (TCP, UDP, ICMP). The IPsec suite consists of three
protocols:
Internet Key Exchange (IKE)
Authentication Header (AH)
Encapsulating Security Payload (ESP)

192.168.30.57.1038 > 192.168.167.40.389
: S 395784:395784(0) win …..
192.168.167.40.389
> 192.168.30.57.1038: S 1757781809:1757781809(0)
ack 395785 win…..
Key and security parameters exchanges
192.168.30.57.500
> 192.168.167.40.500: udp 990 (ttl 128, id 37896)
192.168.30.57.500
> 192.168.167.40.500: udp 92 (ttl 128, id 38152)
IPsec traffic
192.168.30.57 > 192.168.167.40: ip-proto-50
132 (ttl 128, id 32522)
192.168.30.57 > 192.168.167.40: ip-proto-50
132 (ttl 128, id 32778)
Before we start studying in details the various components of IPsec, let’s take a look at some IPsec
traffic through the eyes of tcpdump. It will give us an overview of the IPsec behavior. On the slide,
“Tcpdump Output of an IPsec “Discussion”, we see extracts of a tcpdump trace from a remote PC,
192.168.30.57, establishing an IPsec link with a security gateway, 192.168.167.40. The traffic is
divided in three major parts. In the first two tcpdump lines, we see the TCP port number 389 in the
trace. That indicates an LDAP request. Therefore, we have an authentication activity occurring
between the two IPsec nodes and a X.500 directory service part of a Public Key Infrastructure. This
corresponds to the primary authentication services which must take place before any IPsec process can
begin. Authentication could also be something as simple as the manual exchange of keys through
email, face-to-face, etc.
The next segment of traces shows UDP port number 500 in the tcpdump lines three and four. This well-
known port number is assigned to the ISAKMP protocol. We will study this protocol later in the class.
This portion of network traffic corresponds to the negotiations about the security policies, the
encryption keys and other ancillary parameters. It indicates that Security Associations are being
created.

- ©2000, 2001
7
Security Associations
Unique Identification
•
For the Security Associations concept to work
Each active Security Association created for an IPsec session must
be uniquely identified
• The unique identification is made up of the following information
• SPI, the Security Parameter Index
• The destination IP address
• A security protocol (AH or ESP) identifier
Slide “Security Associations Unique Identification” describes that we have learned that to create a
secure link between two IPsec peers, we have to create Security Associations (one on each IPsec peer).
It is possible to have multiple SAs between two peers or one peer may have multiple SAs with multiple
peers. Consequently, each SA must be uniquely identified.
The unique identification is built from the combination of:
-The Security Parameter Index (points to the right SA in the Security Association database); the SPI is
a random integer number 32 bits long, chosen by the receiving end of a SA. The device initiating an
IPsec negotiation has no idea of what SPI is already being used by the recipient of the request.
Therefore, to avoid duplications, it is left to the recipient to choose the SPI.
-The destination address
-The security protocol identifier, it may be AH(IP protocol 51) or ESP (IP protocol 50).
10 - 8
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
8
Security Associations
A Sample Network
10.0.1.0

IPsec – SANS GIAC LevelTwo
- ©2000, 2001
9
Security Associations
Security Policy Database (SPD)
"Secret"
IPsec "ESP DES HMAC MD5 MINUTES 300
or ESP 3DES HMAC MD5 MINUTES 300"
ISAKMP "DES MD5 MINUTES 1440"
”Top Secret"
IPsec "ESP 3DES HMAC MD5 MINUTES 300"
ISAKMP "DES MD5 MINUTES 1440"
The first SA database, the Security Policy Database (SPD) is described in slide “Security Associations
Security Policy Database (SPD)”.
The slide presents examples of different level of security with different security parameters options
which provide the means to enforce the generic policy formulated on the previous slide. That’s what
the SPD is used for, to list various set of security parameters which will be mapped against specific IP
addresses or network segments.
The security parameters must include the security protocol to be used as well as the encryption and
authentication algorithms. The lifetime of the SA also needs to be specified. Other parameters can be
included, usually parameters that will be used for the normal processing of the encryption or
authentication algorithm.
Let’s examine the security associations for what this site has called a “Secret” association to see what
this policy actually means. The IPsec protocol used will be ESP, the encryption will be done using
DES or 3DES, the authentication algorithm will HMAC MD5, and the security association will be good
for a maximum of 300 minutes or 5 hours. The key exchange will be done using DES encryption,
MD5 authentication and the keys will be good for 1440 minutes.
10 - 10
IPsec – SANS GIAC LevelTwo
- ©2000, 2001

If we look at the tables on the slide, on the Security Gateway B, the one protecting 10.0.0.0, we would
have this configuration information somewhere. For example, the local (protected) network 10.0.1.0
selects the processing mode Apply IPsec and the security policies labeled Secret. On the previous
slide, we listed the policy database entry for a “secret” association.
In the bottom table, we have the 192.168.0.0 network’s perspective. To be able to communicate with
the network 10.0.1.0, its security gateway will have to Apply IPsec, a security level Top Secret and the
tunnel end point to reach that network is Security Gateway B.
Somehow, all this information must be present. However, how it is implemented is very different from
one product to another.
10 - 11
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
11
Security Associations
The Security Association Database (SAD)
SPI Source Destination Tunnel End
Point
Protocol Encryption Authentication

12345 10.0.1.0 192.168.0.0 Sec. Gateway A ESP DES HMAC-MD5
678910 10.0.2.0 192.168.0.0 Sec. Gateway A ESP 3DES HMAC-MD5

The Security Association Database represents the active
Security Associations on an IPsec device.
Let’s examine a second database on slide “Security Associations The Security Association Database
(SAD)”. Two IPsec peers use their SPD to negotiate their secure relationship. When the negotiations
are over and successful, the Security Associations are established and an entry is made in the Security
Association Database (SAD). This database keeps track of all the active SAs. Again, here also the
format of that “database” is quite different form one product to another but the information presented is
the same.

one method of sending traffic between two hosts. There are two modes that can be used to exchange
IPsec connections. One is the transport mode, the other is the tunnel mode.
Let’s assume the IPsec host 10.0.0.10 wishes to send an IP packet to the IPsec host 192.168.0.10 and
they have established transport mode SAs. The resulting IP packet will be built with an IP header
containing the source 10.0.0.10 and the destination 192.168.0.10. The source address and destination
won’t change during the transit.
The transport mode can be used only between hosts. IPsec requires security gateways to use tunnel
mode.
10 - 13
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
13
Security Associations
Tunnel Mode
192.168.0.10
Public IP
Network
Security
Gateway A
Security
Gateway B
Payload
IP header
Source Destination
Tunnel IP header
Source Destination
IP address = 192.67.1.1
IP address = 10.0.1.1
10.0.1.1 192.67.1.1 10.0.010 192.168.0.10
192.168.0.0

– Key management for the SA
The slide “Internet Key Exchange” introduces what we will study in this section as the method used
to establish an active Security Association.
The protocol developed to perform the function is the Internet Key Exchange, IKE. It is a hybrid
protocol which uses ISAKMP and a subset of the Oakley authenticated key exchange method.
ISAKMP and Oakley will be discussed in more detail on the next slide.
The protocol performs the function of a negotiator. It will allow two IPsec nodes to decide on which
algorithms they will use for authentication and encryption as well as how long this will last.
Before the IKE protocol can start, some primary authentication services must be performed. These are
not part of the IKE protocol but are worth mentioning briefly. The first method is the exchange of
secrets between the two nodes. A secret is a password consisting of alpha-numeric characters or
whatever the IPsec implementation used supports. The second is the use of public keys. In this case,
before any IKE starts, the retrieval of the nodes certificates/keys will have to be done prior to IKE.
When everything is in place, the negotiation begins and SAs are established. When this is done, the
protocol is not finished. It will come back in action from time to time to manage the keys for the IPsec
session.
10 - 15
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
15
Internet Key Exchange
Phases and Modes
•IKE has two phases and three modes of operation.
•In phase one, IKE nodes negotiate and establish a secure
link that will be used for the phase two.
•In phase two, the real negotiations are done. The IPsec SA
is created.
•There are three IKE modes: main mode, aggressive mode and
quick mode.
•The main and aggressive modes are used to execute IKE

offering a nominal set of security services: data origin authentication, connectionless integrity and, if
required, protection against replay attacks.
So how does it work? The use of AH is determined by the Security Associations negotiated between
two IPsec nodes. In the security descriptors, there will be a section describing which parameters to use
with AH, like the authentication algorithms required, (HMAC MD5 or HMAC SHA1). The two nodes
will initiate an IKE session and when that session is completed, the two nodes will talk AH.
For the duration of the Security Associations, every IP packet will be modified to add an AH header in
the packet. This header will authenticate the packet, assuring the receiver of the legitimate source of the
packet. Basically, this provides a cryptographic checksum for the contents of the packet. If the
receiver of the packet can authenticate that the cryptographic checksum is correct, this assures that the
packet has not been altered in transit.
10 - 17
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
17
Authentication Header
The Header Structure
Next header
Payload length
Reserved
Security Parameters index (SPI)
Authentication data
Sequence number
07
15 23
31
Look at slide “Authentication Header the Header Structure” to examine the AH. We’ll discuss
where the Authentication Header field is found in the packet in the next slide. The Next Header is an
8-bit field. It is used to identify the next payload type. For example, if the next payload is TCP, this
field would be 0x06.

Data
original IP
header
TCP
When it comes in from a tunnel with AH...
Data
original IP
header
TCPAH
Authenticated
(except for mutable fields in the
New IP header)
New IP header
And After...
Turning to slide “Authentication Header Inbound Processing”, we contrast the inbound datagram
composition using the transport and tunnel modes. When an IPsec node receives an IP packet, it
searches for information on how to handle the packet. With the help of the IP protocol number
presented in the IP header, the IPsec implementation learns that the packet is an AH type. With that
information, it extracts the SPI number and tries to associate the packet with an active Security
Association.
If it can not find one, the packet is discarded. If it finds the appropriate SA, it will determine what
authentication algorithm has been used and will perform the authentication of the packet. When this is
done, if the packet passes the authentication test, the packet is sent to the upper layer protocol or to the
original IP destination, in the case of a tunneled packet.
Mutable fields are those that will change in transit as the packet travels from source to destination.
These are fields such as the Time To Live that is decremented by each router and the IP checksum
which must be recomputed and changed after the TTL changes.
10 - 19
IPsec – SANS GIAC LevelTwo
- ©2000, 2001

Slide “Authentication Header Outbound Processing” looks at the outbound datagram composition
using both the transport and tunnel modes. When an IPsec node sends a packet on the network, the
following happens, a Security Association lookup is performed first to determine if the IP packet needs
protection. The Security Association policy database will give that information. Let’s assume the
packet requires IPsec to be applied and it requires the use of AH for the implementation of the security
services.
On the slide, the first example is the case of a transport mode SA. The AH header is inserted right after
the IP header. The authentication is then performed on the whole packet, excluding the IP header fields
mutable in transit and the ICV field. In fact, the IP header fields are not really excluded but their values
are replaced by zeros. The ICV is then calculated on the resulting packet (without the ICV, of course)
and inserted in the final packet.
The second example represents what happens if the packet is controlled by a tunnel mode SA. In this
case, a new IP header is created, the AH is inserted right after it, and right before the original IP header.
The authentication applies to the whole packet again, with the exclusion of the IP header mutable
fields. In a AH tunnel, the original source and destination IP addresses are just authenticated, not
hidden, as is the case with ESP. We’ll discuss that later.
10 - 20
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
20
Authentication Header
FTP Traffic Through IPsec - the Setup
Ethernet
Ethernet
IP network
CA/X.500 (PKI)
FTP server
10.0.0.10
IPsec security gateway
Remote PC

<mss 1460> (ttl 126, id 55050)
192.168.0.10.1036 > 10.0.0.10.ftp-data: S 126568:126568(0) ack 522326775
win 8760 <mss 1460> (DF) (ttl 128, id 15105)
10.0.0.10.ftp-data > 192.168.0.10.1036: . ack 1 win 8760 (ttl 126, id 55306)
10.0.0.10.ftp-data > 192.168.0.10.1036: P 1:12(11) ack 1 win 8760
(ttl 126, id 55562)
10.0.0.10.ftp-data > 192.168.0.10.1036: F 12:12(0) ack 1 win 8760
(ttl 126, id 55818)
Turn to slide “Bypass IPSec – FTP Exchange A Tcpdump Trace (1)” to see a tcpdump trace of a file
transfer from the FTP server, 10.0.0.10, to the FTP client, 192.168.0.10. The security gateway policies
are as follows. For both inbound and outbound traffic bypass IPsec. It is a plain “in the clear” FTP
session, the security gateway acting only as a gateway.
By examining the traffic, you see all the information that an FTP session would show with no IPsec
nodes involved. You can see the ports used; all the IP header fields are being displayed. Another
interesting fact that we see is the real IP address of the FTP server, (no tunnel here!). Our tcpdump
collecting host is situated between the client and the gateway. On the next slide, we will compare this
traffic with IPsec/AH traffic.
10 - 22
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
22
Authentication Header
A Tcpdump Trace (2)
192.168.0.10 > 192.168.0.1: ip-proto-51 90 (DF) [tos 0xd] (ttl 128, id 65281)
192.168.0.1 > 192.168.0.10: ip-proto-51 94 (DF) (ttl 127, id 16135)
192.168.0.10 > 192.168.0.1: ip-proto-51 80 (DF) [tos 0xd] (ttl 128, id 2)
192.168.0.1 > 192.168.0.10: ip-proto-51 129 (DF) (ttl 127, id 16391)
192.168.0.1 > 192.168.0.10: ip-proto-51 68 (DF) (ttl 127, id 16647)
192.168.0.10 > 192.168.0.1: ip-proto-51 68 (DF) (ttl 128, id 258)
192.168.0.1 > 192.168.0.10: ip-proto-51 64 (DF) (ttl 127, id 16903)

The Whole Packet (1)
Internet Protocol
Version(MSB 4 bits): 4
Header length(LSB 4 bits): 5 (32-bit word)
Service type: Precd=Routine,Delay=Normal,Thrput=Normal,Reli=Normal
Flags: May be fragmented,Last fragment,Offset=0 (0x00)
Transmission Control Protocol
Port File Transfer (Default Data) ---> 1034
Sequence Number: 3904510
Acknowledgement Number: 266981
Header Length(MSB 4 bits): 5 (32-bit word)
Reserved(LSB 4 bits): 0
Code: ACK,PSH,
File Transfer Protocol
hello World
IN HEXADECIMAL
0000: 00 00 86 14 7c 1f 00 a0 90 00 4c 51 08 00 45 00 | ....|..._.LQ..E.
0010: 00 33 bc 01 00 00 7e 06 70 32 c6 67 65 89 c6 67 | .3....~.p2.ge..g
0020: 1e 39 00 14 04 0a 00 3b 93 fe 00 04 12 e5 50 18 | .9.....;.t....P.
0030: 22 38 5f e9 00 00 68 65 6c 6c 6f 20 57 6f 72 6c
| "8_e..hello Worl
0040: 64 | d
This slide “ Bypass IPsec – FTP Exchanges The Whole Packet (1)” shows a whole packet broken
down field by field for the FTP transfer performed in the clear. We see the TCP header, and the data
“hello world”. Because we are bypassing IPsec, this is exactly what you would see if IPsec were not
involved.
It is interesting to compare this packet with the one on the next slide that shows the same data “hello
world”, but inside an AH tunneled IP packet.
10 - 24
IPsec – SANS GIAC LevelTwo

The next slide “Authentication Header The Whole Packet (2)” shows the packet that transferred the
data “hello world”. This is an AH tunneled packet, easily identifiable by the IP protocol type field,
value 0x33 (or decimal 51). The tunneling is well presented; the source IP address is the address of the
security gateway instead of the FTP server’s address.
Now we know the packet is authenticated with AH and tunneled. We also know the AH structure and
the location of the AH header in the packet, so, let’s try to identify some fields in the hexadecimal data
to see if everything is as the theory says.
The data showed in hexadecimal contains the AH header as well as the original IP packet, meaning the
IP packet before IPsec was applied.
04 --> is the Next Header field; indicates IPv4
04 --> gives the length of the header, 6; the calculation goes 4 = x - 2 --> x = 6
00 00 --> Reserved
aa fb b9 40 --> the SPI
00 00 00 1e --> this SA sequence number; 1e = 30. Shows the SA has been up for some time since the
sequence number always starts at 1 for any new SA.
b1 0e ea ff 9f c3 57 b3 bd 8d a5 fb --> authentication data
45 00 00 33 a2 04 40 00 7f 06 02 45 c6 67 65 89 c6 67 65 23 00 14 04 13
00 70 43 a6 00 07 b5 33 50 18 20 70 c8 8f 00 00 --> original IP header
68 65 6c 6c 6f 20 57 6f 72 6c 64 obviously is the hexadecimal of hello world
10 - 25
IPsec – SANS GIAC LevelTwo
- ©2000, 2001
25
Encapsulating Security Payload
The encapsulating security payload is the second of the two security protocols
offered by the IPsec suite
•
Provide confidentiality
•
Provide data origin authentication