IPsec (Internet Protocol Security)

DEFINITION

IPsec, also known as the Internet Protocol Security or IP Security protocol, defines the architecture for security services for IP network traffic. IPsec describes the framework for providing security at the IP layer, as well as the suite of protocols designed to provide that security, through authentication and encryption of IP network packets. Also included in IPsec are protocols that define the cryptographic algorithms used to encrypt, decrypt and authenticate packets, as well as the protocols needed for secure key exchange and key management.

IPsec originally defined two mechanisms for imposing security on IP packets: the Encapsulating Security Payload (ESP) protocol, which defined a method for encrypting data in IP packets, and the Authentication Header (AH) protocol, which defined a method for digitally signing IP packets. The Internet Key Exchange(IKE) protocol is used to manage the cryptographic keys used by hosts for IPsec. IPsec can be used to protect network data, for example, by setting up circuits using IPsec tunneling, in which all data being sent between two endpoints is encrypted, as with a Virtual Private Network (VPN) connection; for encrypting application layer data; and for providing security for routers sending routing data across the public internet. IPsec can also be used to provide authentication without encryption, for example to authenticate that data originates from a known sender.

Internet traffic can be secured from host to host without the use of IPsec, for example by encryption at the application layer (Layer 7 of the OSI model) with HTTP Secure (HTTPS) or at the transport layer (Layer 4 of the OSI model) with the Transport Layer Security (TLS) protocol. However, when traffic uses encryption or authentication at these higher layers, threat actors may still be able to intercept protocol information that may expose data that should be encrypted.

IPsec protocols

IPsec is defined for use with both current versions of the Internet Protocol, IPv4 and IPv6. IPsec protocol headers are included in the IP header, where they appear as IP header extensions when a system is using IPsec.

The most important protocols considered a part of IPsec include:

· The IP Authentication Header (AH), specified in RFC 4302, defines an optional packet header to be used to guarantee connectionless integrity and data origin authentication for IP packets, and to protect against replays.

· The IP Encapsulating Security Payload (ESP), specified in RFC 4303, defines an optional packet header that can be used to provide confidentiality through encryption of the packet, as well as integrity protection, data origin authentication, access control and optional protection against replays or traffic analysis.

· Internet Key Exchange (IKE), defined in RFC 7296, "Internet Key Exchange Protocol Version 2 (IKEv2)," is a protocol defined to allow hosts to specify which services are to be incorporated in packets, which cryptographic algorithms will be used to provide those services, and a mechanism for sharing the keys used with those cryptographic algorithms.

· Previously defined on its own, the Internet Security Association and Key Management Protocol (ISAKMP) is now specified as part of the IKE protocol specification. ISAKMP defines how Security Associations (SAs) are set up and used to define direct connections between two hosts that are using IPsec. Each SA defines a connection, in one direction, from one host to another; a pair of hosts would be defined by two SAs. The SA includes all relevant attributes of the connection, including the cryptographic algorithm being used, the IPsec mode being used, encryption key and any other parameters related to the transmission of data over the connection.

Numerous other protocols and algorithms use or are used by IPsec, including encryption and digital signature algorithms, and most related protocols are described in RFC 6071, "IP Security (IPsec) and Internet Key Exchange (IKE) Document Roadmap."

How IPsec works

The first step in the process of using IPsec occurs when a host recognizes that a packet should be transmitted using IPsec. This may be done by checking the IP address of the source or destination against policy configurations to determine whether the traffic should be considered "interesting" for IPsec purposes. Interesting traffic triggers the security policy for the packets, which means that the system sending the packet applies the appropriate encryption and/or authentication to the packet. When an incoming packet is determined to be "interesting," the host verifies that the inbound packet has been encrypted and/or authenticated properly.

The second step in the IPsec process, called IKE Phase 1, allows the two hosts using IPsec to negotiate the policy sets they use for the secured circuit, authenticate themselves to each other, and initiate a secure channel between the two hosts. IKE Phase 1 sets up an initial secure channel between hosts using IPsec; that secure channel is then used to securely negotiate the way the IPsec circuit will encrypt and/or authenticate data sent across the IPsec circuit.

There are two options during IKE Phase 1: Main mode or Aggressive mode. Main mode provides greater security because it sets up a secure tunnel for exchanging session algorithms and keys, while Aggressive mode allows some of the session configuration data to be passed as plaintext but enables hosts to establish an IPsec circuit more quickly.

Under Main mode, the host initiating the session sends one or more proposals for the session, indicating the encryption and authentication algorithms it prefers to use, as well as other aspects of the connection; the responding host continues the negotiation until the two hosts agree and establish an IKE Security Association (SA), which defines the IPsec circuit. When Aggressive mode is in use, the host initiating the circuit specifies the IKE security association data unilaterally and in the clear, with the responding host responding by authenticating the session.

The third step in setting up an IPsec circuit is the IKE Phase 2, which itself is conducted over the secure channel setup in IKE Phase 1. It requires the two hosts to negotiate and initiate the security association for the IPsec circuit carrying actual network data. In the second phase, the two hosts negotiate the type of cryptographic algorithms to use on the session, as well as agreeing on secret keying material to be used with those algorithms. Nonces, randomly selected numbers used only once to provide session authentication and replay protection, are exchanged in this phase. The hosts may also negotiate to enforce perfect forward secrecy on the exchange in this phase.

Step four of the IPsec connection is the actual exchange of data across the newly created IPsec encrypted tunnel. From this point, packets are encrypted and decrypted by the two endpoints using the IPsec SAs setup in the previous three steps. The last step is termination of the IPsec tunnel, usually when the communication between the hosts is complete, when the session times out or when a previously specified number of bytes has been passed through the IPsec tunnel. When the IPsec tunnel is terminated, the hosts discard keys used over that security association.

IPsec VPN and IPsec modes

IPsec protocols can be used to assemble a VPN connection, to encrypt and/or authenticate all traffic between two or more points. IPsec circuits, including VPNs, can be set up to use two modes:

Tunnel mode: Usually used between secured network gateways, IPsec tunnel mode enables hosts behind one of the gateways to communicate securely with hosts behind the other gateway. For example, any users of systems in an enterprise branch office can securely connect with any systems in the main office if the branch office and main office have secure gateways to act as IPsec proxies for hosts within the respective offices. The IPsec tunnel is established between the two gateway hosts, but the tunnel itself can carry traffic from any hosts inside the protected networks. Tunnel mode is useful for setting up a mechanism for protecting all traffic between two networks, from disparate hosts on either end.

IPsec enables an encrypted tunnel across the public internet for securing LAN packets sent between remote locations.

Transport mode: When two individual hosts set up a directly connected IPsec VPN connection, the circuit can be said to be an example of a transport mode IPsec circuit. For example, a transport mode IPsec circuit might be set up to allow a remote IT support technician to log in to a remote server to do maintenance work. Transport mode IPsec is used in cases where one host needs to interact with another host; the two hosts negotiate the IPsec circuit directly with each other, and the circuit is usually torn down after the session is complete.

Implementing IPsec

IPsec support has generally been included in most mainstream (and other) operating systems available since the late 1990s, including desktop and server operating systems, as well as router and other network security appliances.  While older systems may support some version of IPsec, enterprises should deploy IPsec using operating systems that are current and up to date on security patches. Also, older systems that support older versions of IPsec and appear to enable secure IPsec circuits may not in fact be keeping data secure.