What is IPsec (Internet Protocol Security)?

What is IPsec used for?

IPsec is used for protecting sensitive data, such as financial transactions, medical records and corporate communications, as it's transmitted across the network. It's also used to secure virtual private networks (VPNs), where IPsec tunneling encrypts all data sent between two endpoints. IPsec can also encrypt application layer data and provide security for routers sending routing data across the public internet.

IPsec can also be used to provide authentication without encryption -- for example, to authenticate that data originated from a known sender. It is also common for IPsec to be used in conjunction with cloud architectures to maintain secure data pathways as information moves to and from cloud environments.

Encryption at the application or the transport layers of the Open Systems Interconnection (OSI) model can securely transmit data without using IPsec. At the application layer, HTTPS performs the encryption. While at the transport layer, the Transport Layer Security (TLS) protocol provides the encryption. However, encrypting and authenticating at these higher layers increase the chance of data exposure and attackers intercepting protocol information.

IPsec protocols

IPsec authenticates and encrypts data packets sent over both IPv4- and IPv6-based networks. IPsec protocol headers are found in the IP header of a packet and define how the data in a packet is handled, including its routing and delivery across a network. IPsec adds several components to the IP header, including security information and one or more cryptographic algorithms.

The IPsec protocols use a format called Request for Comments (RFC) to develop the requirements for the network security standards. RFC standards are used throughout the internet to provide important information that enables users and developers to create, manage and maintain the network.

The following are key IPsec protocols:

• IP AH. AH is specified in RFC 4302. It provides data integrity and transport protection services. AH was designed to be inserted into an IP packet to add authentication data and protect the contents from modification.
• IP ESP. Specified in RFC 4303, ESP provides authentication, integrity and confidentiality through encryption of IP packets.
• IKE. Defined in RFC 7296, IKE is a protocol that enables two systems or devices to establish a secure communication channel over an untrusted network. The protocol uses a series of key exchanges to create a secure tunnel between a client and a server through which they can send encrypted traffic. The security of the tunnel is based on the Diffie-Hellman key exchange.
• Internet Security Association and Key Management Protocol. ISAKMP is specified as part of the IKE protocol and RFC 7296. It is a framework for key establishment, authentication and negotiation of an SA for a secure exchange of packets at the IP layer. Each SA defines a connection in one direction, from one host to another. The SA includes all attributes of the connection, including the cryptographic algorithm, the IPsec mode, the encryption key and any other parameters related to data transmission over the connection.

IPsec also supports additional protocols, including those for digital signature and others outlined in the IPsec and IKE Document Roadmap (RFC 6071).

Emerging IPsec applications

IPsec has evolved to support newer applications beyond VPNs. Organizations now use IPsec to secure cloud communications, enabling protected data exchange between on-premises systems and cloud environments. Additionally, IPsec is integral to 5G network security frameworks, ensuring that both user and application data remain private and protected in next-generation networks

How does IPsec work?

There are five key steps involved with how IPsec works:

1. Host recognition

The IPsec process begins when a host system recognizes that a packet needs protection and should be transmitted using IPsec policies. Such packets are considered "interesting traffic" for IPsec purposes, and they trigger security policies. For outgoing packets, this means the appropriate encryption and authentication are applied. When an incoming packet is determined to be interesting, the host system verifies that it has been properly encrypted and authenticated.

2. Negotiation, or IKE Phase 1

In the second step, the hosts use IPsec to negotiate the set of policies they will use for a secured circuit. They also authenticate themselves to each other and set up a secure channel between them that is used to negotiate the way the IPsec circuit will encrypt or authenticate data sent across it. This negotiation process occurs using either main mode or aggressive mode.

With main mode, the host initiating the session sends proposals indicating its preferred encryption and authentication algorithms. The negotiation continues until both hosts agree and set up an IKE SA that defines the IPsec circuit they will use. This method is more secure than aggressive mode because it creates a secure tunnel for exchanging data.

In aggressive mode, the initiating host does not allow for negotiation and specifies the IKE SA to be used. The responding host's acceptance authenticates the session. With this method, the hosts can set up an IPsec circuit faster.

3. IPsec circuit, or IKE Phase 2

Step 3 sets up an IPsec circuit over the secure channel established in IKE Phase 1. The IPsec hosts negotiate the algorithms that will be used during the data transmission. The hosts also agree upon and exchange the encryption and decryption keys they plan to use for traffic to and from the protected network. The hosts also exchange cryptographic nonces, which are random numbers used to authenticate sessions.

4. IPsec transmission

In the fourth step, the hosts exchange the actual data across the secure tunnel they've established. The IPsec SAs set up earlier are used to encrypt and decrypt the packets.

5. IPsec termination

Finally, the IPsec tunnel is terminated. Usually, this happens after a previously specified number of bytes have passed through the IPsec tunnel or the session times out. When either of those events happens, the hosts communicate, and termination occurs. After termination, the hosts dispose of the private keys used during data transmission.

IPsec in cloud security

As organizations migrate to cloud services, IPsec plays a critical role in securing data as it moves between local infrastructure and cloud providers. Many cloud vendors support IPsec-based VPNs to create secure, encrypted tunnels that safeguard data from unauthorized access during transit. This layer of security aligns with compliance mandates, such as GDPR, that require organizations to protect data, particularly when it traverses public networks.

How is IPsec used in a VPN?

A VPN essentially is a private network implemented over a public network. Anyone who connects to the VPN can access this private network as if directly connected to it. VPNs are commonly used in businesses to enable employees to access their corporate network remotely.

IPsec is commonly used to secure VPNs. While a VPN creates a private network between a user's computer and the VPN server, IPsec protocols implement a secure network that protects VPN data from outside access. VPNs can be set up using one of the two IPsec modes: tunnel mode and transport mode.

IPsec vs. other security protocols

While IPsec secures data at the IP layer, other security protocols offer alternatives at different OSI layers. For instance, HTTPS operates at the application layer, encrypting data over websites, while TLS works at the transport layer, securing connections like emails and instant messages.

IPsec's ability to secure entire networks makes it ideal for VPNs, while TLS or HTTPS might be more suitable for specific connection types.

What are the IPsec modes?

In simple terms, transport mode secures data as it travels from one device to another, typically for a single session. Alternatively, tunnel mode secures the entire data path, from Point A to Point B regardless of the devices in between.

Tunnel mode. Usually used between secured network gateways, IPsec tunnel mode enables hosts behind one of the gateways to communicate securely with hosts behind the other gateway. For example, any users of systems in an enterprise branch office can securely connect with any systems in the main office if the branch office and main office have secure gateways to act as IPsec proxies for hosts within the respective offices. The IPsec tunnel is established between the two gateway hosts, but the tunnel itself carries traffic from any hosts inside the protected networks. Tunnel mode is useful for setting up a mechanism for protecting all traffic between two networks, from disparate hosts on either end.

Transport mode. A transport mode IPsec circuit is when two hosts set up a directly connected IPsec VPN connection. For example, this type of circuit might be set up to enable a remote IT support technician to log in to a remote server to do maintenance work. IPsec transport mode is used in cases where one host needs to interact with another host. The two hosts negotiate the IPsec circuit directly with each other, and the circuit is usually torn down after the session is complete.

Comparing IPsec VPN vs. SSL VPN

A Secure Socket Layer (SSL) VPN is another approach to securing a public network connection. The two can be used together or individually depending on the circumstances and security requirements.

With an IPsec VPN, IP packets are protected as they travel to and from the IPsec gateway at the edge of a private network and remote hosts and networks. An SSL VPN protects traffic as it moves between remote users and an SSL gateway. IPsec VPNs support all IP-based applications, while SSL VPNs only support browser-based applications, though they can support other applications with custom development.

Each type has unique strengths, with IPsec providing broader support for complex network topologies and SSL often offering easier client implementation.

Learn more about how IPsec VPNs and SSL VPNs differ in terms of authentication and access control, defending against attacks and client security. See what is best for your organization and where one type works best over the other. Explore the difference between site-to-site and remote access VPNs and the benefits and use cases for each.