What is TLS encryption and how does it work

Transport Layer Security (TLS) is one of the most important and widely used security protocols. It protects a significant proportion of the data that gets transmitted online. It’s most prominently used to secure the data that travels between a web browser and website via HTTPS, but it can also be used to secure email and a host of other protocols.

TLS is valuable because it ensures the other party in a connection is who they say they are, shows whether data retains its initial integrity, and provides confidentiality through encryption.

TLS uses a range of different algorithms and schemes to accomplish these purposes. It can seem complicated, but this article will cover one aspect at a time to give you an in-depth look at how TLS works to secure connections.

What does TLS do?

When sending information online, we run into three major security problems:

  • How can we know whether the person we are communicating with is really who they say they are?
  • How can we know that the data hasn’t been tampered with since they sent it?
  • How can we prevent other people from seeing and accessing the data?

These issues are crucial, especially when we are sending sensitive or valuable information. TLS uses a range of cryptographic techniques to address each of these three problems. Together, they allow the protocol to authenticate the other party in a connection, check the integrity of data and provide encrypted protection.

Let’s simplify things and pretend that you are trying to transfer information back and forth with a friend who lives across the country. If the information is sensitive, you will be very worried about the three major problems mentioned above.

You can’t just send a letter and hope for the best, especially if you suspect that your communications will be targeted by attackers. Instead, you need a system that allows you to verify that your recipient is legitimate, a way that you can check whether messages have been altered, and a way to protect them from prying eyes.

TLS fulfills these requirements using a number of different processes. It starts with what is known as a TLS handshake, which is where authentication takes place and the keys are established.

To return to our letter analogy, the authentication part of TLS would be kind of like sending a letter via a courier that requires identification. When the courier delivers the letter, they would compare the person’s ID to their face and verify whether or not the recipient was the correct person.

The key establishment phase would be a little bit like if your letter contained half of a pin number that you intended to use in future communications. You would ask your recipient to come up with the second half of the number and send it to you in their return letter.

Once the courier had verified the identity and the pin number had been established, you would have everything you need to securely send information. In reality, these stages are a lot more complex, but we’ll get to that later.

TLS securely exchanges information with the application protocol. To carry on our analogy, securely transmitting data over TLS would be like writing out a message and putting it in an envelope. You would write your signature across the seal, so that if the letter was tampered with, your recipient would be able to tell by the ripped signature (this is actually done with math, but again, we will cover it in depth later).

You would then put the letter into a small metal box that had a combination lock, setting the combination as the pin number that you jointly established with your recipient. You would send the box through the courier that checks identification when packages are delivered. Your recipient would reply in the same way, and any future communications would follow the same steps.

TLS solves all three of our problems in a relatively similar way. The courier serves to authenticate the recipient and make sure that the box is being delivered to its intended person. The locked box serves as a form of encryption, preventing anyone but your partner from accessing the letters. The signed envelope lets you know whether on not the message has been tampered with.

This is a very rough approximation of what TLS does. In reality, TLS takes place between clients and servers, rather than two people that are sending mail to each other. The analogy is just to give you a visualization of what is happening and the reasoning behind it. In the following sections, we will cover what actually happens in detail.

TLS vs. SSL

When reading about TLS, you will often see mention of SSL or even as TLS/SSL. Secure Sockets Layer (SSL) is the old version of TLS, but many in the industry still refer to TLS under the old moniker. This article will use the term TLS throughout, but it’s important to note that the names are often used interchangeably. You can read more about SSL in our guide.

The history of TLS

It all started with the need to secure the transport layer. As we mentioned above, the precursor to TLS was SSL. The first versions of SSL were developed in the nineties by Netscape, a company that built one of the early web browsers.

SSL 1.0 was never released because it contained serious vulnerabilities. Version 2.0 came out with Netscape Navigator 1.1 in 1995, however it still contained a number of serious flaws. SSL 3.0 was a heavily redesigned version and came out in 1996, with many of the security issues resolved.

In 1996, the IETF released a draft of SSL 3.0 in RFC 6101. The IETF formed a working group to standardize SSL, publishing the results in 1999 as TLS 1.0. It was documented in RFC 2246 and the standardization included some changes to the original protocol, as well as the name change. These modifications came about as a result of negotiations between Netscape, Microsoft and the IETF working group.

In 2006, the IETF released RFC 4346, which documents TLS 1.1. This version contained new security provisions and a number of other updates. Version 1.2 was released just two years later in 2008. It included support for authenticated encryption ciphers, a number of changes to how hash functions were used and many other improvements.

The next version didn’t arrive until 2018, when TLS 1.3 was defined. It features a host of changes, including enforced forward secrecy, removal of support for weaker algorithms and much more.

TLS 1.0 and 1.1 are now set to be deprecated in 2020, but version 1.2 is in widespread use. Version 1.3 is starting to see increased adoption.

TLS: The technical details

TLS consists of many different elements. The fundamental part is the record protocol, the underlying protocol responsible for the overarching structure of everything else.

tls-3

Diagram showing the TLS stack. TLS protocol stack by Jeffreytedjosukmono. Licensed under CC0.

The record protocol contains five separate subprotocols, each of which are formatted as records:

  • Handshake – This protocol is used to set up the parameters for a secure connection.
  • Application – The application protocol begins after the handshake process, and it is where data is securely transmitted between the two parties.
  • Alert – The alert protocol is used by either party in a connection to notify the other if there are any errors, stability issues or a potential compromise.
  • Change Cipher Spec – This protocol is used by either the client or the server to modify the encryption parameters. It’s pretty straightforward, so we won’t cover it in depth in this article.
  • Heartbeat – This is a TLS extension that lets one side of the connection know whether its peer is still alive, and prevents firewalls from closing inactive connections. It’s not a core part of TLS, so we’ll be skipping it in this guide.

Each of these subprotocols are used in different stages to communicate different information. The most important ones to understand are the handshake and the application protocols, because these are responsible for establishing the connection and then securely transmitting the data.

The TLS handshake protocol

This is where the connection is established in a secure manner. It may seem complex if you are new to some of the concepts, but each of these are covered later on in the article if you need to refer to them.

There are three basic types of TLS handshake: the basic TLS handshake, the client-authenticated TLS handshake and the abbreviated handshake.

The basic TLS handshake

tls-2

Diagram showing the TLS handshake process. Full TLS 1.2 Handshake by FleshGrinder. Licensed under CC0.

In this type of handshake, only the server is authenticated and not the client. It begins with the negotiation phase, where a client sends a Client Hello message. This contains the highest version of TLS that the client supports, possible cipher suites, an indication of whether it supports compression, a random number and some other information

The Client Hello message is met with a Server Hello message. This response contains the session ID, protocol version, cipher suite and compression (if any is being used) that the server selected from the client’s list. It also includes a different random number.

It depends on the cipher suite that has been selected, but the server will generally follow this by sending a Certificate message for authentication. This validates its identity and contains its public key.

If ephemeral Diffie-Hellman or anonymous Diffie-Hellman key exchanges are being used, then this is followed by a Server Key Exchange message. Other key exchange methods skip this part. When the server has finished with its side of the negotiation, it sends a Server Hello Done message.

Now it’s the client’s turn again. Depending on the chosen cipher suite, it will send a Client Key Exchange message. This can contain a public key or a premaster secret, which is encrypted with the server’s public key.

Both parties then use the random numbers and the premaster secret to come up with a master secret. Keys are derived from the master secret, which are then used to authenticate and encrypt the communications.

The client then sends a Change Cipher Spec message. This tells the server that the following messages will now be authenticated and encrypted (although sometimes encryption may not be used).

The client then follows this up with a Finished message, which is encrypted and also contains a Message Authentication Code (MAC) for authentication. The server decrypts this message and verifies the MAC. If any of these processes fail, then the connection should be rejected.

Now it’s the server’s turn to send a Change Cipher Spec message, as well as a Finished message with the same content as above. The client then also tries to decrypt and verify the contents. If this is all completed successfully, the handshake is finished. At this point, the application protocol is established. Data can then be exchanged securely in the same way as the Finished message from above, with authentication and optional encryption.

Client-authenticated TLS handshake

This handshake is much like the basic TLS handshake, but the client is authenticated as well. The main difference is that after the server sends its Certificate message, it also sends a Certificate Request message, asking for the client’s certificate. Once the server is finished, the client sends its certificate in a Certificate message.

The client then sends its Client Key Exchange message, just like in the basic TLS handshake. This is followed by the Certificate Verify message, which includes the client’s digital signature. Since it is calculated from the client’s private key, the server can verify the signature using the public key that was sent as part of the client’s digital certificate. The rest of the Client-authenticated TLS handshake follows along the same lines as the basic TLS handshake.

Abbreviated TLS handshake

Once a handshake has already taken place, TLS allows much of the process to be cut out by using an abbreviated handshake instead. These handshakes use the session ID to link the new connection to the previous parameters.

An abbreviated handshake allows both parties to resume the secure connection with the same setup that was negotiated earlier. Because some of the cryptography that is normally involved in initiating a handshake can be pretty heavy on computational resources, this saves time and makes the connection easier.

The process begins with the Client Hello message. It’s much like the earlier Client Hello message, but it also contains the session ID from the earlier connection. If the server knows the session ID, it includes it in its Server Hello message. If it does not recognize the session ID, it will return a different number, and a full TLS handshake will have to take place instead.

If the server does recognize the session ID, then the Certificate and Key Exchange steps can be skipped. The Change Cipher Spec and Finished messages get sent in the same way as the basic TLS handshake shown above. Once the client has decrypted the message and verified the MAC, data can be sent across the secure TLS connection.

There is also a TLS extension that allows connections to be resumed with session tickets instead of session IDs. The server encrypts data about the session and sends it to the client. When the client wants to resume this connection, it sends the session ticket to the server, which decrypts it to reveal the parameters.

Session tickets aren’t used as frequently, because they require the extension. Despite this, they can be advantageous in certain situations, because the server doesn’t have to store anything.

Unpacking the TLS handshake

The three most important steps of the handshake include:

  • the parameters are selected,
  • it conducts authentication, and
  • the keys are established.

Let’s cover them in a bit more detail so that you can understand what’s really going on.

The parameters

At the start of the handshake, the client and the server negotiate the parameters of the connection by mutual agreement. The first of these is which version of TLS will be used. This is the highest version that both parties support, which tends to be the most secure.

The parties also decide which key exchange algorithm they will use to establish the master key. The hash function, encryption algorithm, and compression method are also agreed upon in this stage. These will be covered in detail when we discuss the Application protocol later in the article.

Authentication: Digital certificates

Authentication is a key part of securing a communication channel, because it lets both parties know that they are actually talking to who they think they are and not an impostor. In TLS and many other security mechanisms, this is achieved with what are known as digital certificates.

Digital certificates are electronic documents that show the link between an individual or entity and their public key. This link is validated by a certificate authority (CA), which is a trusted organization that verifies that the two are actually related, then uses its own reputation to grant trust to the certificate.   

Different certificate levels represent varying degrees of trust. The important thing to know is that if a client or a server has a reliable and valid certificate, then it is reasonable to assume that the public key is legitimate and that you are not dealing with an attacker.

A note on public keys

Public-key encryption (also known as asymmetric encryption) is an important part of cryptography, and it is used extensively in the different aspects of TLS. Here’s a quick primer for those who are unfamiliar with how it works.

The short explanation is that public-key cryptography uses a key pair for encryption and decryption, rather than just a single key. The sender uses their intended recipient’s public key to encrypt data. Once it has been encrypted, it can only be decrypted by the recipient’s matching private key. Of course, the public key can be shared publicly while the private key must be kept secret.

Public-key encryption allows parties to share information securely, even if they have never met or had an opportunity to exchange keys beforehand. It does this through some unique properties of prime numbers. You can find out more in our article on public-key encryption.

Establishing a master secret

As we saw above when we discussed the process of the basic TLS handshake, after a party (or both parties) proves its identity with its public certificate, the next step is to establish the master secret, also known as the shared secret. The master secret is the base for deriving the keys used to both encrypt and check the integrity of data transmitted between the two parties.

The TLS handshake can use a number of different mechanisms to share this secret securely. These include RSA, several different types of the Diffie-Hellman key exchange, PSK, Kerberos, and others. Each has its own advantages and disadvantages, such as providing forward secrecy, but these differences are out of the scope of this article.

The exact process will depend on which key exchange method has been chosen, but it follows the rough steps mentioned in The basic TLS handshake section.

The premaster secret is derived according to whichever key exchange method had previously been selected. The client encrypts the premaster secret with the server’s public key to safely send it across the connection.

The client and server then both use the premaster secret and the random numbers that they sent at the start of the communication to come up with the master secret. Once the master key has been calculated, it is used to come up with either four or six separate keys. These are the:

  • Client write MAC key – This key is used by the server to check the integrity of data that was sent by the client.
  • Server write MAC key – The server write MAC key is used by the client to check the integrity of data that was sent by the server.
  • Client write encryption key – The server uses this key to encrypt data that was sent by the client.
  • Server write encryption key – The client uses this key to encrypt data that was sent by the server.
  • Client write IV key – The client write IV key is used by the server in AEAD ciphers, but not when other key exchange algorithms are used.
  • Server write IV key – Similarly, this key is used by the client in AEAD ciphers, but not when other key exchange algorithms are used.

Establishing the master key is an important part of the TLS handshake, because it enables both sides of the connection to securely derive keys that can be used for both authentication and encryption. Separate keys are used for both processes as a precaution.

Once the authentication and encryption keys have been derived, they are used to protect both Finished messages, as well as records sent through the application protocol.

The application protocol

Once a secure connection has been established by the TLS handshake, the application protocol is used to protect the transmitted data. It can be configured to use a wide range of algorithms to suit different scenarios.

Authentication algorithms

The integrity of messages can be checked with many different algorithms. These include:

  • HMAC-MD5
  • HMAC-SHA1
  • HMAC-SHA2
  • AEAD

To prove the integrity of the data being sent, the sender runs the information through a hash function to return a unique string of characters. These are special formulas that will always return the same result whenever they receive the same input.

The sender signs this data with their private key to form what is known as a digital signature, The digital signature is then attached to the message and sent to the recipient. To check whether the data retains its integrity and hasn’t been tampered with, the recipient decrypts the hash with the sender’s public key. They then use the same hash function on the data that was sent. The recipient then compares the two values.

If they are the same, it means that the data has not been altered since it was signed by the sender. If they are different, it is likely that the data has been tampered with, or there has been some other error.

That’s the short version of how hash functions can be used to show the integrity of data. If you would like a more in-depth understanding, check out our article on encryption, salting and hashing.

Encryption algorithms

TLS uses symmetric-key encryption to provide confidentiality to the data that it transmits. Unlike public-key encryption, just one key is used in both the encryption and decryption processes. Once data has been encrypted with an algorithm, it will appear as a jumble of ciphertext. As long as an appropriate algorithm is used, attackers will not be able to access the actual data, even if they intercept it.

TLS can use many different algorithms, such as Camellia or ARIA, although the most popular is AES.

Compression

Compression is the process of encoding data to make it take up less room. TLS supports compression if both sides of the connection decide to use it. Despite this ability, it is generally recommended to avoid using TLS to compress data, especially since the CRIME attack (see the TLS Security Issues section below) was found to be able to take advantage of compressed data for session hijacking.

Padding

Padding adds extra data to a message before it is encrypted. It’s a common cryptographic process that is used to help prevent hints in the structure of encrypted data from giving away its true meaning. TLS generally applies PKCS#7 padding to records before they are encrypted.

Alert protocol

If the connection or security becomes unstable, compromised, or a serious error has occurred, the alert protocol allows the sender to notify the other party. These messages have two types, either warning or fatal. A warning message indicates that the session is unstable and allows the recipient to determine whether or not the session should be continued.

A fatal message tells the recipient that the connection has been compromised or a serious error has occurred. The sender should close the connection after they send the message. The alert protocol also contains information about what is causing the particular connection problem. This can include things like decryption failure, an unknown certificate authority, an illegal parameter and much more.

TLS & the OSI model

The OSI model is a way to conceptualize and standardize how we look at our different communication systems and protocols. It’s important to note that it’s just a model, and some of our protocols don’t conform to it.

The OSI has seven separate layers that show the levels that protocols operate at, however, TLS doesn’t fit into any single one. It flows over another transport protocol like TCP, which should imply that it is above the fourth layer, the transport layer. It’s most prominently used like a transport layer, but because it conducts handshakes, this would imply that it is part of the presentation or application layers.

As you can see, TLS simply doesn’t conform to the OSI model. This doesn’t mean that TLS is broken or wrong, if anything, it just shows that the OSI model is flawed and not able to account for all of our communication protocols.

TLS usage

TLS is used to secure a significant proportion of our online communications. Normally it is implemented over protocols like the Transmission Control Protocol (TCP), but it can also be used in the Datagram Congestion Control Protocol (DCCP) and User Datagram Protocol (UDP).

It can secure protocols like HTTP, SMTP, FTP, XMPP and NNTP, as well as others. The most common application is as Hypertext Transfer Protocol Secure (HTTPS), which protects the connection between a web browser and a website. You can tell when HTTPS is being used to secure your online connection, because a little green lock icon will appear to the left of the URL at the top of your browser.

TLS can also be used to build VPNs, such as in OpenConnect and OpenVPN. It uses its encryption and authentication capabilities to form a tunnel that can connect hosts and networks to each other. TLS-based VPN technologies like OpenVPN are advantageous over alternatives like IPsec, because OpenVPN is not known to have any serious security issues. These VPNs can also be easier to configure.

Another of its uses is to secure email through STARTTLS. When TLS is implemented, it prevents attackers from being able to access messages as they travel between mail servers.

TLS security issues

Like most protocols, TLS has had a number of past vulnerabilities and theoretical attacks against its various implementations. Despite this, the latest versions are considered secure for practical purposes.

Past versions, such as SSL 2.0 and 3.0 (and TLS 1.0, which is essentially the same as SSL 3.0) feature numerous security flaws, but since they are old and deprecated protocols, we won’t go into the details. You should be using TLS 1.2 and 1.3 to secure your connections instead.

The newer versions of TLS have numerous upgrades that make it less-vulnerable than SSL. Despite this, the protocol has still had the following security issues:

Renegotiation attacks

One of TLS’s features is that it allows client and server pairs to renegotiate the parameters of their existing connection. In 2009, it was discovered that this could be exploited by attackers so that they can inject traffic to make it look like it comes from the client. Servers will accept the request as legitimate, which means that attackers could potentially be manipulating the outgoing messages.

This attack does not allow the attacker to see the response, but it still has the potential to be damaging. An extension that will prevent these attacks is currently a proposed standard.

BEAST

The Browser Exploit Against SSL/TLS (BEAST) attack was first discovered by researchers in 2011. It takes advantage of the cipher block chaining vulnerability in TLS, which can be used to decrypt messages. This attack only affects TLS 1.0, which is an old and weaker version of the protocol. Although it will not be deprecated until 2020, users should be using versions 1.2 and 1.3 instead.

Timing attacks

These side channel attacks analyze how long it takes for an algorithm to run, then use that information to work backwards and figure out the key. In 2013, the Lucky Thirteen attack was discovered to leverage both a timing attack and a padding oracle attack while the message authentication code (MAC) is being checked. This attack can be used to break the TLS algorithm, although it is not considered dangerous for the majority of TLS users.

CRIME & BREACH

The CRIME attack works against a range of protocols. When data has been compressed, it can elicit content from authentication cookies. This information can be used for session hijacking. While it affects a number of protocols, the attack is particularly worrying when HTTP compression is being used, because there are no effective mitigation strategies.

In 2013, the Browser Reconnaissance and Exfiltration via Adaptive Compression of Hypertext (BREACH) exploit was found to affect HTTP compression in a similar way. This version of the attack can recover email addresses and other valuable data that was encrypted with TLS. The BREACH attack can be mitigated by disabling HTTP compression, or using techniques like cross-site request forgery (CSRF) protection.

Downgrade attacks

These are attacks that trick servers into using earlier and less-secure versions of TLS. Attackers could use these techniques to negotiate the use of less-secure key exchanges and ciphers. The Logjam attack is a good example because it could make vulnerable servers use 512-bit Diffie-Hellman, which is weak. Attackers can then break this key exchange mechanism and extract the keys, allowing them complete access to the session.

Heartbleed

Heartbleed was a security flaw that was accidentally introduced into the OpenSSL cryptography library in 2012, but not publicized until 2014. Because this is such a commonly used implementation of TLS, it caused significant global damage.

One of the developers for the TLS heartbeat extension added a buffer over-read vulnerability, which allows some extra data to be exposed. The error was not picked up when the code was reviewed, which led to a number of significant attacks.

Since the OpenSSL library is implemented so widely, the international cost of mitigating the issue ended up being quite expensive. Server administrators had to install the new patch and regenerate certificates and key pairs that may have been compromised during the two year period that the vulnerability existed.

DROWN

The Decrypting RSA with Obsolete and Weakened eNcryption (DROWN) attack was announced in 2016 and it takes advantage of server support for SSL 2.0. It uses a chosen-ciphertext attack against servers that still support SSL 2.0, as well as those that share the same public key certificate with another server that supports SSL 2.0.

When the attack was announced, it could be launched with initial setup costs of around $18,000 and about $400 for each separate attack. When the DROWN attack is successful, it acquires the session keys.

The attack can be mitigated by not sharing server certificates. The OpenSSL group has also released patches that remove support for older protocols and ciphers, but these only work if the server’s certificate is not shared with other servers that support SSL 2.0.

Unholy PAC

This attack was found in 2016. It leverages weaknesses found in the Web Proxy Autodiscovery Protocol (WPAD). When a user tries to visit a website over a TLS encrypted connection, the flaw can make the URL visible. Since URLs are sometimes sent to users as a form of authentication, the Unholy PAC attack makes it possible for an attacker to take over a user’s account.

Is TLS safe?

While it may seem like there are a lot of security issues, the reality is that most of these are pretty minor and can or have been mitigated. As technology progresses, vulnerabilities are discovered and new attacks are developed, TLS will continue to have more security problems. Despite this, for now and the foreseeable future, it looks like TLS will still be one of the main and most reliable tools that we use to secure our online world.

Cyber security business technology by TheDigitalArtist licensed under CC0