Decryption Is Key for Enhanced Security and Monitoring

I’m often asked the following questions when discussing NETSCOUT’s nGenius Decryption Appliance (nDA): “How does decryption work?” “Why are we able to intercept encrypted traffic and decrypt it?” “Doesn’t it defeat the purpose of encryption if I can simply listen to traffic on the network and decrypt it?”

Welcome to part one of a three-part series that explains how Transport Layer Security (TLS) encryption works, how nDA is able to decrypt network traffic, and why TLS is still secure even with a decryption appliance inline on your network.

Encryption Basics

There are two major categories of encryption: symmetric and asymmetric. Symmetric encryption uses a single key that both parties need to have for communication to work. It has relatively small key sizes, is fast, and has low resource utilization. Asymmetric encryption needs two separate keys—one called the public key that is available to anyone, and the other called the private key that remains with the person or system that created the public/private key pair. With asymmetric encryption, key sizes are much larger. It also is much slower and needs a lot more resources.

Because it is fast, symmetric encryption is used for large data encryption, with algorithms such as Data Encryption Standard (DES) and Advanced Encryption Standard (AES). Asymmetric encryption, on the other hand, is used for secure message transfer with applications such as RSA, because with asymmetric encryption the same transaction can be encrypted with multiple keys (for example, a public key of one party and the private key of another).

The main problem with symmetric encryption is the need for both parties to have the same key. How do you exchange a key that needs to remain private across an untrusted medium such as the internet? You either need to have exchanged the key privately ahead of time or use some other mechanism to exchange the key. The main issue with asymmetric encryption is speed and resource utilization. Asymmetric encryption is secure because of the private/public key nature but can be unusable for encryption of large amounts of data. So, one of the main use cases is to use both together, creating an asymmetric session to exchange a symmetric key and then using the symmetric key to transfer the large amount of data that needs to be exchanged securely. This, in essence, is how TLS works.

TLS Encryption

The following is very high-level view of how TLS works, focusing specifically on TLS 1.2 versus TLS 1.3. Both versions of TLS rely on a certificate authority (CA). The CA is an entity that verifies the owner of a server and authenticates the identity of the server. This process starts with the owner of the server sending its information to a CA, along with the public key and identity of the server. Once the CA has authenticated the server, it signs this information with its private key, and returns this information (called a certificate) back to the server. The server then uses this certificate in all TLS communication.

The first thing that needs to happen is that the server needs a server certificate, signed by a trusted CA (your browser trusts many CAs out of the box). To do this, the server generates a public key/private key pair, generates a certificate signing request (CSR), signs this with the private key, and sends it to a CA. The CA builds the server certificate using these details, and returns a server certificate signed with its private key.

When a client sends a request to a server, the server responds with its signed certificate and public key. The client authenticates this certificate by decrypting it with the CA’s public key. This allows the client to know that the server is the system it says it is. The client then uses this public key from the server to send a message to establish a session key with this server—a key that only the server can decrypt with its private key—and communication continues with that session key.

TLS 1.2 Session

Let’s look at the initial state of a TLS 1.2 connection. The client has a CA certificate and the CA public key. The client normally will have many of these, but what is important is that it has the CA certificate and public key of the CA that signed the server certificate. The CA also has its certificate, and its public and private keys. The server has the server certificate that was signed by the CA, and its public and private keys. 

To complete a TLS 1.2 handshake, the end goal is to exchange a symmetric key that both the client and server know. This starts with the client sending a Hello message that includes a list of the cipher suites it supports. The server responds with a Hello message, picks one of the cipher suites, and includes its server certificate and its public key. The client authenticates this server certificate by decrypting it with the CA public key. Assuming this works, the client then generates a premaster secret, encrypts it using the server public key, and sends it to the server. It also generates the session key from the premaster secret that it just sent to the server. The server decrypts this message with its private key (it’s the only machine that has this key, and therefore the only machine that can decrypt this message), which recovers the premaster secret. The server then uses this premaster secret to generate the session key and sends an encrypted handshake message to the client, which the client should be able to decrypt. At this point, both the client and server have the same session key (a symmetric key) and use this for the rest of the communication during this session.

TLS 1.3 Session

TLS 1.3 improved upon TLS 1.2 in many ways. There were known security issues with TLS 1.2, including attacks that could be done on certain implementations of the standard. TLS 1.2 has 37 different cipher suites available. Many of these are no longer considered secure for one reason or another. TLS 1.3, on the other hand, supports only five cipher suites.

Another improvement was the speed of establishing a connection. TLS 1.2 required a total of two round-trip connections before data was transmitted. This required more time to establish the connection and is obviously slower than the one or sometimes zero round-trip time necessary to establish a TLS 1.3 session.

Zero round-trip time for TLS 1.3? Yes, that’s correct. If a session was previously established, a TLS 1.3 session can be resumed, which makes establishing the session even faster. 

The final TLS 1.3 improvement is the use of ephemeral keys only. This means that if a session key is compromised for some reason, it will only help only for a small portion of traffic between client and server and will not compromise the entire session.

To Be Continued

Stay tuned for the second and third parts of this series, where I discuss how nDA can decrypt TLS sessions, and why TLS is still secure despite this ability.

Learn more about NETSCOUT's nGenius Decryption Appliance.
Learn more about encrypted traffic visibility.