TLS Basics

In this post we’ll talk about what TLS is and how does it work in SMTP transactions, the same can be applied to other protocols like HTTPS, though.

SMTP Client   SMTP Server Protocol
Waiting connection… 220 Microsoft ESMTP MAIL Service ready SMTP












STARTTLS 220 2.0.0 SMTP server ready SMTP
ClientHello ServerHelloCertificate*ServerKeyExchange*






[ChangeCipherSpec]Finished TLS
Application Data Application Data SMTP over TLS


A normal TLS communication includes the following steps:

  1. An SMTP Client connects to an SMTP Server. This step is done as with any other SMTP transaction. The SMTP Client will wait for the 220 smtp response code from the SMTP Server to initiate the message transmission.
  2. Both hosts initiate the SMTP conversation. At this point the SMTP Client sends the EHLO command with its FQDN as value. This step helps the SMTP Server to learn the SMTP Client name. The SMTP Server responds to this command with a greeting that contains a list of all commands / mechanisms it supports, in case it can handle TLS communication it will offer the STARTTLS command in the list. When the SMTP Client finds this command, it has the option to send the message in clear text (by sending MAIL FROM as the next command) or initiate a secure transmission with TLS (by sending the STARTTLS command without arguments).
  3. The SMTP Client initiates the TLS session. If the SMTP Client supports TLS communications it sends the STARTTLS command. Now the SMTP Server responds with a “220 2.0.0. SMTP Server ready” to inform that it is willing to initiate the TLS session. This is the last point where the communication is SMTP, from now on any information transmitted won’t be in clear text. From this moment both servers change their role to TLS Client and TLS Server respectively.
  4. The TLS Client initiates the TLS Handshake Protocol. The TLS Client sends a list of all the cipher suites it is able to handle.
  5. The TLS Server responds to the Client Hello message. At this point the TLS Server chooses one of the cipher suites offered in the clientHello package and begins the encryption process by sending several package with cryptographic information among which a public certificate may be present.
  6. The TLS Client verifies the public certificate. According to the cipher suite selected, the SMTP Client retrieves the cryptographic information from the server and completes the request. If a public certificate was present, it must validate it to ensure the authenticity of the TLS server identity. If this process is successful the authentication feature of TLS can be achieved, if it’s not, then the client can still continue the process by assuming the risk that there might be a problem with the server’s identity. If the process continues, the client sends back to the server the remaining information to complete the cryptographic algorithms.
  7. TLS Client and server change to encrypted communication. Only at this point is when the conversation is fully encrypted with TLS. Now the SMTP Conversation starts all over again from the EHLO command and the message is transmitted in a secured channel.
  8. The session ends. At this point, the TLS Client has sent all its messages to the server. The TLS tunnel is destroyed but some parameters are held in a TLS cache in both ends just in case they communicate again and want to recycle the same session, this is done to save the time and resources to create a new encrypted channel but it is not encouraged by the TLS standard because secrecy of the keys may be compromised.

Digital Certificates

A digital certificate is a block of digital information that contains private data used in the process of uniquely identifying an specific entity. In TLS communication there are at least three different certificates that are used to accomplish these features:

  • Authenticity. The digital certificate is signed by a certificate authority in which both servers trust.
  • Integrity. The encryption process in TLS uses a symmetric algorithm. This means that all packets of the SMTP conversation are encrypted and decrypted with the same key which is called the Master-Secret. Before encryption of a packet takes place, a hash is taken and then encrypted using the Master-Secret, the result is added to the encrypted packed as a digital signature. Each time a packet is received, the digital signature is verified by decrypting it with the Master-Secret and then taking a hash to the decrypted packet, if both hashes are the same, the integrity of the package is confirmed, otherwise, the packet is dropped and the TLS session ends immediately.

Here is a description of the steps taken when using digital certificates:

  1. Server1 starts the creation of its digital certificate by first generating its own private-public key pair. These keys are just a numerical sequence that obeys a mathematical algorithm in such a way that any message that is encrypted with the private key can only be decrypted with the corresponding public key. The private key must never be exposed or shared, it must always be kept secure.
  2. Server1 uses its private key to generate a “certificate request”. This is a file containing the public key associated with the private key in a format any certificate authority can understand. The request is sent to the Certificate Authority for signing. The request contains information like company name, phone number, country, state, etc.
  3. Certificate Authority (CA) receives the certificate request. As a result, a public certificate is created for Server1 with the data contained in the request. This certificate contains a digital signature from the CA. This signature is created by hashing the public certificate and then encrypting it with the private key of the CA, the result is appended to the public certificate, this will allow any other entity to validate that this certificated came from this particular CA.
  4. Server1 stores / imports the public certificate to its certificate repository. This certificate will be transmitted to the TLS Clients anytime a TLS connection is requested. It also must obtain the CA public certificate (CA Root certificate), that contains the public key of the CA. This certificate is the one that must be deployed to all MTA servers that will be communicating with Server1 using TLS.
  5. Server2 starts a mail transaction using TLS with Server1. Now Server1 responds to the TLS request by sending its new public certificate among the other cryptographic information.
  6. Server2 extracts the name of the CA from the public certificate. It hen looks for the corresponding public certificate (this is known as the CA root certificate) for this CA in its own certificate repository. If there’s no match, Server2 cannot authenticate the identity of Server1 and must decide to continue or abort the TLS process. If there’s a match, it means the CA Root certificate was previously deployed to this server successfully and now the authentication phase can continue. Server2 extracts the public key from the CA Root certificate, takes a hash from Server1’s public certificate and then encrypts the result with the CA’s public key, if this result matches the digital signature in Server1’s public certificate, the authenticity of the data included is verified and now Server1 is successfully authenticated. The remaining step in the verification is that the value of the CommonName parameter in the certificate must match the FQDN sent by Server1 in the response to the EHLO command.
  7. Now that the TLS Server is authenticated the TLS process continues.

The following table shows the TLS roles and corresponding certificates.

Role Certificates Use
TLS Client
  • CA Root Certificate
It is used to extract the CA’s public key to confirm the digital signature of the TLS Server’s public certificate.
TLS Server
  • Public Certificate
This is sent in every TLS session for the TLS Client to verify the TLS Server identity.
  • Private key             (TLS Server’s Private Key)
This key is used for creating the certificate request. After the public certificate is signed, this key is again used to decrypt the packets the TLS Client encrypted with the public key contained in the server’s public certificate.
  • CA Root certificate
Any CA must always maintain this certificate public for any entity to obtain it in order to validate certificates signed by it.
  • Private key            (CA’s Private key)
Any CA uses this key to digitally sign the certificates it provides.

Final notes

  • TLS Server is the one that must have a public certificate if integrity and authenticity are to be achieved.
  • TLS Client only needs the corresponding CA Root certificate to validate public certificates.
  • Digital certificates are optional when transmitting a message over TLS. This is true only when the cryptographic algorithm used to exchange the Master-Secret is DH_anon, in all other cases a certificate is mandatory.
  • Digital certificates are not used to encrypt the SMTP conversation. TLS uses a symmetric encryption mechanism to encrypt the conversation. Certificates are only used to exchange the cryptographic information used to generate the symmetric key (Master-Secret).

Generating the Master-Secret

The Master-Secret is the crucial part on any TLS Communication. This is the key that both peers will use to encrypt and decrypt the SMTP Conversation. It is generated according to the method selected in the cipher suite exchange.

Cipher Suite. This is an ordered list of the cryptographic mechanisms a TLS Client is able to support. The order of the list reflects the priority in which they should be choose, being the first in the list the one with more chances to be picked up by the TLS Server. The parameters on this list indicate the algorithms used for the key exchange (RSA, DH, DHE, etc…), the algorithm used to encrypt the message (AES, DES, RC4, etc…) and the algorithm used to sign each message (SHA or MD5).

The type of encryption TLS will use depends on the combination selected by the SMTP Server.

When creating the Master-Secret the only algorithm that matters is the one for the key exchange. These may be classified in three groups:

Anonymous key exchange. This suite is identified by the “TLS_DH_anon_WITH” prefix on the cipher suite. There is no server authentication when this algorithm is used, meaning no digital certificate is required nor exchanged between both hosts. This is true because they use the Diffie-Hellman (DH) algorithm which was designed to exchange sensitive information between two servers trough an insecure medium. There is no need for certificates because its design allows for two unknown parties to exchange certain values that will allow them to generate the exact same secret by exchanging only certain public values while other are kept private.











p, g




p, g, A

ga mod p = A


p, g



p, g, A


gb mod p = B

p, g, A, B


a, s

p, g, A, B

Ba mod p = s

Ab mod p = s

p, g, A, B

b, s

Two servers choose a “Pre-Secret” which is a number generated by a pseudo-random function (PRF). This number is only known to each server and must be generated in such a way that there is no possibility (or it may be minimal) that the number may be repeated again. Then both servers exchange a pair of numbers that will be known by both, here the TLS Client send a ClientKeyExchange packet and the TLS Server a ServerKeyExchange packet. Based on this public information, the algorithm allows them to calculate exactly the same number that will be considered the Master-Secret without the need to exchange this value through the insecure medium and may become compromised. Even if an attacker can obtain the public information, it won’t be able to calculate the same Master-Secret because each server generated its own Pre-Secret and this number was never transmitted. This kind of communication is not recommended for Internet transactions because it lacks the authenticity feature of TLS. This means that even when it is possible to securely encrypt the SMTP conversation, it is not possible to authenticate the identity of any of the peers.

Authentication and RSA key exchange. In this mode, there is a RSA certificate exchange that allow both servers to both, authenticate and exchange the Master-Secret. This mode is identified by the “TLS_RSA_WITH_” prefix on the cipher suite. The TLS Server authentication takes place by extracting the digital signature of the public certificate and comparing it to the one generated by the TLS Client. For this validation to be successful, the TLS Client must have the appropriate CA Root certificate because is the only one that contains the CA public key.

Once the verification is done, the TLS Client generates a Pre-Master secret and encrypts it with the public key contained in the certificate. The only server that would be able to decrypt this packet will be the one in possession of the corresponding private-key, this prevents that even when an attacker may intercept this packet, it won’t be able to decrypt the Pre-Master secret because he lacks the private key. These steps are done for authenticating the TLS Server, but the TLS Client may also be authenticated. This is not recommended as not all TLS Clients have the right certificate or are prepared to respond to this request. Also, the Client authentication is not mandatory.

Public key and digital signature of the public certificate

The key exchange is completed when the TLS Client encrypts its Pre-Master secret with the Server’s public key. The server then decrypts the packet with its own private key and it’s now able to calculate the same Master-Secret as the Client.

Authentication with Diffie-Hellman keys exchange. These algorithms can be easily identified because the cipher suites begin with “TLS_DHE_DSS_WITH_”,  “TLS_DHE_RSA_WITH_”, “TLS_DH_DSS_WITH_” and “TLS_DH_RSA_WITH_”.  In this mode, the server may send a digital certificate that contains the Diffie-Hellman parameters or it may send them in the ServerKeyExchange packet signed with its private key. If the server uses the first option, it is said that this certificate is only for signing because it only contains the DH information to exchange the Pre-Master secret, the authentication phase is completed only when the client receives the Finished packet encrypted with the Master-Secret because at this point both server must have calculated the same Master-Secret. In the second option, the certificate is said to be used for both authentication and signing just as happens for RSA key exchange.

Once the Pre-Secret is known to both servers they calculate the final Master-Secret by their own using the following formula:

Master-Secret = PRF (pre_master_secret, “master_secret”, ClientHello.random + ServerHello.random)


For more information about how an email is generated, transmitted and processed you can check out our book on The SMTP Protocol Fundamentals.

Remember to send us your comments and questions to our Twitter account: @redinskala where you’ll find more information and security tips.

Thanks for your visit!