When email was first introduced over the Internet, it did not support any kind of encryption: all messages would travel over the network in clear. This applied both to mail transmission protocols (e.g. SMTP) and to mail retrieval protocols (e.g. POP3, IMAP).
Later, when security needs arose and SSL/TLS came into use, different approaches were taken. The first idea was to set up TLS-protected connections using different port numbers; while clients using cleartext communications would connect to the traditional ports 25 (SMTP), 110 (POP3) and 143 (IMAP), newer clients wishing to encrypt the communication would connect to the new port numbers 465 (SMTPS), 993 (IMAPS) and 995 (POP3S) – just like HTTPS connections use port 443 instead of the traditional HTTP port 80.
However, it was soon realized that this created additional issues for server management, duplicating the ports in use, and especially posed a significant problem for server-to-server mail transmission, as it would be impossible for a server to know in advance whether the other server supported encryption or not. Thus, a different mechanism was deployed: STARTTLS.
STARTTLS is an extension to the mail transmission protocols that allows the client to request the server to start encrypting the connection with TLS. The connection is still established in clear, but the server, in response to the initial salutation or capability request command (e.g. EHLO for extended SMTP), will provide STARTTLS as one of the available services. At that point in time, if the client supports encryption, it may send the STARTTLS command, and a TLS connection will be negotiated using the receiving server’s certificate; all subsequent traffic will move onto the TLS connection, and will be exchanged in encrypted form.
This mechanism allows for backward compatibility: if a client not supporting TLS connects to the server, it will not send the STARTTLS command and all the exchange will continue in clear text. This paradigm is called opportunistic encryption: it means that the connection will be encrypted only if both sides agree to do so, but otherwise it will not be encrypted.
The problem with opportunistic encryption is that it is vulnerable to a man-in-the-middle attack. If an attacker intercepts network traffic as it exits the originating mail server, he can impersonate the destination server and reply that he does not support STARTTLS. At that point, the originating mail server will simply send the message in clear, allowing the attacker to read it. Moreover, the attacker can then forward the message to the true destination server, so that the legitimate receiver will never realize that the message has been intercepted.
Also, since the originating server has no way to know in advance that the legitimate destination server supports encryption, there is no way for the originating server to refuse sending the message in clear if the other side says that it does not accept STARTTLS.
But there’s more: even if the originating mail server refused to send the message over an unencrypted channel, the attacker in the middle would still be able to intercept it, as the originating server has no way to verify the identity of the server it is talking to. STARTTLS does not include any mechanism to verify the destination server’s identity; even if the destination server sends a valid cryptographic certificate when establishing the TLS connection, it can just be a self-signed certificate with whatever hostname and information written into it, and thus pretend to represent any other server.
The only way to avoid such an attack is to:
- know in advance that the destination server supports encryption, and refuse sending messages to it over an unencrypted channel;
- have a way to verify the identity of the destination server.
Also, another issue with STARTTLS is that it allows the client to authenticate itself before establishing the secure connection, thus sending access credentials (e.g. passwords) in clear over the Internet. For this not to happen, it is necessary that the server does not offer the possibility to authenticate before the STARTTLS command is issued and successfully completed.
For clarity, we will also point out that the separate mail submission service, using SMTP on port 587, is not related to encryption; it is an ordinary SMTP service, which may or may not offer STARTTLS, that can be configured with a policy different than the one for the main SMTP service on port 25, for example by opening port 587 only towards a private network and accepting mail for relay.
