03 Nov Support queries shared: “SSL negotiation failed” errors in SMTP logs

Today’s nugget of MDaemon wisdom comes from conversations I’ve had with a small number of customers reporting they’re unable to send outbound email to specific recipients.

Upon closer inspection of the error in the SMTP (OUT) log, there’s a message reading “SSL negotiation failed“.

So what’s going on?

When MDaemon sends outbound email to a remote SMTP server, one of the first things that happens is the remote server advertises whether it supports the ability to encrypt the session using Secure Socket Layer (SSL). MDaemon fully supports this type of encryption, and it is good practice to encrypt sessions between servers when supported.

Essentially, during this stage something goes wrong with the negotiation, the session errors and then it closes.

Why are the sessions failing?

SSL can use a range of different encryption ciphers and before any data can be encrypted, both ends need to agree on a cipher to use. Problems will arise if a match cannot be agreed.

Historically one of the more popular ciphers was MD5 and this was widely used for SSL sessions. However, recently MD5 has been proven to be insecure in certain circumstances and so is often no longer included in the SSL library files Windows uses.

In an ideal world all SMTP servers should no longer use MD5 as well and instead should switch to a stronger Cipher such as AES, but in reality some servers will still be trying to negotiate MD5 and in some cases MD5 exclusively.

The workaround

So to reiterate, ideally you want to be asking the remote server admin to upgrade to more secure ciphers if that’s possible, but if it’s not, using MDaemon we’re able to come up with a workaround on a site by site basis.

One method would be to simply add support for the MD5 cipher back into the sending server but I don’t like this as it’s bad practice. Instead, what I suggest is you instruct MDaemon not to use any SSL encryption when talking to the specific failing server.

Once you know the mail host names for these servers, they can be added to the MDaemon SSL white list which can be found under…

Security -> Security Settings -> SSL & TLS -> STARTTLS White List.

You can see in this example above that I’ve simply added a wildcard domain for  *.eircom.net this will cover any SMTP server on this domain (which was the failing server in our example) so now any new SMTP sessions MDaemon establishes to or from that specific host will now not use SSL, therefore not triggering the error.

Pingback: Support queries shared: MDaemon SSL errors – Windows XP and Server 2003