I just stumbled across an issue with Let’s Encrypt certificates being used on my personal server for securing IMAP and SMTP. I decided to share it here since, in my opinion, it’s a hard to grab nut because I did not really find clear articles pointing this out during my own debugging.

What has happened?

A few days ago, a friend of mine, with who I share my personal Mailserver, contacted me telling me that out of a sudden his Android Mail app (K-9 Mail) stopped working. At the same time, I had no issues with Thunderbird on MacOSX Big Sur, nor Apple Mail on iOS 14.2 nor Roundcube.

Anyways, I started to verify everything: I looked for the state of certificates, from Let’s Encrypt to verify certbot didn’t fail to refresh them, I checked the Dovecot configuration if it is configured to use the appropriate SSL certs, I checked the content of the SSL certs used on the server using OpenSSL (openssl x509 -in certificate.crt -text -noout ); all appeared OK and I had to tell my buddy that I do not have a clue what his issue might be.

Today, I did something that I rarely do: I started my Thunderbird Mail app on my Windows 10 Gaming PC. Since it’s my gaming PC and the only Windows PC I have in use, I didn’t do this in my routine earlier. I noticed that I had zero emails in my Inbox.
That’s not a too uncommon thing, since I do a quite good job at keeping my Inbox clean by sorting read mail into subfolders. But this time, I knew there is mail since I checked on Apple Mail before.

Where are my emails?? 

Now I was able to reproduce it. Again I searched for an issue on my server but – nothing found! Everything seemed correct. Also, I am running the same setup for several years now and didn’t change anything lately; so: why should it break?

The reason

… Maybe I didn’t change anything, but it turned out: Let’s Encrypt did!

This time, I also tried to grab the certificate from “the outside”, using OpenSSL on my Mailserver, but pointing to its official public DNS name instead of checking the cert by its local file:

~ # openssl s_client -connect mail.marc-richter.info:993 -quiet | echo
depth=0 CN = mail.marc-richter.info
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = mail.marc-richter.info
verify error:num=21:unable to verify the first certificate
verify return:1
~ #

Aha – the server itself seems to have some trouble with the local copy of the issuer certificate?
I then checked the journald entries:

Dez 21 19:32:56 zzzzz.zen-net.de dovecot[3331742]: imap-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy, TLS handshaking: SSL_accept() failed: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown: SSL alert number 46, session=<D1nY...

By sheer luck, I found this article in the Let’s Encrypt blog, informing about that they introduced a new intermediate signing instance around November 24th 2020.
I already noted that the certificate issuer in my browser instances now says “R3” for some of my sites instead of “Let’s Encrypt“, but since that worked without any issues, I didn’t pay too much attention. But for some reason, it seems to cause trouble with some IMAP clients / operating system combinations out there.

The solution

For me (Thunderbird on Windows 10), it solved my issue to download and import the R3 intermediate certificate from https://crt.sh/?id=3479778542 (the link they named in their blog post – click on “Certificate”). Out of a sudden, Thunderbird acted as if there hadn’t been an issue before.
You can import that certificate into the Windows certificate store by double-clicking on it. But that’s not enough for Thunderbird: Since it uses its own certificate store, you need to import it to Thunderbird as well:

Go to Menu (3 black bars in the upper right corner) -> Options -> Privacy & Security -> Scroll down to “Security” and click “Manage certificates…“. In the “Certificate Manager” dialogue, click “Import …” and select the downloaded *.crt file.

That should do the trick.
I hope this quick-tip saves you some time and headaches.