Let's Encrypt Wildcard Certificates On FreeBSD With BIND DNS Validation

Large browser vendors have quite some time ago established model of "trusted" web sites based on SSL certificates signed by "trusted" CAs. This trend increasingly spreads to other applications and protocols. This article gives exact steps needed for obtaining Let's Encrypt's wildcard certificates from FreeBSD host, using DNS validation through dynamic updates of a BIND server's zones, using EFF's ACME client, certbot.

Before Let's Encrypt there was StartSSL

Before I switched to Let's Encrypt, I have been using StartSSL's free service, which was offering free "trusted" certificates that could be used to "secure" single second-level domain (eg. mimar.rs) and its single arbitrary subdomain (eg. www.mimar.rs). In order to obtain these, one would need to register account on StartSSL's website and do some 15 minutes of clicking around. But, less because they lasted whole year, contrary to Let's Encrypt's 90-day lifetime certificates, and more because they happened to be the only free "trusted" certificates around I was aware of, I was satisfied with their service. However, in autumn of 2016, StartSSL got themselves into a scandal which resulted in loss of "trust" by major players like Apple, Google, Microsft, Mozilla and others, and ultimately closed their business on 1st of January, 2018.

Subdomain killed a subfolder star

In summer of 2016, FreeBSD ports tree has already been offering a choice of various ACME clients, of which security/dehydrated caught my eye, mostly because of short list of its dependencies. It took me no more than an hour to set .well-known/acme-challenge aliases in apache and configure dehydrated for http-based validation, after which I looked in awe how an one-liner gets me my certificates in a matter of seconds. But it didn't take long before I started to realize there is much more to Let's Encrypt than just shortening time needed to get my certificates. The newly obtained power to efortlessly get single certificate for a myriad of subdomains got me to rethink my design of placing various web services, such as Nextcloud, phpMyAdmin and Roundcube, into aliased subfolders (e.g. https://www.mimar.rs/roundcube), and I started to give them their own subdomains by means of apache name-based virtual hosts (e.g https://roundcube.mimar.rs). Furthermore, configuration of those apache vhosts could be greatly simplified by pointing all of the subdomain vhosts to the same set of certificates. With setup being so simple, it didn't take long before I started to obtain Let's Encrypt certificates for dozens of my clients' domains and subdomains in a completely automated way, with single crontab line.

Enter wildcard

So, as far as web was concerned, things regarding "trusted" certificates went really well. But I wanted to push things further. Or have I been pushed to push them? I guess those two don't have to be mutually exclusive. Either way, there's much more to the Internet than just web, isn't it? What about other protocols like XMPP? SMTP? IMAP? LDAP? They all suppport secure communication which make use of SSL certificates, either via STARTTLS, or protocol over SSL, and I have been serving them securely for years, using self-signed certificates. However, developers of these services' client software, perhaps unfortunately, followed the path of big browser vendors, and started to warn users about self-signed certificates as "untrusted" as well. With Let's Encrypt around, it was not too hard for me to obtain certificates for subdomains offering these services, but not as easy as single crontab line, either. I had been considering setting up temporary apache vhosts for subdomains which otherwise didn't offer web services, lasting for the duration of domain validation and certificate signing. But then another game changer came into play - as of March 2018, Let's Encrypt started to offer wildcard certificates, eliminating the need to validate each and every subdomain. There's a catch though - they can be only obtained through dns validation via ACME v2, while my solution relied on http validation via ACME v1 protocol.

Tooling switch

Now, although dehydrated supports both ACME v2 protocol and DNS validation, and also provides multiple hook examples which should provide convenient way to validate domains over DNS, I had a hard time figuring out underlying principles from its documentation and configure it for DNS validation. At one point I gave up and started to look for alternatives. Searching the web for related keywords brought my attention to EFF's ACME client which recently changed its name to certbot. Well structured user guide looked promising, but after I saw that the plugin dns-rfc2136, which was supposed to suit my needs, also has good documentation, and that FreeBSD ports tree already includes security/py-certbot-dns-rfc2136, I knew I had to give it a spin. To my amazement, I got my wildcard certificates with everything configured in a matter of minutes (as opposed to futile hours misspent with dehydrated's hooks, but that's is my fault, not dehydrated's), adding just a few lines to my BIND's configuration prior to runing a oneliner. Here's how that looked in a shell.

Oh, one more thing. I run all my services in separate jails. In order to minimize hassle, I run certbot directly on jail host, and mount certs directory via nullfs to all the jails needing it. As jail users will have access to sensitive files, not only world-readable private keys obtained from Let's Encrypt, but also secret needed for BIND zone updates, and Let's Encrypt account credentials, it is assumed that only trusted people log into them.

Now let's get our hands dirty.

Boring stuff part I - BIND

On BIND jail, use tsig-keygen to generate new TSIG key. This can be done in home directory as a non-privileged user.

tsig-keygen -a hmac-sha512 letsencrypt

Complete output of above command needs to be added to our named.conf:

key "letsencrypt" {
    algorithm hmac-sha512;
    secret "i/KJzuKiYGbWlZ7x3qR6OxXjEX3L8XdeOy90F3aFYtRaGJp/8Ig4HmP/WfG6WsFqoe+a31emvJWeapxmNbnDmA==";
};

We also need to configure appropriate update-policy in configuration of any zones we want to have validated:

zone "example.org" {
  type master;
  file "/usr/local/etc/namedb/dynamic/example.org.db";
  allow-transfer { 192.0.2.2; };
  update-policy {
    grant letsencrypt name _acme-challenge.example.org. txt;
  };
};

After restarting named our BIND DNS server is all set up - we can move on to installing and configuring certbot.

Boring stuff part II - Certbot

On BIND jail host, install required package by typing below command as root:

pkg install security/py-certbot-dns-rfc2136

Certbot does not need elevated privileges in order to obtain certificates. We could use personal unprivileged account to run it, but I prefer to create dedicated account just for this purpose, after OpenBSD's founder, Theo de Raadt, advised me so. I believe he would know a thing or two about security :)

pw groupadd certbot -g 20001
pw useradd certbot -u 20001 -c "Certbot unprivileged user" -d /nonexistent -g certbot -s /usr/sbin/nologin

We also need to create certbot's config dir and change its ownership accordingly:

mkdir /usr/local/etc/letsencrypt
chown certbot:certbot /usr/local/etc/letsencrypt/

A file named rfc2136.ini (or any other name which will be passed as a switch to certbot command later on) needs to be created with the following contents, placed into above config dir, chowned to certbot:certbot and chmoded to 600:

dns_rfc2136_server = 192.0.2.1
dns_rfc2136_name = letsencrypt
dns_rfc2136_secret = i/KJzuKiYGbWlZ7x3qR6OxXjEX3L8XdeOy90F3aFYtRaGJp/8Ig4HmP/WfG6WsFqoe+a31emvJWeapxmNbnDmA==
dns_rfc2136_algorithm = HMAC-SHA512

Default paths in current certbot version in FreeBSD (2.8.0 at the time of this article's last update) respect FreeBSD's standards, but they are not created on install. Let's create them, and give them appropriate ownership and permissions:

mkdir /var/log/letsencrypt
mkdir /var/db/letsencrypt
chown certbot:certbot /var/log/letsencrypt/
chown certbot:certbot /var/db/letsencrypt/

Time to obtain our certificates (assuming appropriate sudo privileges are in place):

sudo -u certbot certbot certonly --dns-rfc2136 \
  --dns-rfc2136-credentials /usr/local/etc/letsencrypt/rfc2136.ini \
  --server https://acme-v02.api.letsencrypt.org/directory \
  --email hostmaster@example.org \
  --agree-tos --no-eff-email \
  --domain 'example.org' --domain '*.example.org'

If everything went well, we should get the following message:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for example.org
dns-01 challenge for example.org
Waiting 60 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /usr/local/etc/letsencrypt/live/example.org/fullchain.pem
   Your key file has been saved at:
   /usr/local/etc/letsencrypt/live/example.org/privkey.pem
   Your cert will expire on 2018-07-18. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

If something went wrong, we can check logs in /var/log/letsencrypt/.

Boring stuff part III - smuggling stuff to jails

Services on jail host using newly obtained certificates can now be pointed to certs in /usr/local/etc/letsencrypt/. What remains to be done is to provide read-only access to them from jails as well. This can be done by means of exec.prestart and exec.poststop options in jail host's global section of jail.conf:

exec.prestart   = "/sbin/mount_nullfs -o ro /usr/local/etc/letsencrypt \
                   /usr/jail/${host.hostname}/usr/local/etc/letsencrypt";
exec.poststop   = "/sbin/umount /usr/jail/${host.hostname}/usr/local/etc/letsencrypt/";

Now all the jails will have same path to letsencrypt folder as jail jost.

Obtaining wildcard certificates for multitude of domains could be configured with trivial adjustments to described setup, namely configuring their appropriate DNS zones for dynamic updates and passing these domains to certbot command.

Boring stuff part IV - certificate renewal

This is as easy as typing below command as standard user with appropriate sudo rights:

sudo -u certbot certbot renew

This will renew all the certificates we have previously requested. If you made it this far, than setting up certbot's unprivileged user's crontab should be a no-brainer. Restarting services in jails so that they load fresh certificates can be done with a bit of scripting where we will be checking if new certificates were issued, and restarting appropriate services or complete jails.

Afterword

You have probably noticed that throughout the article I sometimes quoted words security and trusted. In my personal opinion, Let's Encrypt certificates don't offer security, at least no more than self-signed certificates do. Furthermore, on contemporary Internet, where trust is increasingly being built only for the purpose of its later cash-in (I'm not going to call any names here but go figure), I don't trust freshly arrived, heavily promoted, Internet companies. Actually, as we say here in Serbia, I don't trust anyone until we eat a bag of salt together (It takes a lot of time, and a lot of physical contact, to eat a bag of salt with someone). I don't have enough knowledge in the area of cryptography to be able to know for sure there aren't any technical weaknesses that can be exploited, but I believe that power to secure communications should not be entrusted exclusively to corporations. What I got with above setup is compliance with requirements dictated by corporations, and maybe trust from non-technical people using my services, but not trust in general, as I don't trust these certificates, signed by what comes down to "some people on the Internet" more than I would trust those I self-signed, or those which I know were signed by people I trust. And security? Time will tell.