30. apr 2017 12:46 UTC
Like many people I've been switching to LetsEncrypt for my certificate signing needs. I recently changed a bunch of LE related things. This post documents my new method of using the LetsEncrypt
certbot client from a central location, with the certificate consumers (webservers etc) getting their certificates over SSH using a standard CSR. Much like when we were using commercial CAs.
This has a couple of important advantages over my old setup:
I named the system
Certgrinder, although it is more of an idea than a system as such. The machine with the LetsEncrypt credentials and software stack is called the
Certgrinder server. It listens for SSH connections from the
Certgrinder clients. A Certgrinder client generates an RSA keypair and a CSR, and over SSH uses the CSR to get a signed certificate. Nothing groundbreaking about it, and I expect variations of this idea to be running many other places.
example.org) in the DNS pointing to its v4 and v6 IPs. To get a certificate I have to prepare a few things. I need to generate an RSA keypair and an SSH keypair. I also have to configure the webserver proxy/redirect for the LE challenge.
I prefer to have this stuff under a dedicated user, so the steps are:
authorized_keyson the Certgrinder server.
openssl genrsa -out example.com.key 4096
/.well-known/acme-chalenge/to the Certgrinder server.
At this point getting a certificate is simple: Generate a CSR and cat it over SSH to the Certgrinder server, and it will output a signed certificate on stdout. There is no difference in the procedure for new certificates and "renews". Both mean a new CSR and new certificate.
openssl req -new -sha256 -key example.com.key -subj "/C=DK/O=MyExampleOrga/CN=example.com" -reqexts SAN -extensions SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:example.com,DNS:example.org")) -out example.com.csr(Note: <(...) is a bashism)
catto send the CSR over
stdinto the Certgrinder server, redirecting
stdoutto a file:
cat example.com.csr | ssh certgrinder.tyknet.dk > example.com.crt
If you'd rather use my hacked up shell script instead of using your own I have put the steps above into this script which will generate a keypair and a CSR and then get a certificate. The script is designed to be run from crontab daily or weekly since it checks the expiry of the certificate before doing anything. If the certificate expires in less than 30 days a new CSR is generated and a new certificate is issued using the Certgrinder server.
If a non-web TLS server needs a certificate I have a few options when handling the challenge. LetsEncrypt does a challenge over HTTP to the IP of the hostnames specified in the CSR, so I either have to install a webserver to do a redirect or proxy the request to the Certgrinder server, or I can "catch" the request in the firewall and TCP redirect it to a webserver which then does the HTTP redirect. Both methods have their merits and I use both in different situations. Whenever possible I prefer to do the redirect in the firewall, to avoid installing extra software
LetsEncrypts challenge checkers conveniently follow HTTP 301/302 redirects which means I can get away with a very simple web"server" to do the redirecting. Once the challenge proxy/redirect has been sorted the procedure is the same as above.
The Certgrinder server has the LetsEncrypt certbot software stack installed, and it has the credentials used for revocation and stuff. Network-wise it needs to be reachable over SSH from the Certgrinder clients, and over HTTP (possibly via a proxy if you want) from the LetsEncrypt challenge checkers. I follow the steps below to prepare it:
.ssh/authorized_keysfor this user, with appropriate restrictions. Something like
from=2a01:3a0:1:1900:85:235:250:85,command=/usr/local/bin/csrgrinder,no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEAX6ArpY9CLqV4H1BmlikEcFVp9geDSeRNNdaEB57jL firstname.lastname@example.org
/usr/local/bin/csrgrinderand make it executable
The Certgrinder server is stateless - it doesn't save anything when operating. It receives the CSR on stdin and saves it to a temporary file. It then uses the CSR to get a signed certificate to another temporary file. It then outputs the certificate on stdout and deletes both temp files before exiting.
With this setup the keypair is never rolled, which makes it possible to do public key pinning. This is a major advantage, although it is not specifically related to centralizing the LetsEncrypt operations.
Varous pinning methods exist. I am a fan of
TLSA which uses DNS for public key pinning. Some applications (like
irssi) also allow you to pin the public key fingerprint directly in the configuration, very nice.
LetsEncrypt certificates have three months validity, which means certificate pinning is not practical. Certbot defaults to rotating the RSA keys each time a certificate is renewed, so public key pinning is also out. The advantage of the short key lifetime is of course that a key compromise only affects a short time period. The disadvantage is that key pinning is not possible.
With Certgrinder the keys are not rotated so public key pinning is possible. In my opinion the added security of key pinning greatly outweighs the risk of a key compromise. YMMV.
An advantage of the centralised model of Certgrinder is that the LetsEncrypt credentials are never exposed on web- or other servers exposed to users with all the risks that that entails. The LetsEncrypt credentials are used for stuff like revocation so it is pretty important that they don't end up in the wrong hands.
Looking back over this I am amazed I didn't think of this sooner. It is almost like the days before LetsEncrypt: I just generate a keypair and a CSR, and get a signed certificate in exchange. It works very well and I am in the process of updating my Ansible roles and stuff to use Certgrinder rather than installing
A single Certgrinder server can be used to issue certificates for an unlimited number of Certgrinder clients, although I do keep different Certgrinder servers for different projects (so one for my personal stuff, one for UncensoredDNS, one for BornHack and so on).