Introducing Certgrinder, a LetsEncrypt SSH Proxy

by Tykling

30. apr 2017 12:46 UTC

Background

Like many people I've been switching to LetsEncrypt for my certificate signing needs. I recently changed a bunch of LE related things. This post documents my new method of using the LetsEncrypt certbot client from a central location, with the certificate consumers (webservers etc) getting their certificates over SSH using a standard CSR. Much like when we were using commercial CAs.

This has a couple of important advantages over my old setup:

  • I only need to have the Certbot software stack installed in one place, simplifying my servers.
  • I don't expose my LetsEncrypt credentials on any machines with untrusted users on them.

Concept

I named the system Certgrinder, although it is more of an idea than a system as such. The machine with the LetsEncrypt credentials and software stack is called the Certgrinder server. It listens for SSH connections from the Certgrinder clients. A Certgrinder client generates an RSA keypair and a CSR, and over SSH uses the CSR to get a signed certificate. Nothing groundbreaking about it, and I expect variations of this idea to be running many other places.

Example

Lets begin with a simple example. Say I have a webserver which needs a certificate. It has one or more domain names (lets say example.com and example.org) in the DNS pointing to its v4 and v6 IPs. To get a certificate I have to prepare a few things. I need to generate an RSA keypair and an SSH keypair. I also have to configure the webserver proxy/redirect for the LE challenge.

I prefer to have this stuff under a dedicated user, so the steps are:

  1. Create a certgrinder user. The key and certificate will live in the homedir of the certgrinder user.
  2. Create an SSH key for the certgrinder user and add it to authorized_keys on the Certgrinder server.
  3. Create an RSA keypair for the server: openssl genrsa -out example.com.key 4096
  4. Add configuration to the webserver so it proxies or redirects requests for /.well-known/acme-chalenge/ to the Certgrinder server.

At this point getting a certificate is simple: Generate a CSR and cat it over SSH to the Certgrinder server, and it will output a signed certificate on stdout. There is no difference in the procedure for new certificates and "renews". Both mean a new CSR and new certificate.

  1. Generate a CSR for the domains in question. OpenSSL doesn't allow specifying SubjectAltName on the commandline so we have to do a few ugly hacks to avoid writing a new openssl.cnf every time: openssl req -new -sha256 -key example.com.key -subj "/C=DK/O=MyExampleOrga/CN=example.com" -reqexts SAN -extensions SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:example.com,DNS:example.org")) -out example.com.csr (Note: <(...) is a bashism)
  2. Use cat to send the CSR over stdin to the Certgrinder server, redirecting stdout to a file: cat example.com.csr | ssh certgrinder.tyknet.dk > example.com.crt
  3. Remember to configure and reload the webserver so it uses the new certificate. A reload is also needed after "renewing".

If you'd rather use my hacked up shell script instead of using your own I have put the steps above into this script which will generate a keypair and a CSR and then get a certificate. The script is designed to be run from crontab daily or weekly since it checks the expiry of the certificate before doing anything. If the certificate expires in less than 30 days a new CSR is generated and a new certificate is issued using the Certgrinder server.

Non-webservers

If a non-web TLS server needs a certificate I have a few options when handling the challenge. LetsEncrypt does a challenge over HTTP to the IP of the hostnames specified in the CSR, so I either have to install a webserver to do a redirect or proxy the request to the Certgrinder server, or I can "catch" the request in the firewall and TCP redirect it to a webserver which then does the HTTP redirect. Both methods have their merits and I use both in different situations. Whenever possible I prefer to do the redirect in the firewall, to avoid installing extra software

LetsEncrypts challenge checkers conveniently follow HTTP 301/302 redirects which means I can get away with a very simple web"server" to do the redirecting. Once the challenge proxy/redirect has been sorted the procedure is the same as above.

The Certgrinder Server

The Certgrinder server has the LetsEncrypt certbot software stack installed, and it has the credentials used for revocation and stuff. Network-wise it needs to be reachable over SSH from the Certgrinder clients, and over HTTP (possibly via a proxy if you want) from the LetsEncrypt challenge checkers. I follow the steps below to prepare it:

  1. Create a certgrinder user. The Certgrinder clients SSH pubkeys should go in .ssh/authorized_keys for this user, with appropriate restrictions. Something like from=2a01:3a0:1:1900:85:235:250:85,command=/usr/local/bin/csrgrinder,no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEAX6ArpY9CLqV4H1BmlikEcFVp9geDSeRNNdaEB57jL certgrinder@ircd.tyknet.dk
  2. Install the certgrinder script from https://github.com/tykling/certgrinder/blob/master/csrgrinder as /usr/local/bin/csrgrinder and make it executable
  3. Install the LetsEncrypt certbot software stack and install/generate credentials
  4. Configure a webserver to serve the LetsEncrypt challenge root

The Certgrinder server is stateless - it doesn't save anything when operating. It receives the CSR on stdin and saves it to a temporary file. It then uses the CSR to get a signed certificate to another temporary file. It then outputs the certificate on stdout and deletes both temp files before exiting.

Public Key Pinning

With this setup the keypair is never rolled, which makes it possible to do public key pinning. This is a major advantage, although it is not specifically related to centralizing the LetsEncrypt operations.

Varous pinning methods exist. I am a fan of TLSA which uses DNS for public key pinning. Some applications (like irssi) also allow you to pin the public key fingerprint directly in the configuration, very nice.

Security Considerations

LetsEncrypt certificates have three months validity, which means certificate pinning is not practical. Certbot defaults to rotating the RSA keys each time a certificate is renewed, so public key pinning is also out. The advantage of the short key lifetime is of course that a key compromise only affects a short time period. The disadvantage is that key pinning is not possible.

With Certgrinder the keys are not rotated so public key pinning is possible. In my opinion the added security of key pinning greatly outweighs the risk of a key compromise. YMMV.

An advantage of the centralised model of Certgrinder is that the LetsEncrypt credentials are never exposed on web- or other servers exposed to users with all the risks that that entails. The LetsEncrypt credentials are used for stuff like revocation so it is pretty important that they don't end up in the wrong hands.

Final Thoughts

Looking back over this I am amazed I didn't think of this sooner. It is almost like the days before LetsEncrypt: I just generate a keypair and a CSR, and get a signed certificate in exchange. It works very well and I am in the process of updating my Ansible roles and stuff to use Certgrinder rather than installing certbot everywhere.

A single Certgrinder server can be used to issue certificates for an unlimited number of Certgrinder clients, although I do keep different Certgrinder servers for different projects (so one for my personal stuff, one for UncensoredDNS, one for BornHack and so on).

Search this blog

Tags for this blogpost

pki letsencrypt certgrinder