OCSP Stapling in Nginx

by Tykling

22. oct 2013 23:24 UTC

Background

This post is about getting OCSP Stapling to work in nginx. OCSP is short for Online Certificate Status Protocol and is a close to realtime method of checking an TLS certificates validity.

This blogpost is based on nginx 1.4.3 (you need at least nginx 1.3.7 for OCSP stapling to work) compiled against OpenSSL 1.0.1e in a FreeBSD 9.2 jail. To compile nginx against a newer OpenSSL than the one included in the FreeBSD base system, simply install /usr/ports/security/openssl and rebuild nginx from ports, it will pick up the newer libcrypto.so automatically.

Keep in mind that all this relates to TLS certificate validity checking, OCSP is not relevant for servers with self-signed certificates.

OCSP

These days an TLS client has two ways of checking if the certificate it has been presented with has been revoked. The old way involves downloading and parsing Certificate Revocation Lists, and the new way is OCSP. Clients continue to use both methods, but they prefer OCSP over CRLs where possible, because OCSP provides more realtime infomation.

Using OCSP to check certificate validity is possible when a certificate has an authority information access extension containing an OCSP url (most certificates from an OCSP enabled CAs have this now).

When OCSP is possible the client connects to the CA right after receiving the certificate and to check validity. This is close to realtime, especially when compared to the CRLs which are typically updated weekly.

OCSP has a few problems though:

• There are huge privacy issues with it: The CA will get an OCSP request from every client that does an OCSP check, effectively leaving the CA with a list of all the client IP addresses of a given websites visitors. CA's are trusted by clients, but still, this needs to be fixed.
• It requires an extra DNS lookup and an extra TCP connection on the client, making TLS connections slower.

OCSP stapling

OCSP stapling gives TLS website operators a way of protecting their users privacy while improving performance and keeping the same level of security, a rare combination indeed.

Enabling OCSP stapling on your webserver protects your sites visitors privacy by giving them an up-to-date assurance of the certificates validity, without forcing them to sacrifice their privacy by asking the CA directly. In other words, OCSP stapling gives the same security as regular OCSP, but without the privacy issues.

With OCSP stapling the TLS server (nginx in this case) periodically asks the CA's OCSP server for the revocation status of the certificate it is serving, and the CA server replies with a signed and timestamped OCSP response. This response is saved in a cache in nginx and it is sent along with the certificate (stapled to it) to the clients when they connect.

The client uses the CA signature and the timestamp to verify the validity of the response. The client now knows that the certificate is valid without making any extra requests, and without sacrificing privacy.

OCSP stapling removes a bit of the realtime-ness of OCSP though, since the response is cached on the webserver. nginx caches responses for one hour, which is still whole lot better than the full week that my current CA has between the Effective date and Next update fields of the CRL for the certificate for this blog. Without OCSP that means a whole week could pass between my certificate being revoked and my clients being told about it!

Debugging OCSP responses

First of all, it is important to have a good way of debugging this. I prefer the openssl commandline client but the output can be a bit verbose without a bit of grep.

With OCSP stapling working:

[tykling@web ~]$ /usr/local/bin/openssl s_client -connect blog.tyk.nu:443 -servername blog.tyk.nu -status < /dev/null 2>&1 | grep -i "OCSP response"
OCSP response:
OCSP Response Data:
 OCSP Response Status: successful (0x0)
 Response Type: Basic OCSP Response
[tykling@web ~]$

With OCSP stapling disabled or misconfigured:

[tykling@web ~]$ /usr/local/bin/openssl s_client -connect blog.tyk.nu:443 -servername blog.tyk.nu -status < /dev/null 2>&1 | grep -i "OCSP response"
OCSP response: no response sent
[tykling@web ~]$

The reason why -servername is needed in the command above is because I use Server Name Indication (SNI) which is how you run several TLS websites on the same IP+port.

nginx gotchas

Now, the nginx documentation on this is decent, but I ran into a couple of problems that I wanted to document here.

This assumes that you already have one or more TLS websites running in nginx, and you want to enable OCSP stapling for (some of?) them. This is where I hit the first problem. If you have a default TLS server in nginx (which you should always have) then that server needs to have OCSP stapling enabled or it will not work for any server. Consider the configuration below:

[tykling@web ~]$ grep listen /usr/local/etc/nginx/vhosts/*.conf | grep 443 | grep 178\\.63
/usr/local/etc/nginx/vhosts/blog.tyk.nu.conf: listen 178.63.198.117:443 ssl spdy;
/usr/local/etc/nginx/vhosts/buddyledger.baconsvin.org.conf: listen 178.63.198.117:443 ssl spdy;
/usr/local/etc/nginx/vhosts/pad.tyk.nu.conf: listen 178.63.198.117:443 ssl spdy;
/usr/local/etc/nginx/vhosts/web.tyknet.dk.conf: listen 178.63.198.117:443 ssl spdy default_server;
[tykling@web ~]$

Given this config I will need to enable OCSP stapling in the server defined in web.tyknet.dk.conf (as it has the default_server flag) before it will work in any of the other servers. Also worth noting:

• The old "ssl on;" syntax is deprecated, you should always put ssl directly in the listen line.
• I have SPDY enabled on my TLS servers. This does not affect OCSP stapling though.

Another thing that cost me more time than it should have is the OCSP cache in nginx. Remember how I explained how nginx periodically requests the OCSP information from the CA and caches the response ? Well that cache needs to be primed after nginx has been restarted. This means that even if nginx is properly configured, the first TLS request will never result in a reply with the OCSP response stapled to the certificate. This means: Always test twice after making changes and restarting nginx! Observe:

[tykling@web ~]$ /usr/local/bin/openssl s_client -connect blog.tyk.nu:443 -servername blog.tyk.nu -status < /dev/null 2>&1 | grep -i "OCSP response"
OCSP response:
OCSP Response Data:
 OCSP Response Status: successful (0x0)
 Response Type: Basic OCSP Response
[tykling@web ~]$ sudo /usr/local/etc/rc.d/nginx restart
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Stopping nginx.
Waiting for PIDS: 57497.
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.
[tykling@web ~]$ /usr/local/bin/openssl s_client -connect blog.tyk.nu:443 -servername blog.tyk.nu -status < /dev/null 2>&1 | grep -i "OCSP response"
OCSP response: no response sent
[tykling@web ~]$ /usr/local/bin/openssl s_client -connect blog.tyk.nu:443 -servername blog.tyk.nu -status < /dev/null 2>&1 | grep -i "OCSP response"
OCSP response:
OCSP Response Data:
 OCSP Response Status: successful (0x0)
 Response Type: Basic OCSP Response
[tykling@web ~]$

Note how the first request after a restart fails. Do not get fooled by this like I was :)

nginx configuration

With all that out of the way we finally get to the actual configuration. Relatively few options control the OCSP functionality:

• ssl_stapling enables and disables fetching and stapling OCSP responses to certificates, set to on to enable.
• ssl_stapling_verify enables verification of the OCSP responses received from the CA. You want this enabled because the responses are transported over plain HTTP and could potentially be falsified by an attacker. Set to on to enable. This should really default to being enabled!
• ssl_trusted_certificate should contain your CA root certificate and any intermediate certificates needed to validate your server certificate. nginx uses this to verify the OCSP response received from the CA. If you have an intermediate certificate in your configured ssl_certificate file, you will need that same intermediate certificate in the ssl_trusted_certificate file. The order of the certificates does not seem to matter, it worked for me with CA root cert first followed by the intermediate certificates, and the other way around. The file should look like so:

-----BEGIN CERTIFICATE-----
...CA ROOT PEM certificate here...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...CA intermediate PEM certificate here...
-----END CERTIFICATE-----

(or the other way around).

Failing to specify ssl_trusted_certificate will result in an entry in error_log that looks a bit like this when trying to access the server with OCSP enabled but without a way to verify the response:

2013/10/23 11:07:24 [error] 61770#0: OCSP_basic_verify() failed (SSL: error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error:Verify error:unable to get local issuer certificate) while requesting certificate status, responder: ocsp2.globalsign.com

This is because the intermediate certificate is not marked as trusted, resulting in a failure when trying to verify the OCSP response. Note that nginx only logs errors like these once per client, so repeating the same test will not result in more logentries before nginx is restarted. How many identical messages do you need anyway ?

Final thoughts

That was a lot of words about something which is basically pretty simple to configure in nginx. Generally nginx is very simple to configure, once you understand the technology you are configuring! It can be difficult to just copy/paste a nginx config snippet and have it work, well that is because you are not supposed to do that, which is also the reason I haven't included a complete OCSP enabled TLS config here. You are supposed to understand the technologies in use on your server! Want to enable OCSP ? Read up on how it works before trying to enable it, it will make everything a lot easier. A big part of being a good system administrator is familiarizing yourself with the technologies and concepts you work with.

In my opinion, this is part of what makes nginx so awesome. There is a definite no-bullshit approach to the configuration. Once you get used to nginx it it is almost painful to look at an apache configuration file (look at try_files in nginx compared to the nightmare that is mod_rewrite in Apache, for example). nginx is beautifully clean and simple to configure, with sensible defaults, just as I like it!