by Tykling
25. feb 2018 13:34 UTC
SSHFP records has been around for a long time. They were first defined in RFC4255 and conceptually they are very similar to DANE/TLSA. It is basically a way to pin a hash of the SSH public key in the DNS, to allow clients to verify the public key they see when connecting. This means that I can avoid the unhelpful SSH host fingerprint message that we are all used to seeing when we SSH to a new host.
So instead of seeing:
tyk@laptop $ ssh people.bornhack.org The authenticity of host 'people.bornhack.org ()' can't be established. ECDSA key fingerprint is SHA256:i9RJZMuAzCYUQCClgjBkWQK5itFx/sGVIvBw5p8XTQI. Are you sure you want to continue connecting (yes/no)? ^C tyk@laptop $
I see this:
tyk@laptop $ echo "VerifyHostKeyDNS yes" >> ~/.ssh/config tyk@laptop $ ssh -v people.bornhack.org ...snip... debug1: Server host key: ecdsa-sha2-nistp256 SHA256:i9RJZMuAzCYUQCClgjBkWQK5itFx/sGVIvBw5p8XTQI debug1: found 3 secure fingerprints in DNS debug1: matching host key fingerprint found in DNS ...snip... [tykling@people2 ~]$
Note: I ran ssh with -v above because when this works there is no output from ssh at all even when connecting first time. Verbose mode just helps to show what is happening.
For years I've been wanting to use SSHFP on my infrastructure but I haven't had a good way of managing it. Publishing SSHFP records for all 3 currently enabled algorithms (rsa, ecdsa, ed25519) across 100+ jails and servers is only the beginning of it.
This is where the Ansible server comes into play. Ansible has SSH access to all my servers and jails anyway, since that is how it works. I already run a script every night on my Ansible server which loops over hosts in inventory files. This script should be easy to expand to also check SSHFP records and send an email if anything needs fixing.
I just need something to connect to a host over SSH and check which host keys it offers, calculate the corresponding SSHFP records, and check if they can be found in DNS. My existing checks already use the exitcode (0 for success, non-0 for failures) and I want to do something similar for the SSHFP check.
Before getting into the tools it is always a good idea to know how to do something manually, so lets take a look at how to generate SSHFP records with the builtin OpenSSH tools.
OpenSSH comes with tools that are capable of generating SSHFP records with no third party tools. Two different tools are in play here:
ssh-keygen -r can generate SSHFP records. By default it shows SSHFP records for the public keys it finds on the local systems sshd. It also supports using -f to point it at a file containing a public key retrieved from a remote server.ssh-keyscan can be used to retrieve public keys from remote ssh servers.In combination these two tools can generate SSHFP records for a remote server, although it is not as simple and straightforward as I've come to expect from OpenSSH in general. This is mainly because ssh-keyscan outputs a different format than ssh-keygen expects.
The ssh-keygen(1) manpage says about the -r switch:
-r hostname
Print the SSHFP fingerprint resource record named hostname for the specified public key file.
This had me confused for a while. Somehow I thought that it connected to the hostname, but it turns out that the hostname is only used in the output after the SSHFP records has been generated. The hostname has absolutely no impact on the generation of the SSHFP records - they are generated based on the local ssh host keys, or whatever is found in the file specified with -f. Observe:
[ansible@ansible2 ~]$ ssh-keygen -r people2.servers.bornhack.org people2.servers.bornhack.org IN SSHFP 1 1 e4cad1143fa9ff1e2941ddb81a0a11e0126c69a3 people2.servers.bornhack.org IN SSHFP 1 2 efd65ee8e1eacad017ddfe776ffb7dac332d103a46cc3915fc34d5e1c24f77b1 people2.servers.bornhack.org IN SSHFP 3 1 6fe82444672abe3840418fd2fef6cb89a7b9064f people2.servers.bornhack.org IN SSHFP 3 2 d9b5f19a98ca8fdad01dc50d18f1ce0820e5dd29630fde549e63d98ca5d561e7 people2.servers.bornhack.org IN SSHFP 4 1 ad7e9e70fc5f50327988fdd9680f7766d09272bc people2.servers.bornhack.org IN SSHFP 4 2 0e41969e52e821643b6dadb26a802cdb93772bdd25550635cd8e0dfa80b35a27 [ansible@ansible2 ~]$ ssh-keygen -r invalid.example.com invalid.example.com IN SSHFP 1 1 e4cad1143fa9ff1e2941ddb81a0a11e0126c69a3 invalid.example.com IN SSHFP 1 2 efd65ee8e1eacad017ddfe776ffb7dac332d103a46cc3915fc34d5e1c24f77b1 invalid.example.com IN SSHFP 3 1 6fe82444672abe3840418fd2fef6cb89a7b9064f invalid.example.com IN SSHFP 3 2 d9b5f19a98ca8fdad01dc50d18f1ce0820e5dd29630fde549e63d98ca5d561e7 invalid.example.com IN SSHFP 4 1 ad7e9e70fc5f50327988fdd9680f7766d09272bc invalid.example.com IN SSHFP 4 2 0e41969e52e821643b6dadb26a802cdb93772bdd25550635cd8e0dfa80b35a27 [ansible@ansible2 ~]$
Note how the same SSHFP records are being generated both times, only the hostname in the output changes. The SSHFP records it generates are for public sshd keys found on the local machine (which happens to be called ansible2).
To get it to generate SSHFP records for a remote server, which is what I want here, first I need to get the remote SSH public key and put it in a temporary file, which I then pass to ssh-keygen. Getting remote SSH public keys is a job for ssh-keyscan.
ssh-keyscan connects to remote SSH servers, fetches the public ssh host keys for one or more algorithms, and prints them to stdout, along with a comment about the keytype to stderr. A normal invocation looks like this:
[ansible@ansible2 ~]$ ssh-keyscan people2.servers.bornhack.org # people2.servers.bornhack.org:22 SSH-2.0-OpenSSH_7.5 FreeBSD-20170903 people2.servers.bornhack.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGvLsvX486xQxSz1oKleVSuOqLIu4z6QcIdJhnDJOsKICxiw14UfKtcxmZ5nCoS3GGyTXaHa0TNTpR0uFZQRD/M= # people2.servers.bornhack.org:22 SSH-2.0-OpenSSH_7.5 FreeBSD-20170903 people2.servers.bornhack.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjmezS6ONDHDcObSWyu4EZ2Zbf1NGMpJ/5RA5eERsulEAu1st38PCX8tGecxF5v/z/VbzTntGGuW7V2mCz/zRtH+xu68aXGlEqRMmXIzAS60yIb9DxWilyPwpwcelG6+4Jw2Z3kEyeodvUgdEycfnSa+2s76Bqv0avCicd+u2yK6tb3RARRH4bN70y8amWv/nAFVPRUVWYpKkeVj0DbrzCxria4KrtLlLcpLjFcDAjgwKvW2mS/h+Q6KzckSljXyr8zCwIIKlFrRKvvTrIkiEviTTos4znMDTZO36I0WhVeqg9wZWswWkAuYr4VKzhJT49wH2yGbzDA9v/E/hwhcS/ # people2.servers.bornhack.org:22 SSH-2.0-OpenSSH_7.5 FreeBSD-20170903 people2.servers.bornhack.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHDPircCflFUaCvjGYluQtsoc2WbdsU9+CYJ56KXa2IY [ansible@ansible2 ~]$
I can also choose to ask for just one key type with -t like so:
[ansible@ansible2 ~]$ ssh-keyscan -t ed25519 people2.servers.bornhack.org # people2.servers.bornhack.org:22 SSH-2.0-OpenSSH_7.5 FreeBSD-20170903 people2.servers.bornhack.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHDPircCflFUaCvjGYluQtsoc2WbdsU9+CYJ56KXa2IY [ansible@ansible2 ~]$
Since the line prefixed with # are printed to stderr it is simple enough to get just the key. But the format is not exactly what ssh-keygen -r -f <file> expects, so if you feed it as-is to ssh-keygen it throws a rather unhelpful error:
[ansible@ansible2 ~]$ ssh-keyscan -t ed25519 people2.servers.bornhack.org > keytest # people2.servers.bornhack.org:22 SSH-2.0-OpenSSH_7.5 FreeBSD-20170903 [ansible@ansible2 ~]$ ssh-keygen -r people2.servers.bornhack.org -f keytest Failed to read v2 public key from "keytest": No such file or directory. [ansible@ansible2 ~]$
Two issues prevent this from working. First of all ssh-keygen expects spaces and ssh-keyscan outputs tabs as seperator characters. Second ssh-keygen only wants the keytype and key, not the hostname. Putting this together it is possible to make a Bash oneliner to get a remote SSH host key and generate the appropriate SSHFP records:
[ansible@ansible2 ~]$ for algo in rsa dsa ecdsa ed25519; do echo "Getting $algo key..."; ssh-keygen -r people2.servers.bornhack.org -f <(ssh-keyscan -t $algo people2.servers.bornhack.org 2>/dev/null | cut -w -f 2- | tr " " " "); done Getting rsa key... people2.servers.bornhack.org IN SSHFP 1 1 2ebc778e8e444969c39a5dacba284c3d04412d2e people2.servers.bornhack.org IN SSHFP 1 2 a0eec08658e951ae76d169e560fd8b0871524ae71983d3b801983ab1162fbbc7 Getting dsa key... Failed to read v2 public key from "/tmp//sh-np.whEUx7": No such file or directory. Getting ecdsa key... people2.servers.bornhack.org IN SSHFP 3 1 a758382193ffc6a8bbd09411341a10d585e3050d people2.servers.bornhack.org IN SSHFP 3 2 8bd44964cb80cc26144020a58230645902b98ad171fec19522f070e69f174d02 Getting ed25519 key... people2.servers.bornhack.org IN SSHFP 4 1 e2fefe52725c55041c92bb0d4b3e27a721370746 people2.servers.bornhack.org IN SSHFP 4 2 ef90b45994021f0ef232e72e6263e4b1d22167f29316f4aab09eec2122cdfc49 [ansible@ansible2 ~]$
I used this as a reference while working with check_sshfp - it is always good to have a reference so you know you are getting correct results. The oneliner above could easily be converted to a Posix sh script without the <() bash-ishm if one was so inclined. The -w switch for cut is not present in Gnu cut as far as I can tell, replace as needed.
Note: The OpenSSH 7.5 on FreeBSD I am using in this example does not generate DSA keys which explains the error above. Checking the servers SSH public keys verifies this:
[tykling@people2 ~]$ ls -l /etc/ssh/ssh_host_*_key.pub -rw-r--r-- 1 root wheel 195 Feb 23 21:11 /etc/ssh/ssh_host_ecdsa_key.pub -rw-r--r-- 1 root wheel 115 Feb 23 21:11 /etc/ssh/ssh_host_ed25519_key.pub -rw-r--r-- 1 root wheel 415 Feb 23 21:11 /etc/ssh/ssh_host_rsa_key.pub [tykling@people2 ~]$
The reason for this is simply that the rc.d init script for sshd on FreeBSD does not enable DSA keys:
[tykling@people2 ~]$ grep -E "sshd_(rsa|dsa|ecdsa|ed25519)_enable" /etc/rc.d/sshd
: ${sshd_rsa_enable:="yes"}
: ${sshd_dsa_enable:="no"}
: ${sshd_ecdsa_enable:="yes"}
: ${sshd_ed25519_enable:="yes"}
[tykling@people2 ~]$
DSA keys were disabled in FreeBSD commit r294560 which was committed in January 2016.
Various tools for generating and checking SSHFP records exist, there is even one in Ports, and I also found one intended for monitoring use, which is what I need.
The former didn't really seem right for my usecase. Almost 500 lines of code, mostly stuff I would never use, and I saw no way of making it return a nonzero exit code in case of issues which is what I had in mind.
The latter is built for Nagios, but that doesn't really matter, Nagios plugins are simple and just exit 0 if OK, 1 if WARNING, 2 if CRITICAL. That seemed like a good starting point. It is written for Python3 but works with 2 as well. 112 lines of code and comments, and 25 of those are the LICENSE. It basically just worked - it takes a hostname as argument, runs ssh-keyscan using subprocess.Popen(), and checks if the corresponding SSHFP records could be found in DNS, and returns an exitcode and a line of output accordingly.
I forked check_sshfp because I wanted to add two features:
The forked repository is here. The changes I made should not affect it's ability to be used as a Nagios plugin, although I haven't tried - it's been years since I've used Nagios, and I can't say I miss it.
A run of my forked check_sshfp against a host with no SSHFP records looks like this, and returns exitcode 1:
[ansible@ansible2 ~/check_sshfp]$ python check_sshfp people2.servers.bornhack.org SSHFP WARNING: No record for algorithm ssh-ed25519 offered by server. Add this record: people2.servers.bornhack.org IN SSHFP 4 2 ef90b45994021f0ef232e72e6263e4b1d22167f29316f4aab09eec2122cdfc49 SSHFP WARNING: No record for algorithm ssh-rsa offered by server. Add this record: people2.servers.bornhack.org IN SSHFP 1 2 a0eec08658e951ae76d169e560fd8b0871524ae71983d3b801983ab1162fbbc7 SSHFP WARNING: No record for algorithm ecdsa-sha2-nistp256 offered by server. Add this record: people2.servers.bornhack.org IN SSHFP 3 2 8bd44964cb80cc26144020a58230645902b98ad171fec19522f070e69f174d02 [ansible@ansible2 ~/check_sshfp]$ echo $? 1 [ansible@ansible2 ~/check_sshfp]$
After adding the records it looks like this:
[ansible@ansible2 ~/check_sshfp]$ python check_sshfp people2.servers.bornhack.org SSHFP OK: 3 records for 3 algorithms [ansible@ansible2 ~/check_sshfp]$ echo $? 0 [ansible@ansible2 ~/check_sshfp]$
If I delete one of the records so two valid remain but one is missing:
[ansible@ansible2 ~/check_sshfp]$ python check_sshfp people2.servers.bornhack.org SSHFP WARNING: No record for algorithm ssh-rsa offered by server. Add this record: people2.servers.bornhack.org IN SSHFP 1 2 a0eec08658e951ae76d169e560fd8b0871524ae71983d3b801983ab1162fbbc7 [ansible@ansible2 ~/check_sshfp]$ echo $? 1 [ansible@ansible2 ~/check_sshfp]$
If I add the missing key but make a typo so the hash is wrong:
[ansible@ansible2 ~/check_sshfp]$ python check_sshfp people2.servers.bornhack.org SSHFP CRITICAL: Fingerprint mismatch: SSHFP a0eec08658e951ae76d169e560fd8b0871524ae71983d3b801983ab1162fbbc8, observed a0eec08658e951ae76d169e560fd8b0871524ae71983d3b801983ab1162fbbc7. Delete the wrong record and add this instead: people2.servers.bornhack.org IN SSHFP 1 2 a0eec08658e951ae76d169e560fd8b0871524ae71983d3b801983ab1162fbbc7 [ansible@ansible2 ~/check_sshfp]$ echo $? 2 [ansible@ansible2 ~/check_sshfp]$
This is perfect. In case I forget to add or update an SSHFP record I will get an email telling me exactly what I need to do.
The only remaining job is to call my forked check_sshfp once per day. I renamed my checkpkg.sh script in my ansible-roles repo to check_stuff.sh since it now checks more than just pkg audit. I then added these lines to the script:
# get the sshfp output
output=$(/usr/local/bin/python /usr/home/ansible/check_sshfp/check_sshfp $host)
if [ $? -ne 0 ]; then
# sshfp issues found, send mail
echo "$output" | /usr/bin/mail -s "$(basename "$1") - $host: SSHFP issues found!" "$2"
fi
That's it! Happy SSHFPing!