by Tykling
17. jan 2019 14:17 UTC
Today I was testing out https support on a partner API. They've been using a selfsigned certificate during development, and since they will move to production soon the certificate has been replaced by a CA signed certificate from GoDaddy.
I was still getting certificate errors though, and I suspected that they'd forgotten to include the intermediate certificate from the issuer. I wanted to check locally which intermediate to use so I could send an easily actionable error report to the partner.
I've changed the partners domain to example.com to protect the innocent.
First things first, I use openssl s_client -showcerts for connecting and showing the certificate:
[tsr@mrx2 ~]$ echo | openssl s_client -showcerts -connect chayntest.example.com:7000
CONNECTED(00000003)
depth=0 OU = Domain Control Validated, CN = *.example.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, CN = *.example.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/OU=Domain Control Validated/CN=*.example.com
i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
-----BEGIN CERTIFICATE-----
MIIFJDCCBAygAwIBAgIJAKPtTjF+G3fhMA0GCSqGSIb3DQEBCwUAMIG0MQswCQYD
VQQGEwJVUzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMKU2NvdHRzZGFsZTEa
MBgGA1UEChMRR29EYWRkeS5jb20sIEluYy4xLTArBgNVBAsTJGh0dHA6Ly9jZXJ0
cy5nb2RhZGR5LmNvbS9yZXBvc2l0b3J5LzEzMDEGA1UEAxMqR28gRGFkZHkgU2Vj
dXJlIENlcnRpZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTE3MDMxNTEyMDkwMFoX
DTE5MDMxNDA4MDIzOFowODEhMB8GA1UECxMYRG9tYWluIENvbnRyb2wgVmFsaWRh
dGVkMRMwEQYDVQQDDAoqLndhb28ub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEAvt4LPzD5G63y/QI8Iue9QFqmgQeBLHVd5mR7Bsj/SXNuVzLxFTQA
x0ZSWa6Ux0y734r1zoHz4f3sM8OVVwdQgq9AbvL8Sj9wozNFUIQSCCO6vxRW74W+
7ff54pwxhfxiy14cw3koAqmW6uqcrbJZFEPvFWyoq11TyXB6aIfv+/dIMs03lrdG
Iaf5aPbZcBnB5iH+6CR13ZBIeyegXg93pWnevETBus9GJTW15Q/J2IBn1KdWVjg1
6H1nWmuyTMhR6NiDPVfP7l9uJGc6HxfdH+XBLkbEnz+BZT+CnzVmW/CF68dFiqNK
6z4Zv54hbZolyjwiQkV+clvBlKMjeG5CAwIDAQABo4IBsjCCAa4wDAYDVR0TAQH/
BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDgYDVR0PAQH/BAQD
AgWgMDcGA1UdHwQwMC4wLKAqoCiGJmh0dHA6Ly9jcmwuZ29kYWRkeS5jb20vZ2Rp
ZzJzMS00NDAuY3JsMF0GA1UdIARWMFQwSAYLYIZIAYb9bQEHFwEwOTA3BggrBgEF
BQcCARYraHR0cDovL2NlcnRpZmljYXRlcy5nb2RhZGR5LmNvbS9yZXBvc2l0b3J5
LzAIBgZngQwBAgEwdgYIKwYBBQUHAQEEajBoMCQGCCsGAQUFBzABhhhodHRwOi8v
b2NzcC5nb2RhZGR5LmNvbS8wQAYIKwYBBQUHMAKGNGh0dHA6Ly9jZXJ0aWZpY2F0
ZXMuZ29kYWRkeS5jb20vcmVwb3NpdG9yeS9nZGlnMi5jcnQwHwYDVR0jBBgwFoAU
QMK9J47MNIMwojPX+2yz8LQsgM4wHwYDVR0RBBgwFoIKKi53YW9vLm9yZ4IId2Fv
by5vcmcwHQYDVR0OBBYEFEoeWTF5dogGGN9DOCT/+L4avvEfMA0GCSqGSIb3DQEB
CwUAA4IBAQBFF/N1sNJjQbLloXKRtmwRlwSZ+zJv6L0hKVYjgXSHSqS6faSSxPZx
M1O0ERWkyT4iyGX93v1w/wqJFmHqU1xeu6dbggaHc3l5lN3RIidM3+sDbSbZd5Jr
VNrfVsJgdtKFLpBHRTDbhuDzKb8MdAc5wbktCYNpoFn5mDHKY7mvOScr96+K8Vbp
kzVJBj5CYT2N33KGop83gIzw4JGNbaNgsajPAdY/tXZUtvT3BwUpt+slh8RG0NsH
2XJpzZteJZYu+k8LZtHIeXO29/5Q+/sUoYOVfphUmJej4rai+ChaCSAS2pzsVh+r
rjA6T74+9zDOdrWhp6b01nFxKM4k4roK
-----END CERTIFICATE-----
---
Server certificate
subject=/OU=Domain Control Validated/CN=*.example.com
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2011 bytes and written 433 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 08C7F3C3174C6E5338B01111A552336920C16237E1C4337AD6421A4C6904BC9F
Session-ID-ctx:
Master-Key: E7FFBF6E1F932BE1386EA315370D68AB3E2C6CF4FB75CD328123E1627BE0BB4A462E3CCC17D5FA6465D54FDF11AC3F80
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 6e 3d 11 dc 55 42 af 04-51 7e 45 0b 99 8a ab 4d n=..UB..Q~E....M
0010 - bb 08 af a4 9c 22 0e b3-ab ad de 93 8b 2b d0 5b .....".......+.[
0020 - 97 12 9c e9 95 f0 aa 42-84 37 00 02 af a0 86 01 .......B.7......
0030 - 9c f2 d3 8e 76 db 81 77-50 c3 eb 3e 2e 5f 55 fc ....v..wP..>._U.
0040 - dc b4 c2 6f d1 22 30 96-f4 bc 00 30 2f bb 24 7c ...o."0....0/.$|
0050 - 11 17 0b a5 51 af 52 d9-98 d9 23 0b 32 08 36 18 ....Q.R...#.2.6.
0060 - 55 80 e5 b7 2f 14 53 61-ce b4 96 91 6e 30 1e b8 U.../.Sa....n0..
0070 - 87 4e 23 dc 9e 0b 65 40-8a d1 f8 fd 91 df 86 43 .N#...e@.......C
0080 - 71 f6 0d 8a 4c d9 b2 02-f0 d3 d7 1a 60 ee fa 2d q...L.......`..-
0090 - 02 c6 3a 71 99 3c e9 20-50 fa 74 98 50 bb 25 3f ..:q.<. P.t.P.%?
00a0 - 4d 2e df ea 01 3a 78 ba-f1 a5 fc 5b ba 93 5e 45 M....:x....[..^E
00b0 - 08 74 91 56 28 1b 5d b1-43 b5 b4 fa 97 c5 74 e5 .t.V(.].C.....t.
Start Time: 1547734089
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
DONE
[tsr@mrx2 ~]$
The important part here is verify error:num=20:unable to get local issuer certificate which tells me that OpenSSL is unable to verify the certificate using my operating systems list of trusted CAs. This might be because of a missing intermediate certificate.
To work out which intermediate needs to be installed I first create an empty folder to work in, and then copy the server certificate into a file - that is, from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE----- and everything between:
[tsr@mrx2 ~]$ mkdir cert [tsr@mrx2 ~]$ cd cert/ [tsr@mrx2 ~/cert]$ cat <wildcard.example.com.crt > -----BEGIN CERTIFICATE----- > MIIFJDCCBAygAwIBAgIJAKPtTjF+G3fhMA0GCSqGSIb3DQEBCwUAMIG0MQswCQYD > VQQGEwJVUzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMKU2NvdHRzZGFsZTEa > MBgGA1UEChMRR29EYWRkeS5jb20sIEluYy4xLTArBgNVBAsTJGh0dHA6Ly9jZXJ0 > cy5nb2RhZGR5LmNvbS9yZXBvc2l0b3J5LzEzMDEGA1UEAxMqR28gRGFkZHkgU2Vj > dXJlIENlcnRpZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTE3MDMxNTEyMDkwMFoX > DTE5MDMxNDA4MDIzOFowODEhMB8GA1UECxMYRG9tYWluIENvbnRyb2wgVmFsaWRh > dGVkMRMwEQYDVQQDDAoqLndhb28ub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A > MIIBCgKCAQEAvt4LPzD5G63y/QI8Iue9QFqmgQeBLHVd5mR7Bsj/SXNuVzLxFTQA > x0ZSWa6Ux0y734r1zoHz4f3sM8OVVwdQgq9AbvL8Sj9wozNFUIQSCCO6vxRW74W+ > 7ff54pwxhfxiy14cw3koAqmW6uqcrbJZFEPvFWyoq11TyXB6aIfv+/dIMs03lrdG > Iaf5aPbZcBnB5iH+6CR13ZBIeyegXg93pWnevETBus9GJTW15Q/J2IBn1KdWVjg1 > 6H1nWmuyTMhR6NiDPVfP7l9uJGc6HxfdH+XBLkbEnz+BZT+CnzVmW/CF68dFiqNK > 6z4Zv54hbZolyjwiQkV+clvBlKMjeG5CAwIDAQABo4IBsjCCAa4wDAYDVR0TAQH/ > BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDgYDVR0PAQH/BAQD > AgWgMDcGA1UdHwQwMC4wLKAqoCiGJmh0dHA6Ly9jcmwuZ29kYWRkeS5jb20vZ2Rp > ZzJzMS00NDAuY3JsMF0GA1UdIARWMFQwSAYLYIZIAYb9bQEHFwEwOTA3BggrBgEF > BQcCARYraHR0cDovL2NlcnRpZmljYXRlcy5nb2RhZGR5LmNvbS9yZXBvc2l0b3J5 > LzAIBgZngQwBAgEwdgYIKwYBBQUHAQEEajBoMCQGCCsGAQUFBzABhhhodHRwOi8v > b2NzcC5nb2RhZGR5LmNvbS8wQAYIKwYBBQUHMAKGNGh0dHA6Ly9jZXJ0aWZpY2F0 > ZXMuZ29kYWRkeS5jb20vcmVwb3NpdG9yeS9nZGlnMi5jcnQwHwYDVR0jBBgwFoAU > QMK9J47MNIMwojPX+2yz8LQsgM4wHwYDVR0RBBgwFoIKKi53YW9vLm9yZ4IId2Fv > by5vcmcwHQYDVR0OBBYEFEoeWTF5dogGGN9DOCT/+L4avvEfMA0GCSqGSIb3DQEB > CwUAA4IBAQBFF/N1sNJjQbLloXKRtmwRlwSZ+zJv6L0hKVYjgXSHSqS6faSSxPZx > M1O0ERWkyT4iyGX93v1w/wqJFmHqU1xeu6dbggaHc3l5lN3RIidM3+sDbSbZd5Jr > VNrfVsJgdtKFLpBHRTDbhuDzKb8MdAc5wbktCYNpoFn5mDHKY7mvOScr96+K8Vbp > kzVJBj5CYT2N33KGop83gIzw4JGNbaNgsajPAdY/tXZUtvT3BwUpt+slh8RG0NsH > 2XJpzZteJZYu+k8LZtHIeXO29/5Q+/sUoYOVfphUmJej4rai+ChaCSAS2pzsVh+r > rjA6T74+9zDOdrWhp6b01nFxKM4k4roK > -----END CERTIFICATE----- > EOF [tsr@mrx2 ~/cert]$
Now, the issuer of this certificate is also mentioned in the OpenSSL output above, including a URL to their repository:
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
I went to http://certs.godaddy.com/repository/ and downloaded the certificate labeled GoDaddy Class 2 Certification Authority Root Certificate - G2 with the filename gdroot-g2.crt and put that in the same folder.
[tsr@mrx2 ~/cert]$ fetch https://certs.godaddy.com/repository/gdroot-g2.crt gdroot-g2.crt 1367 B 16 MBps 00s [tsr@mrx2 ~/cert]$ ls -l total 9 -rw-r--r-- 1 tsr tsr 1367 Nov 1 23:11 gdroot-g2.crt -rw-r--r-- 1 tsr tsr 1842 Jan 17 14:15 wildcard.example.com.crtBefore OpenSSL can use my new CA dir I need to run
c_rehash:
[tsr@mrx2 ~/cert]$ ls -l total 9 -rw-r--r-- 1 tsr tsr 1367 Nov 1 23:11 gdroot-g2.crt -rw-r--r-- 1 tsr tsr 1842 Jan 17 14:15 wildcard.example.com.crt [tsr@mrx2 ~/cert]$ c_rehash . Doing . [tsr@mrx2 ~/cert]$ ls -l total 10 lrwxr-xr-x 1 tsr tsr 21 Jan 17 14:44 376492c8.0 -> wildcard.example.com.crt lrwxr-xr-x 1 tsr tsr 13 Jan 17 14:44 cbf06781.0 -> gdroot-g2.crt -rw-r--r-- 1 tsr tsr 1367 Nov 1 23:11 gdroot-g2.crt -rw-r--r-- 1 tsr tsr 1842 Jan 17 14:15 wildcard.example.com.crt [tsr@mrx2 ~/cert]$A quick test verifies that the test setup works. I still haven't added the intermediate cert so I would expect the same result as with my OS CAdir. I add
-CApath . to the OpenSSL command to make it use the new one:
[tsr@mrx2 ~/cert]$ echo | openssl s_client -showcerts -CApath . -connect chayntest.example.com:7000 > /dev/null depth=0 OU = Domain Control Validated, CN = *.example.com verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 OU = Domain Control Validated, CN = *.example.com verify error:num=21:unable to verify the first certificate verify return:1 DONE [tsr@mrx2 ~/cert]$
Still no dice, as expected.
Lets see what happens when we add the intermediate into the mix. I go back to the GoDaddy cert page and downloaded the first intermediate I could find in the G2 section, labeled GoDaddy Secure Server Certificate (Intermediate Certificate) - G2 with the filename gdig2.crt.pem. I then run c_rehash again to make OpenSSL recognise the new certificate:
[tsr@mrx2 ~/cert]$ fetch https://certs.godaddy.com/repository/gdig2.crt.pem gdig2.crt.pem 1728 B 42 kBps 00s [tsr@mrx2 ~/cert]$ c_rehash . Doing . [tsr@mrx2 ~/cert]$ echo | openssl s_client -showcerts -CApath . -connect chayntest.example.com:7000 > /dev/null depth=2 C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2 verify return:1 depth=1 C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2 verify return:1 depth=0 OU = Domain Control Validated, CN = *.example.com verify return:1 DONE [tsr@mrx2 ~/cert]$
Much better. I have confirmed that gdig2.crt.pem is the missing intermediate for the partners server and I sent them an email (which was handled very quickly).
I could have just told them "still doesn't work", but in my experience going the extra mile for a partner or customer can pay off in unexpected ways sometimes.