download.docker.com SSL certificate is NOT trusted (self-signed).

I recently tried to update my docker engine on my WSL Ubuntu 22.04 but it failed due to SSL certificate error. So I looked it up a little bit and found out that the docker PPA server (download.docker.com) SSL Certificate was self-signed / having unknown CA issuer.

As a result it is not trusted by curl or apt tool, and most modern software will just reject the connection to a such site.

This is the dig output from download.docker.com on my machine:

$ dig download.docker.com

; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> download.docker.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32776
;; flags: qr rd ad; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;download.docker.com.           IN      A

;; ANSWER SECTION:
download.docker.com.    0       IN      A       146.112.49.147
download.docker.com.    0       IN      A       146.112.49.38
download.docker.com.    0       IN      A       146.112.49.252
download.docker.com.    0       IN      A       146.112.238.215
download.docker.com.    0       IN      A       146.112.49.228
download.docker.com.    0       IN      A       146.112.49.135
download.docker.com.    0       IN      A       146.112.49.203
download.docker.com.    0       IN      A       146.112.49.29

;; Query time: 10 msec
;; SERVER: 172.20.32.1#53(172.20.32.1) (UDP)
;; WHEN: Wed Apr 10 18:02:02 EDT 2024
;; MSG SIZE  rcvd: 184

I looked up these IP addresses, the CDN cluster is Cisco OpenDNS LLC in Miami, Florida, which is very close to where I live. I am not sure if this is a CDN issue or docker PPA server issue, but please fix it as I am not the only person that is impacted by this.

I tested these IP addresses with SSL Security Test | ImmuniWeb and found the above issues. If you wish to reproduce the issue you can do the same thing or use openssl s_client -connect <IP>:443 to see the ssl error (unable to get local issuer certificate).

$ openssl s_client -CApath /etc/ssl/certs/ -connect 146.112.49.203:443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=2 C = US, ST = California, L = San Francisco, O = Cisco, CN = Cisco Umbrella Primary SubCA
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 O = Cisco, CN = Cisco Umbrella Secondary SubCA mia-SG
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cisco Systems, Inc.", CN = *.opendns.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "Cisco Systems, Inc.", CN = *.opendns.com
   i:O = Cisco, CN = Cisco Umbrella Secondary SubCA mia-SG
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr  8 11:53:06 2024 GMT; NotAfter: Apr 13 11:53:06 2024 GMT
 1 s:O = Cisco, CN = Cisco Umbrella Secondary SubCA mia-SG
   i:C = US, ST = California, L = San Francisco, O = Cisco, CN = Cisco Umbrella Primary SubCA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr  9 02:00:48 2024 GMT; NotAfter: Apr 20 02:00:48 2024 GMT
 2 s:C = US, ST = California, L = San Francisco, O = Cisco, CN = Cisco Umbrella Primary SubCA
   i:O = Cisco, CN = Cisco Umbrella Root CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 21 19:53:18 2019 GMT; NotAfter: May 21 19:53:18 2024 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDSTCCAjGgAwIBAgIDeXK3MA0GCSqGSIb3DQEBCwUAMEAxDjAMBgNVBAoMBUNp
c2NvMS4wLAYDVQQDDCVDaXNjbyBVbWJyZWxsYSBTZWNvbmRhcnkgU3ViQ0EgbWlh
LVNHMB4XDTI0MDQwODExNTMwNloXDTI0MDQxMzExNTMwNlowcDELMAkGA1UEBhMC
VVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28x
HDAaBgNVBAoME0Npc2NvIFN5c3RlbXMsIEluYy4xFjAUBgNVBAMMDSoub3BlbmRu
cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDnIhgpZmkZuJUh
WznbIkZigfxIWxwRs5txX4jt7rQcrX5CbdN1YspUYf+kEhJF095JPsq5ih11I3Ix
Nf4qPP4DcrX8hbScwiSnkIqn5LMM0IpaKlTWxzH7P7wlIdqKuzizRxOwwPDuVKAs
Cyz0THXbydDMc8Mp/MOEEV+BRdXUA/hS4GohJM9g0kxXd85UxIc8kzw0Tgk6PsZ9
oSPVxmR1chC3nGvA3p84LFTqzjOPJTHVWLmpnNU4fDt3knEdOX6Nj8Fr0t7+0h0C
ZWq/FLQCCQTYSRysPQbiuaXiqG4yaZ5YxCb5JdgMHJZaWrXxoXI40r7YA6PRIQbv
36naRXHLAgMBAAGjHDAaMBgGA1UdEQQRMA+CDSoub3BlbmRucy5jb20wDQYJKoZI
hvcNAQELBQADggEBAJCbZbYdVzRjqSh/e0aOKyrqcZFIwi/BfNgE9zXu4V7ihg08
Xc8yJwKUg7xdlOs3269xQTafIVdfTxU3bigu490f96cjgAhYdGrzFe+8beUduUJ7
0qVzm/jZC4iZISvwV2gPHYMq4HkvrhtlMPNZ+gSPdS6crVJQaVPdVIr9BQoF2usL
nxt0aY6Ph/VvSLho0wDWynma6xn5oXrfdGPBdeCc2NRm+XBfVM5R/ZTEeXluEWDV
01AvTm9JJpn/q9ncPt7fPhR9B9zJF61iTpXwk1og42PM3u5Aqvrr8FI9lBpKmy30
CV2ZMnpoM9j5hojY3BDXPe0mOV3kEnLBL+LEQIE=
-----END CERTIFICATE-----
subject=C = US, ST = California, L = San Francisco, O = "Cisco Systems, Inc.", CN = *.opendns.com
issuer=O = Cisco, CN = Cisco Umbrella Secondary SubCA mia-SG
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3708 bytes and written 373 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: AEF1290B041707BF4C297AF80A101C37D698276ED2F15182B77B5BEEC441B3D0
    Session-ID-ctx:
    Resumption PSK: 777024A130333F1A2450CFA2D2C815E535BFC4FBEC1A85CAAB56C00E52C96A08D78151C0C43C268DA08468F18EB48990
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 2d 38 86 06 73 ee 01 87-8f ef 02 00 9b ea 61 ba   -8..s.........a.
    0010 - 8a 0d 7b ec 9f d2 41 bb-7f 0b 0a 18 e6 a5 b7 ba   ..{...A.........
    0020 - da 3b 0b da b9 c1 d0 3f-18 7e 07 ae 5f c8 f5 15   .;.....?.~.._...
    0030 - 73 7b 48 de 29 86 c2 ed-a8 10 aa bf 78 7c b7 72   s{H.).......x|.r
    0040 - c8 6d ef 96 7a 0e 28 62-b1 f8 4a 0a 85 8d 76 96   .m..z.(b..J...v.
    0050 - 5b 05 57 2c 7c de 49 95-98 ab a6 b8 bd d2 98 01   [.W,|.I.........
    0060 - 9e 4a 3d af 1e 1f f4 a0-70 6c 13 17 7e 4d 24 bf   .J=.....pl..~M$.
    0070 - 08 0a dc 94 06 01 77 97-2b 1e 6f 4f d4 20 2e 5a   ......w.+.oO. .Z
    0080 - 77 d0 ed 74 be 21 bc e2-d6 58 96 95 59 43 78 7a   w..t.!...X..YCxz
    0090 - 48 36 1d dc 4e 04 cc b1-2d d1 84 ca 1a 68 8e 84   H6..N...-....h..
    00a0 - c8 46 3d ba b2 40 72 6f-f7 f7 f5 78 3f 88 7b cb   .F=..@ro...x?.{.
    00b0 - f2 ce 90 2e 62 e8 27 c5-9f 33 c2 b0 d4 4f 35 0a   ....b.'..3...O5.

    Start Time: 1712785945
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: E70BA267F24FCAE6A393A09E36A6DA9821EE208D7FF0982CC8D4FD79D796ED5F
    Session-ID-ctx:
    Resumption PSK: DDADDA01EE6E2736D46E5A09263C8671DCB060249339D0A9F1A1F52A0E853DCA7B97D34C84713B1043CC6420172EC0F4
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 2d 38 86 06 73 ee 01 87-8f ef 02 00 9b ea 61 ba   -8..s.........a.
    0010 - d2 39 fe 6a 90 ac 26 44-8b f4 37 5a 64 ee 6f 51   .9.j..&D..7Zd.oQ
    0020 - 77 17 a2 8f 47 ed 29 88-81 4a 35 45 88 34 f4 f5   w...G.)..J5E.4..
    0030 - 07 91 57 fc 37 95 72 3a-53 f2 e2 ab 0b 56 a7 94   ..W.7.r:S....V..
    0040 - 36 41 2c 70 4d 32 6a 86-7b 9d c0 f8 57 0e 14 d0   6A,pM2j.{...W...
    0050 - c3 15 d3 2b d3 b3 1b 53-97 ae d2 91 f6 c8 71 47   ...+...S......qG
    0060 - bb 45 d5 55 ee 72 3a 4d-00 2c 34 8f a3 36 bd a9   .E.U.r:M.,4..6..
    0070 - 68 cf 2c 66 a9 e1 21 5c-1d e9 b5 d1 e5 d6 45 66   h.,f..!\......Ef
    0080 - a2 e4 69 97 c1 96 47 3a-5f e9 a3 83 16 e8 b2 06   ..i...G:_.......
    0090 - e9 4e 56 91 e4 2f fd 8e-11 01 ad 63 49 30 c3 0c   .NV../.....cI0..
    00a0 - e9 3f 55 fd 5a af ec 3c-72 c1 56 61 69 03 34 6a   .?U.Z..<r.Vai.4j
    00b0 - 1e a6 d3 ac d4 72 82 c3-be 27 c8 c0 f6 0d e6 b8   .....r...'......

    Start Time: 1712785945
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

Thanks

Are you trying this at home or inside a corporate network? Maybe there is some proxy involved that decrypts on the perimeter and re-crypts with a corporate cert inside the local network.

I first found it in the corporate network and thought the same thing. Then I updated the CA certs from the public repository and still saw the same error. I ran immuniweb and openssl on multiple devices to confirm the issue is because of the docker download server uses self signed SSL certificates.

Like I said, you can use openssl s_client -connect command to test against the IPs listed in my post and will see the same ( unable to get local issuer certificate) error, which is due to server sent a self signed cert and openssl cannot find a local CA cert to verifiy it.

I am not sure if this is DNS pollution problem because I later on checked on the OpenDNS and didn’t find these IP records in the OpenDNS database.

I will wait for Docker DevOps team to confirm if the above IPs actually belong to docker, if not I think we have a DNS poisoning attack right now. :upside_down_face:

@fredriceliu Hey, I can confirm from Docker Security side that this is not our IPs or certificates. Our certs are signed by AWS.

╰─ dig download.docker.com                          

; <<>> DiG 9.10.6 <<>> download.docker.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2648
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;download.docker.com.		IN	A

;; ANSWER SECTION:
download.docker.com.	243	IN	CNAME	d2h67oheeuigaw.cloudfront.net.
d2h67oheeuigaw.cloudfront.net. 60 IN	A	3.161.213.71
d2h67oheeuigaw.cloudfront.net. 60 IN	A	3.161.213.40
d2h67oheeuigaw.cloudfront.net. 60 IN	A	3.161.213.83
d2h67oheeuigaw.cloudfront.net. 60 IN	A	3.161.213.29

;; Query time: 59 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Apr 11 11:26:24 EDT 2024
;; MSG SIZE  rcvd: 155
Connecting to 3.161.213.40
CONNECTED(00000005)
depth=2 C=US, O=Amazon, CN=Amazon Root CA 1
verify return:1
depth=1 C=US, O=Amazon, CN=Amazon RSA 2048 M01
verify return:1
depth=0 CN=*.docker.com
verify return:1
---
Certificate chain
 0 s:CN=*.docker.com
   i:C=US, O=Amazon, CN=Amazon RSA 2048 M01
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct  2 00:00:00 2023 GMT; NotAfter: Oct 31 23:59:59 2024 GMT
 1 s:C=US, O=Amazon, CN=Amazon RSA 2048 M01
   i:C=US, O=Amazon, CN=Amazon Root CA 1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 23 22:21:28 2022 GMT; NotAfter: Aug 23 22:21:28 2030 GMT
 2 s:C=US, O=Amazon, CN=Amazon Root CA 1
   i:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
 3 s:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
   i:C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGXzCCBUegAwIBAgIQCE156iNdbf8JCmdLBJvaIjANBgkqhkiG9w0BAQsFADA8
MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRwwGgYDVQQDExNBbWF6b24g
UlNBIDIwNDggTTAxMB4XDTIzMTAwMjAwMDAwMFoXDTI0MTAzMTIzNTk1OVowFzEV
MBMGA1UEAwwMKi5kb2NrZXIuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAmCP2/2lIdGLqjZoiSX7EGX/TNl5h49u4/iecKf4xRqxLzt9aUJptiROX
fYV5gUMfmbnKV1qjiR6DOOKZ7C6/gMIIKVqDIkbMCxjs4b7hH61XcXuyRBZHphyn
5CiQ5fCsayJfyjnOJUrn0EwyUyzete2eSF99L7SHk4rVfG1XVyPxS5JVK1Hzd/lB
qFiXAzrDrHHiFs+L3ESpTI204knV7y++e5bDcdfYxLx59TpHEQcqzRp8zVOKayRO
jHj5maczdXN11PCpmXknlIcuXQVCl04gbQepAtlJTIXh1UugCEvjGbv49+cD/iVp
W9zxIN9vaYu4T4cfGQO6ef2J8HYcUwIDAQABo4IDgDCCA3wwHwYDVR0jBBgwFoAU
gbgOY4qJEhjl+js7UJWf5uWQE4UwHQYDVR0OBBYEFGn1KBnm0DAYaxWABU0NvBZV
Ua3EMIGuBgNVHREEgaYwgaOCDCouZG9ja2VyLmNvbYIKZG9ja2VyLmNvbYIQKi5o
dWIuZG9ja2VyLmNvbYITKi5kb2NrZXJwcm9qZWN0Lm9yZ4IJZG9ja2VyLmlvggsq
LmRvY2tlci5pb4ISKi5jbG91ZC5kb2NrZXIuY29tghoqLm1hc3Rlci5kb2NrZXJw
cm9qZWN0Lm9yZ4IYKi5jbG91ZC1zdGFnZS5kb2NrZXIuY29tMBMGA1UdIAQMMAow
CAYGZ4EMAQIBMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
KwYBBQUHAwIwOwYDVR0fBDQwMjAwoC6gLIYqaHR0cDovL2NybC5yMm0wMS5hbWF6
b250cnVzdC5jb20vcjJtMDEuY3JsMHUGCCsGAQUFBwEBBGkwZzAtBggrBgEFBQcw
AYYhaHR0cDovL29jc3AucjJtMDEuYW1hem9udHJ1c3QuY29tMDYGCCsGAQUFBzAC
hipodHRwOi8vY3J0LnIybTAxLmFtYXpvbnRydXN0LmNvbS9yMm0wMS5jZXIwDAYD
VR0TAQH/BAIwADCCAYEGCisGAQQB1nkCBAIEggFxBIIBbQFrAHcA7s3QZNXbGs7F
XLedtM0TojKHRny87N7DUUhZRnEftZsAAAGK7sUNmAAABAMASDBGAiEAv3UFBgu+
AkX7PXiOEZ6DBar4qf9MDo3JmAg+sMcNUawCIQCvdtis/G4/AFAOr9cYPKm265sU
Mq80rAXc5ulITSfOLgB3AEiw42vapkc0D+VqAvqdMOscUgHLVt0sgdm7v6s52IRz
AAABiu7FDccAAAQDAEgwRgIhAPNIzKgm7X12iZI1Tow2srG8guulawJV6JWuNMig
dcWzAiEAugLuchDEQwSr16Svj2E8l+pJ4+D5BZQNh0gO/eyRahMAdwDatr9rP7W2
Ip+bwrtca+hwkXFsu1GEhTS9pD0wSNf7qwAAAYruxQ2dAAAEAwBIMEYCIQDkKE6U
wOUzy9chy/WtaXG5fNK3gVEYQ9neqiNC/795wAIhANsU0WCJVGQFEVA9MMM1wRyk
4APqrbYHZYdJLBrcub8MMA0GCSqGSIb3DQEBCwUAA4IBAQAEpkfyiw68ZlcgDJo/
pryrEtJ5Z4pqW1yk0GEP2WJ9PXyTZicRi7U+gaahpovy89SYnMBeH3zhYT1UKp3G
PzCQNsnPD606F4Q3vlJfz7+PlycvSYEbxLbxvb53SyTw5TWOIfeSQSQrI0/hCRf2
w2CISXtNOilRMP36buT8oz1TZirHJSpGZRe0NUzhuw2cmzeRUILJ+QvLAzEK49Vn
zCy5azGAv6EzAcXY7cIMByacfNtTR2nhA/r2dktcNTpMpLdQ43NJGemctXaDpXMb
hxQvMIK3itct8JQqHtSKLIsJphzgMTrDCXhV1o7zv7zm5SpfvUf4G4GAHeCKyKD0
fTe8
-----END CERTIFICATE-----
subject=CN=*.docker.com
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5635 bytes and written 391 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 01A807A317712435C3837FF59F47A19A3E724580E12FF96815B513105FC4FB4D
    Session-ID-ctx: 
    Resumption PSK: 5E0E9C5F2B01CF5167633603F435211E7C8948A03C7D72251AEFC1D81D6A1CBA
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 31 37 31 32 38 33 33 31-30 30 30 30 30 00 00 00   1712833100000...
    0010 - c7 6b a5 2e 56 23 80 37-24 18 56 c0 5c 4d 71 3d   .k..V#.7$.V.\Mq=
    0020 - f7 62 fd 30 a1 e9 3b 50-f1 95 2b fc 7c 45 85 ba   .b.0..;P..+.|E..
    0030 - e9 01 d5 1a 40 76 10 d5-6d 86 68 48 69 2d e8 61   ....@v..m.hHi-.a
    0040 - 87 5f 29 9b 7c 1e eb ca-8b 65 95 5f b5 bb 0f 45   ._).|....e._...E
    0050 - 37 f7 e7 6d 2b 04 9a df-e8 b9 21 5c 65 17 91 23   7..m+.....!\e..#
    0060 - 7c 22 20 33 dc fd a9 1c-11                        |" 3.....

    Start Time: 1712849152
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---

if not I think we have a DNS poisoning attack right now

Yes this is possible. I’d check into your logs on DNS server at 172.20.32.1

I will do that, thank you very much.

It appears your company enforces TLS inspection (also known as TLS interception) in the corporate network. You should ask your company’s help desk about it. They should be able to provide the certificate of the signing CA used by the TLS inspection appliance. You will need to add this certificate to the trust store of the WSL distribution you are using.