Docker container has no connectivity during docker build

I’m pulling my hair out about this one. I am trying to use docker on an EC2 instance and during docker build my intermediate containers do not appear to have network connectivity and therefore cannot install required things and fail.

Info

  • Docker version: 17.06.0-ce, build 02c1d87
  • Hostoperating system: Ubuntu 16.04
  • Error:
 ---> Running in 772fcd53ad5c
Err:1 http://archive.ubuntu.com/ubuntu xenial InRelease
  Could not connect to archive.ubuntu.com:80 (91.189.88.149). - connect (111: Connection refused) [IP: 91.189.88.149 80]
Err:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
  Could not connect to security.ubuntu.com:80 (91.189.88.161). - connect (111: Connection refused) [IP: 91.189.88.161 80]
Err:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
  Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.149 80]
Err:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
  Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.149 80]
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease  Could not connect to archive.ubuntu.com:80 (91.189.88.149). - connect (111: Connection refused) [IP: 91.189.88.149 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease  Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.149 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-backports/InRelease  Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.149 80]
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease  Could not connect to security.ubuntu.com:80 (91.189.88.161). - connect (111: Connection refused) [IP: 91.189.88.161 80]
W: Some index files failed to download. They have been ignored, or old ones used instead. 

Tried

  • Checked to see that archive.ubuntu.com was up - it was

  • Checked that the host can access achive.ubuntu.com and can apt-get update - it can

  • Checked that docker postrouting is in iptables - it does appear to be

  • Checked that net.ipv4.ip_forward = 1, it is

  • Checked that there are not any conflicting iptables rules - I purged all other iptables rules, restarted docker - still no access

  • Checked that there is no proxy involved - not so sure about this one, but is the running in EC2 behind a vanilla VPC

  • Purged everything and ran with --no-cache - still fails

  • Uncommented DNS in /etc/default/docker - no change (and in the error connection refused includes the IP address so DNS seems to be working)

  • ubuntu