Updated Docker and Everything is Broken

Hi All!

I have a nightmare situation I could really use some assistance with and have no idea where to start.

I’ve been using Docker and Docker Compose for many years now and just hit a brick wall.

Here’s some context:
I manage several GCP Compute instances running CentOS 7 with Docker and Docker Compose, and after updating one of them yesterday with yum update the VM crashed and was no-longer accessible by any means (I usually use ssh).

So after a day of trying to figure out what happened and reading through endless documentation as well as post regarding similar issues I had no choice but to completely rebuild (by means of creating a new VM) and installing Docker and Docker Compose fresh. The only thing I could do to recover all the files I previously used to orchestrate my Docker containers was to make a snapshot of the bad VM boot disk and create a new external disk from the snapshot and attach it to the new VM (just to copy all my docker-compose.yml and Dockerfile’s for use to re-setup everything from ground zero.

I probably hadn’t updated Docker in about 8 or 9 months on the VM that died and the files I was using and any new containers created up till the VM died over the last couple months always worked fine without a problem. Specifically when launching new websites.

For the life of me I can’t get anything to work right now. Even working with the simplest docker-compose.yml that build from a custom Dockerfile. The build process don’t fail and the containers seem to run just fine but all I get is permission denied when trying to access a domain publicly with the following.

“403 Forbidden You don’t have permission to access this resource.”
I started with using the initial build files to stripping them down to practically nothing just to try and get a simple index page to display. Which has caused me a headache so horrible I feel like I’m bouncing it off a brick wall.

It’s likely something silly that I’m just over-looking but for the life of me I can’t figure it out.

I primarily use Docker Compose (and have noticed a few things, you no-longer need to use docker-compose [commands] etc. but instead now use just docker compose [commands] - plus now it seems if you don’t use volume down even when building with no-cache they don’t get update “which was super frustrating till I figured that one out”)

Aside from that I need some help figuring this out and I’m hoping if I provide my general workflow someone might have an answer for me.

So here goes:

To start I use a 3 container system to handle multiple domains on a VM with docker as well as handles the creation of SSL certificates with Letsencrypt (which are orchestrations I pull from Docker hub created by another user - which has worked great for as long as I can remember).

Those are as follows: (in a docker-compose.yml)

  • NGINX which uses image: nginx
  • Docker-gen which uses image: jwilder/docker-gen:latest
  • Letsencrypt which uses image: jrcs/letsencrypt-nginx-proxy-companion:latest

For those to work I need to first create an external network with a subnet and gateway (so I can keep things organized with internal IP’s) which prevents IP’s from changing when say google migrates my VM and it restarts because it uses reverse proxy and I need the logs on the hosted websites to provide a users real IP for various different reasons (blocking, etc.)

I don’t think any of that setup is causing the problem but at this point I honestly couldn’t say for sure. If needed I can update this with exactly what’s in my docker-compose.yml for those 3 containers (which is pretty much identical across several VM’s running this setup) on the VM’s that I haven’t had any problem with everything still works as expected (but they too haven’t had Docker updated in months, and at this point I don’t want to touch them until I have a better idea of what is going on.

All that is running from root in a directory labeled ‘host’

Then for each website hosted on a VM I have a directory inside host “named for the domain” which then contains their own docker-compose.yml with a build subdirectory containing the Dockerfile to build from which usually starts with:
FROM php:$version-apache-buster followed by installing php extensions, adding any config and .ini files and installing any tools I would generally use like composer, git, drush etc. when I bash into the running container to make updates etc. to the website (as it’s pretty much constant with my clients)

90% of the sites I host are built on Drupal CMS and for the most part I’ve found it easiest to just manage Drupal from within the container versus having it setup in the build process because I use bind mounted volumes to an attached persistent SDD in the VM to make sure I don’t lose anything when a container needs updating like PHP version, etc.

Currently on this new VM the containers build fine and I’m not doing anything differently as far as I can tell yet I cannot access them publicly through a web browser (as stated I just get 403 Forbidden). That’s with just a simple index.php - if I can get past that I’d be good to go. However after 2 solid days no luck I need to know if something changed in Docker in the most resent updates that might be working against me.

In apache my configuration looks correct when comparing it to sites on another VM with DocumentRoot set to the correct location.
The only difference I’ve notice (and maybe this is the real problem, that I’m not sure how to handle is) if I bash into the nginx container which lists each domains as a server in default.conf is on this new server they list as follows:


upstream thedomain.extension {
# Cannot connect to network of this container
server down;
## Can be connected with “docker-network-name” network
# web-containername
server [the-ip-i-set-in-docker-compose]:80;
Where on the older VM’s the # Cannot connect to network of this container

  • is not present but the ## Can connect with proper IP is

So I’m seeing that newly added information and am wondering if that is what’s causing it, but I have no idea why that’s now present or why it would be causing the problem (my assumption is cause the new VM is running the latest nginx and it’s just a newly added notation) and considering there is still a # Can be connected this is not the problem (but I just don’t know)

I’m hopeful someone may have run into this and no what’s going on as I can’t be the only one this has happened to!

I can also provide the individual docker-compose.yml and Dockerfile info if someone would be willing to help me figure this out - as my brain is fried on this one (I’ve never experienced this issue like this) anytime in the past if I ended up with a 403 forbidden it was as simple as a permission issue where doing a chown -R www-data:www-data on the DocumentRoot fixed the problem.


Still having issues however now instead of 403 forbidden, I’m now getting 404 not found with a simple php debian/apache container with a single page “index.php”

and nothing suggested anywhere on the web is working!

Figured it all out…

Should someone else run into a situation like this here is what my problem was:
1st and the major issue was with docker-gen (jwilder/docker-gen) was using an outdated nginx.tmpl which is what caused:

Cannot connect to network of this container

server down;
in the:
upstream domain_name.tld { } in the default.conf running in the nginx container

  • once I updated to the latest nginx.tmpl that docker-gen used that statement disappeared and I no-longer received the notice of “403 Forbidden You don’t have permission to access this resource.” when visiting the domain url via browser.

Having rebuilt from the ground up I assumed the nginx.tmpl needed for docker-gen would have been the same as the one I originally had downloaded (as it was required you download that file separate from the containers used to orchestrate nginx with docker-gen using reverse proxy for multiple hosted domain containers before running them) however pulling the latest versions of the 3 containers being they had been updated since I originally set things up that template was also updated and the update was needed as well.

The second problem mentioned in previous reply “404 Not Found”

  • was caused by a stupid/simple typo of the DocumentRoot path in my Dockerfile when overriding default-site config in <VirtualHost *:80>…
  • instead of “/var/www/html”, I had “/var/www/htm” (dohhh)

In my haste trying to figure out what the problem was I rewrote my Dockerfile pretty much stripping everything out I could think of that might have been causing the problem when I was getting the 403 Forbidden error.

  • which turns out it had nothing to do with any of my original Dockerfile’s or docker-compose.yml’s.

Now all is good to go, everything working as expected - all sites are back up on the new VM.

Hopefully my stupidity will help someone else in the future…