Hi All!
I have a nightmare situation I could really use some assistance with and have no idea where to start.
I’ve been using Docker and Docker Compose for many years now and just hit a brick wall.
Here’s some context:
I manage several GCP Compute instances running CentOS 7 with Docker and Docker Compose, and after updating one of them yesterday with yum update the VM crashed and was no-longer accessible by any means (I usually use ssh).
So after a day of trying to figure out what happened and reading through endless documentation as well as post regarding similar issues I had no choice but to completely rebuild (by means of creating a new VM) and installing Docker and Docker Compose fresh. The only thing I could do to recover all the files I previously used to orchestrate my Docker containers was to make a snapshot of the bad VM boot disk and create a new external disk from the snapshot and attach it to the new VM (just to copy all my docker-compose.yml and Dockerfile’s for use to re-setup everything from ground zero.
I probably hadn’t updated Docker in about 8 or 9 months on the VM that died and the files I was using and any new containers created up till the VM died over the last couple months always worked fine without a problem. Specifically when launching new websites.
For the life of me I can’t get anything to work right now. Even working with the simplest docker-compose.yml that build from a custom Dockerfile. The build process don’t fail and the containers seem to run just fine but all I get is permission denied when trying to access a domain publicly with the following.
“403 Forbidden You don’t have permission to access this resource.”
I started with using the initial build files to stripping them down to practically nothing just to try and get a simple index page to display. Which has caused me a headache so horrible I feel like I’m bouncing it off a brick wall.
It’s likely something silly that I’m just over-looking but for the life of me I can’t figure it out.
I primarily use Docker Compose (and have noticed a few things, you no-longer need to use docker-compose [commands] etc. but instead now use just docker compose [commands] - plus now it seems if you don’t use volume down even when building with no-cache they don’t get update “which was super frustrating till I figured that one out”)
Aside from that I need some help figuring this out and I’m hoping if I provide my general workflow someone might have an answer for me.
So here goes:
To start I use a 3 container system to handle multiple domains on a VM with docker as well as handles the creation of SSL certificates with Letsencrypt (which are orchestrations I pull from Docker hub created by another user - which has worked great for as long as I can remember).
Those are as follows: (in a docker-compose.yml)
- NGINX which uses image: nginx
- Docker-gen which uses image: jwilder/docker-gen:latest
- Letsencrypt which uses image: jrcs/letsencrypt-nginx-proxy-companion:latest
For those to work I need to first create an external network with a subnet and gateway (so I can keep things organized with internal IP’s) which prevents IP’s from changing when say google migrates my VM and it restarts because it uses reverse proxy and I need the logs on the hosted websites to provide a users real IP for various different reasons (blocking, etc.)
I don’t think any of that setup is causing the problem but at this point I honestly couldn’t say for sure. If needed I can update this with exactly what’s in my docker-compose.yml for those 3 containers (which is pretty much identical across several VM’s running this setup) on the VM’s that I haven’t had any problem with everything still works as expected (but they too haven’t had Docker updated in months, and at this point I don’t want to touch them until I have a better idea of what is going on.
All that is running from root in a directory labeled ‘host’
Then for each website hosted on a VM I have a directory inside host “named for the domain” which then contains their own docker-compose.yml with a build subdirectory containing the Dockerfile to build from which usually starts with:
FROM php:$version-apache-buster followed by installing php extensions, adding any config and .ini files and installing any tools I would generally use like composer, git, drush etc. when I bash into the running container to make updates etc. to the website (as it’s pretty much constant with my clients)
90% of the sites I host are built on Drupal CMS and for the most part I’ve found it easiest to just manage Drupal from within the container versus having it setup in the build process because I use bind mounted volumes to an attached persistent SDD in the VM to make sure I don’t lose anything when a container needs updating like PHP version, etc.
Currently on this new VM the containers build fine and I’m not doing anything differently as far as I can tell yet I cannot access them publicly through a web browser (as stated I just get 403 Forbidden). That’s with just a simple index.php - if I can get past that I’d be good to go. However after 2 solid days no luck I need to know if something changed in Docker in the most resent updates that might be working against me.
In apache my configuration looks correct when comparing it to sites on another VM with DocumentRoot set to the correct location.
The only difference I’ve notice (and maybe this is the real problem, that I’m not sure how to handle is) if I bash into the nginx container which lists each domains as a server in default.conf is on this new server they list as follows:
thedomain.extension
upstream thedomain.extension {
# Cannot connect to network of this container
server 127.0.0.1 down;
## Can be connected with “docker-network-name” network
# web-containername
server [the-ip-i-set-in-docker-compose]:80;
}
Where on the older VM’s the # Cannot connect to network of this container
- is not present but the ## Can connect with proper IP is
So I’m seeing that newly added information and am wondering if that is what’s causing it, but I have no idea why that’s now present or why it would be causing the problem (my assumption is cause the new VM is running the latest nginx and it’s just a newly added notation) and considering there is still a # Can be connected this is not the problem (but I just don’t know)
I’m hopeful someone may have run into this and no what’s going on as I can’t be the only one this has happened to!
I can also provide the individual docker-compose.yml and Dockerfile info if someone would be willing to help me figure this out - as my brain is fried on this one (I’ve never experienced this issue like this) anytime in the past if I ended up with a 403 forbidden it was as simple as a permission issue where doing a chown -R www-data:www-data on the DocumentRoot fixed the problem.