Connect() failed (111: Connection refused) while connecting to upstream,

I have an awx docker deployment with nginx running in awx_web container. I am unable to get to the awx login page and am getting a 502 bad gateway…per the logs from awx_web’s awxserver.error.log…

connect() failed (111: Connection refused) while connecting to upstream, client: 168.44.245.38, server: awxserver, request: "GET / HTTP/1.1", upstream: "uwsgi://127.0.0.1:8050", host: "awxserver"

my nginx.con


worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens off;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
         server 127.0.0.1:8050;
         }
    upstream daphne {
        server 127.0.0.1:8051;
    }

    server {
        listen 443 ssl;
        server_name awxserver;
   
       access_log /var/log/nginx/awx.access.log;
        error_log /var/log/nginx/awx.error.log;

        ssl_certificate         /etc/nginx/awx2023.cer;
        ssl_certificate_key     /etc/nginx/awx2023.key;
        ssl_session_timeout     5m;
        ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
        ssl_protocols TLSv1.2;
        ssl_prefer_server_ciphers on;
        #client_body_buffer_size 10K;
        #client_max_body_size 200m;
        #client_header_buffer_size   8k;
        #large_client_header_buffers 4 32k;
        #3/16/23 added
        proxy_buffering off;
        proxy_buffer_size 16k;
        proxy_busy_buffers_size 24k;
        proxy_buffers 64 4k;

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG- 
        CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_pass https://204.67.211.34/;
        
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;            proxy_set_header X-Forwarded-Port 443;
            uwsgi_param HTTP_X_FORWARDED_PORT 443;
        }
    }
}

I have tried stopping, restarting containers, and multiple edits to my nginx.conf to no avail. I had run the awx installer/install.yml file before and had this environment up and running for a long time. I had to make some permanent changes and updated the docker-compose.yml, did a docker compose up -d, and since then the environment has been broken. Rerunning the install.yml playbook from the awx team does not help.

version: '2'
services:

  web:
    image: ansible/awx:15.0.0
    container_name: awx_web
    depends_on:
      - redis
      - postgres
    ports:
      - "80:8052"
      - "443:443"
    hostname: awxweb
    user: root
    restart: unless-stopped
    volumes:
      - supervisor-socket:/var/run/supervisor
      - rsyslog-socket:/var/run/awx-rsyslog/
      - rsyslog-config:/var/lib/awx/rsyslog/
      - "~/.awx/awxcompose/SECRET_KEY:/etc/tower/SECRET_KEY"
      - "~/.awx/awxcompose/environment.sh:/etc/tower/conf.d/environment.sh"
      - "~/.awx/awxcompose/credentials.py:/etc/tower/conf.d/credentials.py"
      - "~/.awx/awxcompose/nginx.conf:/etc/nginx/nginx.conf:ro"
      - "~/.awx/awxcompose/redis_socket:/var/run/redis/:rw"
      - "/var/lib/awx/projects:/var/lib/awx/projects:rw"
      - "/root/.awx/awxcompose/nginx/awx2023.key:/etc/nginx/awx2023.key"
      - "/root/.awx/awxcompose/nginx/awx2023.cer:/etc/nginx/awx2023.cer"
      - "./resolv.conf:/etc/resolv.conf"

    dns:
      - 10.1.2.3
      - 10.2.3.4
    environment:
      http_proxy:
      https_proxy:
      no_proxy:

  task:
    image: ansible/awx:15.0.0
    container_name: awx_task
    depends_on:
      - redis
      - web
      - postgres
    command: /usr/bin/launch_awx_task.sh
    hostname: awx
    user: root
    restart: unless-stopped
    volumes:
      - supervisor-socket:/var/run/supervisor
      - rsyslog-socket:/var/run/awx-rsyslog/
      - rsyslog-config:/var/lib/awx/rsyslog/
      - "~/.awx/awxcompose/SECRET_KEY:/etc/tower/SECRET_KEY"
      - "~/.awx/awxcompose/environment.sh:/etc/tower/conf.d/environment.sh"
      - "~/.awx/awxcompose/credentials.py:/etc/tower/conf.d/credentials.py"
      - "~/.awx/awxcompose/redis_socket:/var/run/redis/:rw"
      - "/var/lib/awx/projects:/var/lib/awx/projects:rw"
    dns:
      - 10.1.2.3
      - 10.2.3.4
    environment:
      http_proxy:
      https_proxy:
      no_proxy:
      SUPERVISOR_WEB_CONFIG_PATH: '/etc/supervisord.conf'

  redis:
    image: redis
    container_name: awx_redis
    restart: unless-stopped
    environment:
      http_proxy:
      https_proxy:
      no_proxy:
    command: ["/usr/local/etc/redis/redis.conf"]
    volumes:
      - "~/.awx/awxcompose/redis.conf:/usr/local/etc/redis/redis.conf:ro"
      - "~/.awx/awxcompose/redis_socket:/var/run/redis/:rw"

  postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - "~/.awx/pgdocker/10/data/:/var/lib/postgresql/data:Z"
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      http_proxy:
      https_proxy:
      no_proxy:


volumes:
  supervisor-socket:
  rsyslog-socket:
  rsyslog-config:

Why is the connectiong getting refused and not displaying my awx web interface?

Where is Nginx? According to the image history on Docker Hub, AWX is listening on port 8052 only inside the container. You forwarded port 80 into the container’s port 8082, but unless Nginx is connected to the network namespace of the awx container, nginx upstream has to be configured to use port 80 and not port 8080. In addition to that you configured Nginx to use port 8050. I don’t know if anything is listening on that. Make sure the ports are right.

Nginx is running in the awx_web container. It looks like all traffic on 80 is being forwarded to 8052.

I see docker proxies listening on 80 and 443.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      19958/docker-proxy
tcp        0      0 0.0.0.0:4400            0.0.0.0:*               LISTEN      1261/enlinuxpc64
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1518/sshd
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      1519/splunkd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1784/master
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      19979/docker-proxy
tcp        0      0 127.0.0.1:199           0.0.0.0:*               LISTEN      1529/snmpd
tcp6       0      0 :::80                   :::*                    LISTEN      19965/docker-proxy
tcp6       0      0 :::443                  :::*                    LISTEN      19986/docker-proxy

“NetworkMode”: “awxcompose_default”,
“PortBindings”: {
“443/tcp”: [
{
“HostIp”: “”,
“HostPort”: “443”
}
],
“8052/tcp”: [
{
“HostIp”: “”,
“HostPort”: “80”
}
]
},

“NetworkSettings”: {
“Bridge”: “”,
“SandboxID”: “be6debf9addaa4e675a30d3a5292a6e540b46b9fd4852044852d1cd594dbf8f5”,
“HairpinMode”: false,
“LinkLocalIPv6Address”: “”,
“LinkLocalIPv6PrefixLen”: 0,
“Ports”: {
“443/tcp”: [
{
“HostIp”: “0.0.0.0”,
“HostPort”: “443”
},
{
“HostIp”: “::”,
“HostPort”: “443”
}
],
“8052/tcp”: [
{
“HostIp”: “0.0.0.0”,
“HostPort”: “80”
},
{
“HostIp”: “::”,
“HostPort”: “80”
}
]
},

“Networks”: {
“awxcompose_default”: {
“IPAMConfig”: null,
“Links”: null,
“Aliases”: [
“awx_web”,
“web”,
“f2d53e783339”,
“awxweb”]

I am not very familiar with how the upstreams work. Am I understanding this correctly, or need more direction?