Concurrent service update without downtime

General
1

I’m trying to host a Django website and maintain it with Docker. Because this service uses WebSockets, I happen to have 4 Docker services:

  • The PostgreSQL database (could be anything, even nothing since Django can work with sqlite3).
  • Gunicorn (which supports HTTP requests to the Django app).
  • Daphne (which supports WS requests to the Django app).
  • nginx which behaves as a front-end and proxy-reverse to Gunicorn or Daphne depending on the location (/ or /ws).

Simple and straightforward. The problems arise when I modify my Django app. To be sure, the app used by Gunicorn and the one used by Daphne are exactly identical (they use the same Git repository with the same code and same settings). This allows for easier development. It means that when the code changes, I have to update both Gunicorn and Daphne. Although both run as Docker services with a start-first policy, the nginx Docker service fails to reach either during the 10-15 seconds that the update lasts. I would have thought Docker would start Gunicorn and Daphne into new containers and, only when they are ready to serve, update them in the Docker network so that the nginx Docker service would always have something to work with.

This is more a question on the theory of my hierarchy. However, the problem might not be in the theory, but in my configuration and I’ll paste it here (at least, what I believe to be relevant).

# docker-compose.yml
version: '3.7'

services:
  gunicorn:
    restart: always
    image: myproj:latest
    command: gunicorn myproj.wsgi:application --bind :8000
    volumes:
      - ./:/usr/src/app/
    environment:
      - SECRET_KEY=please_change_me
      - SQL_ENGINE=django.db.backends.postgresql
      - SQL_DATABASE=postgres
      - SQL_USER=user
      - SQL_PASSWORD=MyPasswordHere
      - SQL_HOST=db
      - SQL_PORT=5432
      - DATABASE=postgres
    depends_on:
      - db
  daphne:
    restart: always
    image: myproj:latest
    command: daphne --root-path=/usr/src/app/ -b 0.0.0.0 -p 9000 --proxy-headers myproj.asgi:application
    volumes:
      - ./:/usr/src/app/
    environment:
      - SECRET_KEY=please_change_me
      - SQL_ENGINE=django.db.backends.postgresql
      - SQL_DATABASE=postgres
      - SQL_USER=user
      - SQL_PASSWORD=MyPasswordHere
      - SQL_HOST=db
      - SQL_PORT=5432
      - DATABASE=postgres
    depends_on:
      - db
      - gunicorn
  nginx:
    restart: always
    image: myproj-nginx:latest
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./static/:/etc/nginx/static/
  db:
    restart: always
    image: postgres:10.5-alpine
    volumes:
      - ./postgres-data:/var/lib/postgresql/data

I don’t think sharing the myproj image would be relevant here. However, I will share my myproj-nginx Dockerfile and configuration file as it might, possibly, help:

# myproj-nginx/Dockerfile
FROM nginx:1.15.0-alpine

RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d
COPY letsencrypt /etc/letsencrypt

# Andd myproj-nginx/nginx.conf
upstream gunicorn {
    server gunicorn:8000;
}

upstream daphne {
    server daphne:9000;
}

server {
    if ($host = www.nolink) {
        return 301 ... can't post that in Docker forums
    } # managed by Certbot


    if ($host = example.net) {
        return 301 ... can't post that in Docker forums
    } # managed by Certbot


    listen 80;
    listen [::]:80;
    server_name nolink www.nolink;
    return 404; # managed by Certbot
}

server {
    server_name nolink www.nolink;
    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


    location / {
        proxy_pass http://gunicorn;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }
    location /ws/ {
        proxy_pass http://daphne;
        proxy_http_version 1.1;
        proxy_read_timeout 86400;
        proxy_redirect     off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;
    }
    location /static/ {
        alias /etc/nginx/static/;
    }
}

This configuration seems to work rather well when the stack is started (with docker stack deploy -c docker-compose.yml myproj). However, when trying to update gunicorn and daphne, nginx gets a 10-15 seconds when it can’t connect to either and sends back 502 responses. This is some downtime I would really like to see gone.

Any help would be welcome! Thanks in advance,