Does anyone know the minimum paid version of Docker required to be able to use sticky cookie sessions?
Docker Enterprise used to have a layer7 loadbalancer called interlock.
Since Docker sold EE, there is no docker offering with build in layer7 loadbalancer/reverse proxy.
You will need to configure your own loadbalancer. I can highly recommend to use Traefik for this.
Or you implement your own loadbalancer configuration based on haproxy, nginx or the apache webserver.
1 Like
Ok great, thanks! Also, if using nginx, should I deploy it as a Docker container? Or can nginx just be run directly on the host machine (manager node)? I have already attempted this with Traefik, and again with nginx with no luck.
Here is my current Docker swarm setup: One manager node, and multiple worker nodes. We have one container running on the manager node, which is hosting JupyterHub. When a user visits our site, they need to authenticate with GitLab. Once they are authenticated, they are redirected back to the container with cookie information. Next, JupyterHub spawns a container for the user on one of the worker nodes. What we would like is to deploy 2 of the JupyterHub containers on the one manager node, and have the users always be communicating with the same JupyterHub container. When we deploy a second container, we get the error “Bad Cookies: Missing Auth State”, or something to that effect, after the user has authenticated with GitLab and been redirected to the other manager container.
nginx can be run on the host inside a container - shouldn’t matter. For sake of simplicity I prefer to run things in containers. If you share what you tried so far, we can try to see what needs to be done differently.
I can see why you want to use sticky sessions.
Thanks again for your help! First I deployed the Traefik Docker container, following their instructions for sticky sessions, i.e. put it on the same Docker network as my service, added the sticky cookie labels, etc. I would still get the Missing Auth State error. Then I tried nginx in a container, as well as directly on the host, using a proxy_pass directive and IP_HASH, and received the same error.
Seems I was not explicit enough about what I had in mind, when I was asking for “share what you tried so far”.
Please share the exact configurations (=compose files?) you used to configure your containers.
We would prefer to deploy nginx directly on the manager node. Currently, we use nginx as a reverse-proxy to allow for our certbot to auto-renew SSL certs on port 80, while our JupyterHub container has port 8080 on the host machine mapped to its port 80. The JupyterHub container automatically forwards http traffic to https. Here is the nginx config file so far:
upstream jupyterhub {
#ip_hash;
server 0.0.0.0:8080;
}
server {
server_name example.com;
location / {
proxy_pass http://jupyterhub;
proxy_set_header Host $host;
}
}
ip_hash is commented out as it had no effect. The proxy_pass directive works as intended. We currently have just the one service deployed.
Compose file:
# Jupyterhub docker-compose configuration
version: "3.5"
services:
multiuser:
#: begin docker image selection
#:
image: jupyterhub-multiuser
#: end docker image selection
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.role == manager
volumes:
- "/usr/bin/docker:/usr/bin/docker:ro"
- "/var/run/docker.sock:/var/run/docker.sock:rw"
- "./SSL/README:/srv/jupyterhub/SSL/README:ro"
- "./SSL/:/srv/jupyterhub/SSL:ro"
- "./access:/srv/jupyterhub/access:ro"
- "./logo:/srv/jupyterhub/logo:ro"
- "./jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py:ro"
- "./jupyterhub_configold.py:/srv/jupyterhub/jupyterhub_configold.py:ro"
- "/mnt/jupyterhub_users:/jupyterhub_users:rw"
- "./cull_idle_servers.py:/srv/jupyterhub/cull_idle_servers.py:ro"
ports:
- "443:443"
- "8080:80"
networks:
- default
hostname: jupyterhub
environment:
DOCKER_NETWORK_NAME: ${DOCKER_NETWORK_NAME}
DOCKER_MACHINE_NAME: ${DOCKER_MACHINE_NAME}
JUPYTER_HUB_AUTH_CL: ${JUPYTER_HUB_AUTH_CL}
JUPYTER_HUB_AUTH_UR: ${JUPYTER_HUB_AUTH_UR}
JUPYTER_HUB_CLNT_ID: ${JUPYTER_HUB_CLNT_ID}
JUPYTER_HUB_CLNT_SE: ${JUPYTER_HUB_CLNT_SE}
JUPYTER_HUB_ADMN_NM: ${JUPYTER_HUB_ADMN_NM}
JUPYTER_HUB_WHT_LST: ${JUPYTER_HUB_WHT_LST}
JUPYTER_HUB_RO_PATH: ${JUPYTER_HUB_RO_PATH}
JUPYTER_HUB_VL_PATH: ${JUPYTER_HUB_VL_PATH}
JUPYTER_SGLEUSR_IMG: ${JUPYTER_SGLEUSR_IMG}
JUPYTER_USERGROUPID: ${JUPYTER_USERGROUPID}
JUPYTER_HUB_API_TOKEN: ${JUPYTER_HUB_API_TOKEN}
PROM_ADDRESS: ${PROM_ADDRESS}
PROM_USER: ${PROM_USER}
PROM_PASS: ${PROM_PASS}
command: jupyterhub -f /srv/jupyterhub/${JUPYTER_HUB_CONFIG} --debug
networks:
default:
external: true
name: ${DOCKER_NETWORK_NAME}
The deployment contraint pins the container to the manager node and you published container port 80 on the host port 8080.
the nginx.conf looks almost fine to me: the proxy_pass points to an upstream and you retain the host heade. Though I am currious if server 0.0.0.0:8080 actualy realy works. Since the container is reachable by 127.0.0.1:8080, I would have used server 127.0.0.1:8080 instead.
I just googled for jupyterhub nginx and found the docs for reverse proxy setup: Using a reverse proxy — JupyterHub 2.0.0 documentation
You might want to add the additional directives from the location as well. The additional directives in the server block are only necessary, if you want to use tls for https.
Thanks again! Ya, I tried 127.0.0.1:8080 and nginx couldn’t access it, or it said upstream server does not exist, or something to that effect.
Just to be sure the nginx config is located in /etc/nginx/conf.d/ and /etc/nginx/nginx.conf remained untouched? Usualy the upstream and server block belong in a http block. The defaul nginx.conf includes all files in conf.d in a http block, which should make your configuration work.
I wonder if I need something like:
upstream jupyterhub {
#ip_hash;
server multiuser_container1:8080;
server multiuser_container2:8080;
}
Because the Docker containerd PID is listening on port 8080, and it probably does round-robin load balancing once a request is forwarded to 0.0.0.0:8080
Yes, the config file is at /etc/nginx/sites-available/myconfig.conf, there is a symbolic link to it in /etc/nginx/sites-enabled, and in /etc/nginx/nginx.conf I have put:
include /etc/nginx/sites-enabled/*.conf;
Hmm, on a second though I come to think that this will not work with a host based reverse proxy, as the upstream will use the publised port of the ingress routing mesh, which itself forards traffic to a virtual ip, which then loadbalances to one of the replicas.
Lets modify your compose yml and your nginx config to work with an nginx container…
Ya, I was thinking that nginx needs to be able to access the containers through the Docker network. Then it could maybe reference the containers based on their hostname or something.
if you desperatly want to force non container patterns to containers… carry on 
Or you wait until I refactor your two files and post them again.
Though, if you scale by replica:2 how do you handle to volumes? I can’t image that jupyterhub will like if more than a single instance writes into the same folders.
I would love if you refactored my two files Also, the mounted volumes aren’t an issue, as they are not written to by the JupyterHub containers. They are mostly read-only, and the one that is read-write is for individual user folders. The single containers write to them, and they are hosted on an nfs server, which I assume is designed to have many users writing concurrently…
*** EDIT because I can’t reply anymore haha *****
Awesome, I will give it a try shortly!!
the compose.yml now has the nginx additionaly and the port mapping removed on the other service:
version: "3.8"
services:
nginx:
image: nginx:1.21.4
volumes:
- ./default.conf:/etc/nginx/conf.d/default.conf
ports:
- "80:80"
networks:
- default
multiuser:
image: jupyterhub-multiuser
deploy:
mode: replicated
replicas: 2
placement:
constraints:
- node.role == manager
volumes:
- "/usr/bin/docker:/usr/bin/docker:ro"
- "/var/run/docker.sock:/var/run/docker.sock:rw"
- "./SSL/README:/srv/jupyterhub/SSL/README:ro"
- "./SSL/:/srv/jupyterhub/SSL:ro"
- "./access:/srv/jupyterhub/access:ro"
- "./logo:/srv/jupyterhub/logo:ro"
- "./jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py:ro"
- "./jupyterhub_configold.py:/srv/jupyterhub/jupyterhub_configold.py:ro"
- "/mnt/jupyterhub_users:/jupyterhub_users:rw"
- "./cull_idle_servers.py:/srv/jupyterhub/cull_idle_servers.py:ro"
networks:
- default
hostname: jupyterhub
environment:
DOCKER_NETWORK_NAME: ${DOCKER_NETWORK_NAME}
DOCKER_MACHINE_NAME: ${DOCKER_MACHINE_NAME}
JUPYTER_HUB_AUTH_CL: ${JUPYTER_HUB_AUTH_CL}
JUPYTER_HUB_AUTH_UR: ${JUPYTER_HUB_AUTH_UR}
JUPYTER_HUB_CLNT_ID: ${JUPYTER_HUB_CLNT_ID}
JUPYTER_HUB_CLNT_SE: ${JUPYTER_HUB_CLNT_SE}
JUPYTER_HUB_ADMN_NM: ${JUPYTER_HUB_ADMN_NM}
JUPYTER_HUB_WHT_LST: ${JUPYTER_HUB_WHT_LST}
JUPYTER_HUB_RO_PATH: ${JUPYTER_HUB_RO_PATH}
JUPYTER_HUB_VL_PATH: ${JUPYTER_HUB_VL_PATH}
JUPYTER_SGLEUSR_IMG: ${JUPYTER_SGLEUSR_IMG}
JUPYTER_USERGROUPID: ${JUPYTER_USERGROUPID}
JUPYTER_HUB_API_TOKEN: ${JUPYTER_HUB_API_TOKEN}
PROM_ADDRESS: ${PROM_ADDRESS}
PROM_USER: ${PROM_USER}
PROM_PASS: ${PROM_PASS}
command: jupyterhub -f /srv/jupyterhub/${JUPYTER_HUB_CONFIG} --debug
networks:
default:
external: true
name: ${DOCKER_NETWORK_NAME}The compose file expects a file called default.conf with the rp rules to be present in the same folder:
upstream jupyterhub {
ip_hash;
server tasks.multiuser:80;
# set DNS resolver as Docker internal DNS
resolver 127.0.0.11 valid=10s;
resolver_timeout 5s;
}
server {
listen 80;
server_name example.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# websocket headers
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Scheme $scheme;
proxy_buffering off;
proxy_pass http://jupyterhub;
}
}Now the upstream server uses tasks.${servicename}:80 as servername, which would return the container ip’s in round robbin mode, but is now pinned with ip_hash. Since we use container to container communication, the container port of jupyter needs to be used in the upstream configuration (this is what I did). Make sure to adjust the server_name directive in the server block.
Not the tasks.servicename dns-rr resolution only exists if deployed as swarm stack using docker stack deploy.
I will be trying this out shortly! Thanks! I wasn’t allowed to reply to this yesterday as I had used up my rookie reply quota 