Hostnames and IPs in a Swarm

Docker Engine Swarm
1

Hi Guys,

I have a web server and a web client.
A few days ago, I used to test the app using the below Compose file locally with Docker Desktop.

  • The web-server listens to the host specified by SERVER_HOST on port 80:90
  • The replicated clients send requests to web-server:8090, and it works like a charm.
# compose.yml
version: '3.8'
services:
  web-server:
    image: web-server
    environment:
      - SERVER_HOST=web-server
    ports:
      - 8090:8090

  web-client:
    image: web-client
    environment:
      - SERVER_HOST=web-server
    command: ['-n', '25000']
    deploy:
      mode: replicated
      replicas: 5
    depends_on:
      - web-server

Since then I’ve created a swarm with multipass comprised of 2 nodes: a manager and a worker.
I’ve updated the compose file as below, and published the images on a private registry on the manager node (172.17.0.1:5000), but once I deploy the stack I see this log on the web-server service:

> docker service logs web_web-server
2023/11/27 18:15:50 Listening and serving  on https://web-server:8090 in debug mode
2023/11/27 18:15:50 HTTP server error: listen tcp 212.95.74.75:8090: bind: cannot assign requested address

In the updated compose file below, you can see I’m using the service name web-server in the SERVER_HOST environment variable that is used by the web-server golang program.

So, the web-server hostname is resolved to 212.95.74.75…

version: '3.8'
services:
  web-server:
    image: 172.17.0.1:5000/web-server
    build:
      context: ..
      dockerfile: ./web/server/Dockerfile
    environment:
      - SERVER_HOST=web-server
    ports:
      - 8090:8090
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

  web-client:
    image: 172.17.0.1:5000/web-client
    build:
      context: ..
      dockerfile: ./web/client/Dockerfile
    environment:
      - SERVER_HOST=web-server
    command: ['-n', '25000']
    deploy:
      mode: replicated
      replicas: 5
      placement:
        constraints: [node.role == worker]
    depends_on:
      - web-server

If I inspect the manager node network configuration, I can see this IP address 192.168.72.14.

> docker node inspect manager
...
   "Status": {
        "State": "ready",
        "Addr": "192.168.72.14"
    },
    "ManagerStatus": {
        "Leader": true,
        "Reachability": "reachable",
        "Addr": "192.168.72.14:2377"
    }
  }
]

If I thought I couldn’t use the service name, I would probably have used this IP but if I use SERVER_HOST=192.168.72.14 in my compose file, it doesn’t work either.

Tech notes

  • nodes IPs
> multipass ls                                                   
Name                    State             IPv4             Image
manager                 Running           192.168.73.4     Ubuntu 22.04 LTS
                                          172.18.0.1
                                          172.17.0.1
worker                  Running           192.168.73.3     Ubuntu 22.04 LTS
                                          172.17.0.1
                                          172.18.0.1
  • web-server service network conf
> docker service inspect --format '{{ json .Endpoint }}' web-server | jq .
{
  "Spec": {
    "Mode": "vip",
    "Ports": [
      {
        "Protocol": "tcp",
        "TargetPort": 8090,
        "PublishedPort": 8090,
        "PublishMode": "ingress"
      }
    ]
  },
  "Ports": [
    {
      "Protocol": "tcp",
      "TargetPort": 8090,
      "PublishedPort": 8090,
      "PublishMode": "ingress"
    }
  ],
  "VirtualIPs": [
    {
      "NetworkID": "vhaoorbx46167zruk4czdbm10",
      "Addr": "10.0.0.203/24"
    },
    {
      "NetworkID": "p4ahe7agvg1yryd81q1n1ouye",
      "Addr": "10.0.5.2/24"
    }
  ]
}
  • manager node docker-related IPv4
ip -4 -o address show | grep docker
3: docker_gwbridge    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge\       valid_lft forever preferred_lft forever
4: docker0    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\       valid_lft forever preferred_lft forever
  • docker networks (can’t find any of the network IDs from the web-server network config)
> docker network ls                                                                   
NETWORK ID     NAME              DRIVER    SCOPE
463982250b78   bridge            bridge    local
6c3dbf997c47   docker_gwbridge   bridge    local
p4ahe7agvg1y   gorilla_default   overlay   swarm
d5648dc92f79   host              host      local
vhaoorbx4616   ingress           overlay   swarm
a1ee9f2562f8   none              null      local
2

Please try to fix the formatting of the second half of your post, it’s hard to read.

1 Like
3

Within the container, your application can listen to 0.0.0.0:8090 (any available IP), you don’t need to restrict it to SERVER_HOST. It only matters on which ports your expose with ports externally.

1 Like
4

Several errors here:

TDLR

  • As @bluepuma77 said, the app must listen to any IP, i.e. 0.0.0.0
  • The registry must be declared on 192.168.73.4:5000
  • The certificate file ca.crt used to create the secured registry must be copied to /etc/docker/certs.d/192.168.73.4\:5000 in the worker node to make its docker daemon trust the manager’s docker daemon. From here the worker can then pull the image and refer to the service by its name web-server

Copying the certificate from the manager to the worker node

From the manager node, I tried to sudo scp /etc/docker/certs.d/192.168.73.4\:5000/ca.crt boulard@worker.local:/home/boulard, but the worker node doesn’t trust the manager node through scp/ ssh. I understood that I would need to add the manager ssh public key to the worker node in ~/.ssh/authorized_keys etc. Eventually, I just used the multipass transfer command to copy the ca.crt
certificate from the manager node to the worker node like this:

multipass transfer manager:/etc/docker/certs.d/192.168.73.4\:5000/ca.crt .
multipass transfer ./ca.crt worker:/etc/docker/certs.d/192.168.73.4\:5000/

Side note: multipass transfer resolves the worker hostname to ubuntu@worker.local, so if you specify multipass transfer ./ca.crt worker:., it will land in /home/ubuntu. I’m saying this cause I’ve created both manager and worker instances with a cloud-init.yml file that creates a boulard system user into both nodes. I probably should have specified the ubuntu user cause the self-signed certificate created on the host to ssh into the nodes has nothing to do with the system user added to the nodes that are part of the docker group. Below is the cloud-init.yml file for reference:

#cloud-config
users:
  - name: ubuntu # add the "ubuntu" system user, multipass requires this EXACT name
    ssh-authorized-keys: # set the public key from "boulard" user to be trusted 
      - ecdsa-sha2-nistp256 AAAAE2... boulard
    sudo: ALL=(ALL) NOPASSWD:ALL # password-less sudo capability, create a file in the /etc/sudoers.d directory called 90-cloud-init-users
    groups: sudo
    shell: /bin/bash
package_update: true
packages:
  - docker
  - avahi-daemon
  - apt-transport-https
  - ca-certificates
  - curl
  - gnupg
  - lsb-release
runcmd:
  - sudo curl -fsSL https://get.docker.com | sudo bash
  - sudo systemctl enable docker
  - sudo systemctl enable -s HUP ssh 
  - sudo groupadd docker 
  - sudo usermod -aG docker ubuntu