Overlay network not passing requests through?

General Discussions General
1

I’ve been using docker for a while but decided to go back through the getting started tutorial as I haven’t been using composer and it seems to have changed a lot with stack and swarm.

I’m up to https://docs.docker.com/get-started/part3/ , I have a docker-compose.yml that’s basically what’s in the tutorial:

version: "3"
services:
  web:
    # replace username/repo:tag with your name and image details
    image: lyte/get-started:part2
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: "0.1"
          memory: 50M
      restart_policy:
        condition: on-failure
    ports:
      - "4001:80"
    networks:
      - webnet
networks:
  webnet:

After deploying:

$ docker stack deploy -c docker-compose.yml getstartedlab
Creating network getstartedlab_webnet
Creating service getstartedlab_web

Confirming they’re running:

$ docker stack ps getstartedlab
ID                  NAME                  IMAGE                    NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
pqbqehczfw2k        getstartedlab_web.1   lyte/get-started:part2   davesmac            Running             Running 2 minutes ago                       
iqqu4z7xna5f        getstartedlab_web.2   lyte/get-started:part2   davesmac            Running             Running 2 minutes ago                       
fm4d2tapguoz        getstartedlab_web.3   lyte/get-started:part2   davesmac            Running             Running 2 minutes ago

Confirming I have a service:

$ docker stack services getstartedlab
ID                  NAME                MODE                REPLICAS            IMAGE                    PORTS
br08pnqiygph        getstartedlab_web   replicated          3/3                 lyte/get-started:part2   *:4001->80/tcp

Curl just gets stuck (never times out or completes):

$ curl -sv localhost:4001
* Rebuilt URL to: localhost:4001/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 4001 (#0)
> GET / HTTP/1.1
> Host: localhost:4001
> User-Agent: curl/7.58.0
> Accept: */*
>

I can confirm the service is running ok in the container (docker exec -it getstartedlab_web.1.... bash, apt update, apt install curl etc).

I’m using Ubuntu 18.04 with:

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.4
 Git commit:        e68fc7a
 Built:             Fri Oct 19 19:43:14 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       e68fc7a
  Built:            Thu Sep 27 02:39:50 2018
  OS/Arch:          linux/amd64
  Experimental:     false

TLDR

I’m assuming there’s something wrong with the way the webnet network is loadbalancing, any ideas on how to debug it?

2

I’ve discovered that if I connect to anything other than the localhost IP (e.g an IP assigned by the DHCP server on my lan to a physical nic) it’s fine.

3

It works if I explicitly type out 127.0.0.1 - it turns out that it’s just IPv6 attempts that are getting stuck indefinitely.

Disabling the ::1 entry in /etc/hosts is a possible work around, but not ideal.

1 Like
4

Same issue. Also noticed that using the machine IP address (in my case, 10.0.0.4) would work. However, it would not work from other nodes in the swarm.

I discovered the swarm ingress network also used 10.0.0.0/24, which conflicted with my local network. Had to destroy all services, remove the ingress network, and re-create it with a different subnet. Directions at https://docs.docker.com/network/overlay/#customize-the-default-ingress-network

I can now access things using the host’s IP address, 127.0.0.1, and using the host ip address from a different machine on the same network. Still having trouble with localhost when run from the swarm master node