Docker Networking Doesn't Make Sense

dmegalon · November 25, 2023, 1:01am

Hi folks,

I am using docker in a linux virtual machine (where all the development is being done)

I am hosting a few services in a docker-compose file:

x-env: &tmenvs
  environment:
    - ADDR_NODE1=censored
    - ADDR_NODE2=censored
    - ENODE=censored
    - ETH_PROVIDER_URL=http://node1:8646

services:
  bootnode:
    ...

  node1:
    depends_on:
      - bootnode
    build:
      context: .
      dockerfile: Dockerfile.node1
    ports:
      - "8646:8646"
    expose:
      - "30306"
      - "8551"
      - "8646"
    entrypoint: ["/start-node1.sh"]
    volumes:
      - ./genesis.json:/genesis.json
    <<: *tmenvs

  wallet-core-node:
    depends_on:
      - node1
    build:
      context: ./wallet-app-node
      dockerfile: Dockerfile.walletnode
    ports:
      - "3001:3001"
    entrypoint: ["npm", "start"]
    <<: *tmenvs

In the wallet-core-node service, I am running a program that connects to node 1 using the following:

const provider = new ethers.JsonRpcProvider(providerUrl);

Where providerUrl is the address of the node1 service, and is set in an environment variable:

ETH_PROVIDER_URL=http://node1:8646

But this leads to a JsonRpc connection error, but when I replace ‘node1’ with the actual ip of the service, the connection works normally.

I’ve tried:

placing console.log statement in wallet-app-core service to verify env variable is correctly being read
hard-coding “http://node1:8646” in the JsonRpcProvider(providerUrl) call
using a shared .env file
using different ports for the provider url

Anyways, I’m 99% certain this is a docker-container networking issue, as everything works fine outside of docker.

rimelek · November 25, 2023, 10:40am

Can you share that error message?
and alos how you installed Docker. The output of the following commands can help us understand:

docker info
docker version
dpkg -l | grep docker
snap list docker

If the IP address works, then the problem could be Docker’s built-in DNS server which usually works well when Docker installed from the official epository.

dmegalon · November 25, 2023, 4:39pm

Viewing logs from inside the wallet-core container shows error:

> wallet-app-node@1.0.0 start
> ts-node --esm src/index.ts

Server running on http://localhost:3001

JsonRpcProvider failed to detect network and cannot start up; retry in 1s (perhaps the URL is wrong or the node is not started)

... JsonRpcProvider retries connection and error repeats indefinitely ...

I should note that I’ve also tried:

using custom network (bridged)
stopping the wallet-app-core service and then restarting it after making absolute certain node1 service is running

I believe I installed docker by downloading the repository and installing using apt.

$docker info
Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 129
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-37-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 25.43GiB
 Name: censored
 ID: 4c95d7e8-1d8f-4c9b-a107-827518ce2b8b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

$docker version
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:07:41 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:41 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$dpkg -l | grep docker
ii  docker-buildx-plugin                       0.11.2-1~ubuntu.22.04~jammy             amd64        Docker Buildx cli plugin.
ii  docker-ce                                  5:24.0.7-1~ubuntu.22.04~jammy           amd64        Docker: the open-source application container engine
ii  docker-ce-cli                              5:24.0.7-1~ubuntu.22.04~jammy           amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras                  5:24.0.7-1~ubuntu.22.04~jammy           amd64        Rootless support for Docker.
ii  docker-compose                             1.29.2-1                                all          define and run multi-container Docker applications with YAML
ii  docker-compose-plugin                      2.21.0-1~ubuntu.22.04~jammy             amd64        Docker Compose (V2) plugin for the Docker CLI.
ii  docker-scan-plugin                         0.23.0~ubuntu-jammy                     amd64        Docker scan cli plugin.
ii  python3-docker                             5.0.3-1                                 all          Python 3 wrapper to access docker.io's control socket
ii  python3-dockerpty                          0.4.1-2                                 all          Pseudo-tty handler for docker Python client (Python 3.x)

$snap list docker
error: no matching snaps installed

And yes, changing environment variable to IP address of node1 service works normally.

rimelek · November 25, 2023, 4:53pm

I can see you have docker compose v1 and v2 as well. Which one are you using? v1 (docker-compose) or v2 (docker compose)?

dmegalon · November 25, 2023, 5:34pm

My docker-compose file uses version ‘3.8’

Unless you meant I have 2 versions of Docker installed, which I would have been unaware of…

meyay · November 25, 2023, 6:01pm

provides v1: docker-compose

provides v2: docker compose

Those are the literal commands. v1 is deprecated since June 2023.

dmegalon · November 25, 2023, 6:46pm

I see,

I removed the older standalone version, and I am only using the plugin now.

Using docker compose version command both prior to removing older version and after yielded same results.

$docker compose version
Docker Compose version v2.21.0

So it looks like I was using v2.

Issue still persists after removing older version.

rimelek · November 25, 2023, 6:54pm

Have you tried to nslookup from a container to see if that resolves the domain name to the correct IP address?
You could also try to run another container in the compose project thta runs an nginx or httpd szerver and use curl from another container to access the webserver using its container name.

dmegalon · November 25, 2023, 7:27pm

root@00f70137ecfe:/app# nslookup node1
Server:         127.0.0.11
Address:        127.0.0.11#53

Non-authoritative answer:
Name:   node1
Address: **node1-IpAddress**

I will look into running another container that runs nginx/httpd zserver.

I do have an http server running in my node1 container, however, and using curl:

root@00f70137ecfe:/app# curl -i **node1-IpAddress**:8646
HTTP/1.1 200 OK
Vary: Origin
Date: Sat, 25 Nov 2023 19:23:35 GMT
Content-Length: 0

root@00f70137ecfe:/app# curl -i http://node1:8646
HTTP/1.1 403 Forbidden
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Sat, 25 Nov 2023 19:24:33 GMT
Content-Length: 23

invalid host specified

which is interesting…

rimelek · November 25, 2023, 7:31pm

So the Docker network seems working, but the application denies access when a hostname is used. At least this is what I could imagine.

The other reason could be that the hostname is not resolved to the correct IP address and you are trying to access a wrong container.

nslookup could answer which one is the case.

dmegalon · November 25, 2023, 11:40pm

Based off of everything I’ve done, it looks like it just denies when using hostname.

meyay · November 26, 2023, 4:25pm

As long as you use a user defined network, service discovery should work.

You nslookup output shows that the built-in dns with the ip 127.0.0.11 is used. This is correct.
Furthermore, the ip for the node1 should be the target container.

Your output indicates that name resolution itself is not the problem, otherwise it would not be able to reach the application in the target container. In case of nginx, it looks like there is no listener for that hostname.

Another issue could be dns caching: depending on the programming language and implementation, resolved hostnames could be cached, in some cases even indefinitely, which would then indeed use an outdated container ip, if the target container was restarted and received a new ip.

rimelek · November 26, 2023, 6:11pm

Oh, so there was an nslookup output. I missed that