Curl or telnet timed out when access a docker network interface

[dockerzung]

My environment: Windows 11 + WSL2 + Ubunto20.4+ Docker desktop

I started up a service (vaultwarden) via a docker-compose file. The service appeared to be running OK without any error as seen using the log. However I could not curl or telnet to vaultwarden docker network interface (172.20.0.2). It stalled producing minimum output or timed out …

curl -v 172.20.0.2
*   Trying 172.20.0.2:80...
* TCP_NODELAY set
^C

 "Containers": {
            "08366d53960256ab1ee4fbadc65cd966ab0a787c41e47645e61985b08cfd9fbb": {
                "Name": "vaultwarden",
                "EndpointID": "fa9fc0ea68e087c4594fef26d90428281cb6b754d2412974629d23316b5a0f4e",
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.19.154.7  netmask 255.255.240.0  broadcast 172.19.159.255
        inet6 fe80::215:5dff:fe2e:eabc  prefixlen 64  scopeid 0x20<link>
        ether 00:15:5d:2e:ea:bc  txqueuelen 1000  (Ethernet)
        RX packets 23632  bytes 3454321 (3.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 251  bytes 14943 (14.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 14  bytes 964 (964.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 964 (964.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Can someone know the reason for the connection time-out?

[meyay (Metin Y.)]

Please share your compose file, so we can see whether you map a host port to your container.

The command you ran would only work inside the docker-desktop wsl distribution. Did you run it there?
If not, where did you run it, on the windows host?

[dockerzung]

here is the compose file

version: '3'

services:
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: always
    environment:
      WEBSOCKET_ENABLED: "true"  # Enable WebSocket notifications.
#      DOMAIN: "https://ubuntu22"
      SIGNUPS_ALLOWED: "false"
      INVITATIONS_ALLOWED:  "false"
      LOG_LEVEL: "warn"
#      DOMAIN: "localhost"
      SMTP_HOST: "smtp.google.com"
      SMTP_FROM: "myemail"
      SMTP_PORT: "587"
      SMTP_SECURITY: "starttls"
      SMTP_USERNAME: "myemail"
      SMTP_PASSWORD: "nothing"

    volumes:
      - ./vw-data:/data

  caddy:
    image: caddy:2
    container_name: caddy
    restart: always
    ports:
      - 80:80  # Needed for the ACME HTTP-01 challenge.
      - 443:443
    volumes:
      - ./caddy:/usr/bin/caddy      #  Your custom build of Caddy
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-config:/config
      - ./caddy-data:/data

    environment:
      DOMAIN: "subdomain.duckdns.org"      # Your domain.
      EMAIL: "zung102@yahoo.com"            # The email address to use for ACME registration.
      DUCKDNS_TOKEN: "token taken from duckdns website"
      LOG_FILE: "/data/access.log"

I ran the command from WSL console …

zung@Dzungabc:~/vaultwarden$ telnet 172.20.0.2
Trying 172.20.0.2...
telnet: Unable to connect to remote host: Connection timed out

[meyay (Metin Y.)]

I assume in a wsl console, you mean inside a distro where you activated the integration? Of course this is not going to work, as the docker engine is not running inside that distro, it runs in the docker-desktop distro. Only the host that runs the docker engine is able to directly communicate with container ips in container networks - though, it’s actually not recommended to communicate to a container by its ip at all.

If you try to access a container by its ip, then you are high likely trying to do something in a way it is not intended to be done.

For host to container access, the container port needs to published, and the host-ip (or localhost) and the published host port need to be used. For container to container access, the service name and the container port need to be used.

Since I see caddy in your compose file, I assume it acts as a reverse proxy and forwards traffic to the vault warden container. In your caddy configuration the reverse_proxy setting must look like this: reverse_proxy vaulwarden:80

[dockerzung]

WSL (Windows Subsystem for Linux) console is like a Ubuntu terminal. From it I ran docker and other Ubuntu work. So it is the host for docker. Can it access docker network (like curl or telnet) ? Should command “ifconfig -a” show Ubuntu network as well as Docker network? Are there other work needed to make it happen?

[meyay (Metin Y.)]

Your response shows that my last post was unclear to you. Instead of asking what was not clear, you choose to explain basic knowledge about wsl and repeat questions you already had without actually addressing anything of what I asked. Clearly we are not on the same page yet.

Please share the output of: wsl --list so we can see the list of available distributions on your machine and which one is the default. Furthermore, share how you start/enter the distribution you are trying to run the commands in.

[dockerzung]

I am sorry for misunderstanding your intention. This is new to me so I was not thinking properly.

PS C:\Users\user1> wsl -l -v
NAME STATE VERSION

• Ubuntu-20.04 Running 2
  docker-desktop-data Running 2
  docker-desktop Running 2
  Ubuntu Running 2
  PS C:\Users\user1> wsl --set-default Ubuntu-20.04
  The operation completed successfully.
  PS C:\Users\user1> wsl -l -v
  NAME STATE VERSION
• Ubuntu-20.04 Running 2
  docker-desktop-data Running 2
  docker-desktop Running 2
  Ubuntu Running 2

I have the same issue running from WSL console or Ubuntu20.4 terminal. The Desktop Docker allows WSL and Ubuntu 20.4 integration. Both are created using Windows Terminal.

[meyay (Metin Y.)]

@meyay

Only the host that runs the docker engine is able to directly communicate with container ips in container networks

With Docker Desktop with WSL2 Backend the “host” mentioned above is the docker-desktop distribution.

You can not access a container by a container ip in Distribution where you enabled the Docker Desktop integration for, as the container networks are private networks inside the docker-desktop distribution.

This is what I tried to say with this:

@meyay

Of course this is not going to work, as the docker engine is not running inside that distro, it runs in the docker-desktop distro. Only the host that runs the docker engine is able to directly communicate with container ips in container networks - though, it’s actually not recommended to communicate to a container by its ip at all.

And I even suggested how it’s done correctly:

@meyay

you try to access a container by its ip, then you are high likely trying to do something in a way it is not intended to be done.

For host to container access, the container port needs to published, and the host-ip (or localhost) and the published host port need to be used. For container to container access, the service name and the container port need to be used.

Since I see caddy in your compose file, I assume it acts as a reverse proxy and forwards traffic to the vault warden container. In your caddy configuration the reverse_proxy setting must look like this: reverse_proxy vaulwarden:80

Long story short: you are trying to use it wrong.

[dockerzung]

I think I understand what you were explaining. In my Windows I also have an Ubuntu app. Invoking it brought up a Ubuntu terminal and Docker does NOT exist here. Is it a totally separate Ubuntu environment and I can proceed to install Docker then Vaultwarden and Caddy? and it now can access this docker network ?

[meyay (Metin Y.)]

The “Ubuntu App” creates a WSL distribution.

Docker Desktop allows to enable WSL integration in its settings → Resources → WSL Integration. If you enable it for the default WSL distribution or any of the listed WSL distributions, the docker client will be available inside that WSL distribution, even though the docker engine does not exist - it will use the docker engine from the docker-desktop WSL distribution as backend.

So assumed, you did NOT activate the WSL integration for your Ubuntu distribution, then yes this distribution would be independent of Docker Desktop and you could install Docker-CE (=docker engine + cli) using the convenience script:

curl -fsSL https://get.docker.com -o get-docker.sh
DRY_RUN=1 sudo sh ./get-docker.sh

DRY_RUN=1 is an optional environment value that can be used to check the steps the installation would perform. Use it if you want to see what would happen, then run without to actually install Docker-CE.

Note: WSL distributions are containers (but not docker containers!) themselves and therefor do not boot and have no systemd available. There are many hacky scripts that allow to enable it (but in my experience break something else) - on Win11 with the Ubuntu 22.04 distribution it is possible to configure systemd.

Without systemd you need to take care yourself that the docker daemon is started inside the distribution.

I still have no idea why you put so much importance in accessing the container by it’s container ip. It is a massive antipattern. I wrote twice that it is not the way to go, and still it seems to be your main goal… I am curious why. Could you elaborate why you feel this is important and the right way to do whatever you try to do?

[dockerzung]

Great thank you for being patient for a person new to this Docker business. I have a better understand of my setup now. The current issue was something from my lack of knowledge. I was trying to install vaultwarden and caddy in WSL2. I ran into some Caddy errors …

2023-01-22 01:23:57 {"level":"info","ts":1674368637.1999962,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["new.vfor27.duckdns.org"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"zung102@yahoo.com"}
2023-01-22 01:23:57 {"level":"info","ts":1674368637.9448936,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"new.vfor27.duckdns.org","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
2023-01-22 01:23:57 {"level":"error","ts":1674368637.9885242,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"new.vfor27.duckdns.org","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.new.vfor27.duckdns.org\" (usually OK if presenting also failed)"}
2023-01-22 01:23:58 {"level":"error","ts":1674368638.0773911,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"new.vfor27.duckdns.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[new.vfor27.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.new.vfor27.duckdns.org\": could not find the start of authority for _acme-challenge.new.vfor27.duckdns.org.: NOERROR (order=https://acme-v02.api.letsencrypt.org/acme/order/914577057/160288502417) (ca=https://acme-v02.api.letsencrypt.org/directory)"}

For some reasons I was trying to curl/telnet to Docker network to find out if Vaultwarden web page was running …As you said this is not possible for my settings.

After further investigation I have found out that the error from Caddy log was due to the fact that Caddy was running as a Docker container in WSL (somehow the Duckdns token was not right). If I ran Caddy directly with the same token it worked fine. The same was true if I ran Caddy Docker as a container in a standalone Ubuntu PC.
I was thinking maybe the Docker used in WSL has some problem so I will attempt to disable the integration of the Dccker Desktop to Ubuntu 20.04. I then installed my own Docker then will install Vaultwarden, Caddy and try it again. I was following the instructions here Installing Vaultwarden formally bitwarden_rs on Ubuntu 20.04 with Nginx · - https://www.llewellynhughes.co.uk/.

However
Set Docker to start on startup

sudo systemctl enable --now docker
 sudo systemctl enable --now docker
[sudo] password for zung:
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

You are saying now I have to make Docker daemon started within WSL ? I still can issue docker --version but maybe it is from Docker Desktop and not from my newly installed Docker? Would installing Docker using your instructions make any difference or I still need to configure systemd ?

[dockerzung]

Thank you very much for taking time to explain how WSL+Ubuntu+Docker work together. As a newbie on WSL and Docker it is really valuable to me.

I was trying to install Vaultwarden with Caddy revers-proxy in my WSL2 environment. Caddy log had showed some errors …

2023-01-22 01:23:57 {"level":"info","ts":1674368637.9448936,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"new.vfor27.duckdns.org","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
2023-01-22 01:23:57 {"level":"error","ts":1674368637.9885242,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"new.vfor27.duckdns.org","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.new.vfor27.duckdns.org\" (usually OK if presenting also failed)"}
2023-01-22 01:23:58 {"level":"error","ts":1674368638.0773911,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"new.vfor27.duckdns.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[new.vfor27.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.new.vfor27.duckdns.org\": could not find the start of authority for _acme-challenge.new.vfor27.duckdns.org.: NOERROR (order=https://acme-v02.api.letsencrypt.org/acme/order/914577057/160288502417) (ca=https://acme-v02.api.letsencrypt.org/directory)"}

For some reasons I was trying curl/telnet to Docker network ip of vaultwarden/caddy. As you said this is not possible with my current setup.

After further investigation I have found out that Caddy ran as a docker container in WSL/Ubuntu20.4 was the culprit. If I ran Caddy by command line or as a docker container from a standalone Ubuntu PC it worked fine. It appeared the same Duckdns token was mangled in WSL+Docker Desktop setting

I attempted to disable Docker Desktop from Ubuntu 20.4, to install Docker package and proceed with Vaultwarden and Caddy as before. I was following the instructions here Installing Vaultwarden formally bitwarden_rs on Ubuntu 20.04 with Nginx ·.

At the step to set Docker to start on startup it got this error

System has not been booted with systemd as init system (PID 1). Can’t operate.
Failed to connect to bus: Host is down

I guessed I need to configure systemd for WSL ? can I remove the new Docker and to follow your instructions? Would that help? I still can docker --version but it could be from Docker Desktop and not from my newly installed Docker?

[meyay (Metin Y.)]

There is absolutely no reason to run docker-ce in wsl… You can just use Docker Desktop, publish a port on the vaultwarden container (the same way you already do it for the caddy container) and then access it with a browser from your host using http://localhost:{host port you mapped agains the vaultwarden port}.

If you want, of course you still can go down the docker-ce route. You discovered yourself what I already wrote about systemd: it is not present in a wsl distribution. You will need to start the docker service using sudo service docker start everythime the wsl distro is started the first time (e.g. after a (re)boot of the host or after wsl -t ${distroname} or wsl --shutdown). If you run your own docker-ce in a wsl distribution, then every port the wsl distribution is listening on will be available on localhost of the windows host as well, BUT it will be not available from the host ip, unless you explicitly allow it in the Windows firewall (if you realy want to walk down that route, please use the forum search or google how it’s done)

[dockerzung]

Thank you very much for your help. I have learnt a few things from you.

I have resolve the issue and got vaultwarden and caddy working as expected.

I still use the docker from Docker Desktop. The culprit is the way this docker reads the Caddyfile. The Duckdns token must be surrounded by quote. Using $DOMAIN environment variable was not helping either. The error messages were not clear. This led me to a wild chase.

Again, thank you for your help