Prometheus with docker works on 0.0.0.0:9323 but not with 127.0.0.1 in daemon.json

[rimelek (Ákos Takács)]

To add more explanation, let’s clarify what the documentation talks about and what you are using. I read the documentation now and I think it is a little confusing.

The documentation sometimes assumes you are using Docker Desktop, but sometimes it shares a command that works on Docker CE (Docker Engine), not on Docker Desktop. You opened the topic in the Docker Engine category and wrote that you were using Ubuntu Server, so you will not get what Docker Desktop offers. Which is accessing ports on localhost on the physical host.

When the documentation first suggests setting 127.0.0.1:9323 in the daemoon json, that is for security reasons, but it will not work with Docker CE. It has nothing to do with IPTables. Every container has its own network namespace, so you cannot refer to a loopback IP and expect accessing ports outside the container as the loopback IP always loops back to the same namespace which means the container’s localhost not the host’s localhost.

Then the documentation suggests adding an exra host on Docker CE, but it doesn’t explain that it will not work when you are using a loopback IP in the daemon json. It would work when using 0.0.0.0, which is definitely insecure on an Ubuntu Server. Fortunately, you don’t have to make it listen on all IP addresses, only on an IP address that the container has access to. Which is the gateway IP address for which you added the extra host. You need to find out the IP address of the gateway and that is what @bluepuma77 showed

@bluepuma77

Run docker network inspect bridge to the see the bridge network IP. Then it’s the IP listed in both, maybe 172.17.0.1.

The actual IP can be different based on settings and maybe Docker versions existing already used subnets.
You can also get the IP from the container like this:

docker run --rm -it  --add-host host.docker.internal:host-gateway curlimages/curl cat /etc/hosts | grep host\.docker\.internal

I actually got this in one of my VMs:

172.16.0.1	host.docker.internal

So you can make the service listen on that IP.

{
        "metrics-addr": "172.16.0.1:9323",
        "experimental" : true
}

But check what the IP is on your machine.

@rpduarte

if i install ping i see that can ping host.docker.internal, and if i get a quick server going in my localhost

@rpduarte

Adding extra hosts to a container with config i have above gives me access to my localhost from the container, i managed to prove this by running a naff python server outside of docker on 8080 and curl it from inside Ubuntu.

I assume you use “localhost” to refer to the physical host machine. But from a network perspective localhost is always the loopback IP address 127.0.0.1 which you cannot access without Docker Desktop or without a little trick forwarding TCP and unix sockets.

Host.docker.internal in production environment

Once again a question that inspired me to try something. Actually I have been thinking for a while about trying to implement a feature like the one in Docker desktop that lets you use host.docker.internal to connect to the localhost of the host machine. I think I have done it. So host.docker.internal is not just a hostname that directly points to the IP address of your host. It will point to the “host gateway” which can be configured but it is 172.17.0.1 by default which is the gateway of the d…

So that is why you could ping host.docker.internal, but the ping command must have returned the gateway IP as well

docker run --rm -it --net hook_default --add-host host.docker.internal:host-gateway curlimages/curl ping host.docker.internal

Output

PING host.docker.internal (172.16.0.1) 56(84) bytes of data.
64 bytes from host.docker.internal (172.16.0.1): icmp_seq=1 ttl=64 time=0.173 ms
64 bytes from host.docker.internal (172.16.0.1): icmp_seq=2 ttl=64 time=0.171 ms

@rpduarte

Also my container doesn’t run on the default bridge it runs on a compose created bridge network. (I think there are limitations with this approach and default bridge network).

It doesn’t matter. The host-gateway is an IP that you can use to access a service running on the host if it is listening on that IP address. It will be accessible from all networks by default. Its default value is the IP of the default docker bridge, but as I mention in the linked topic, it can be changed passing --host-gateway-ip to the Docker daemon which you will not need in this case..