Prometheus with docker works on 0.0.0.0:9323 but not with 127.0.0.1 in daemon.json

[rpduarte]

I am trying to feed prometheus metrics to from docker, on my pi with ubuntu server. Ai have followed the instructions in the following page. However i cant seem to get it working when the dameon.json is set to 127.0.0.1.9323

this is my daemon.json

{
        "metrics-addr": "127.0.0.1:9323",
        "experimental" : true
}

i can get metrics with curl 127.0.0.1:9323/metrics or curl localhost:9323/metrics

i have tried to debug this with an ubuntu image:

services:
     ubuntu:
    image: ubuntu
    entrypoint: 
      - /usr/bin/tail 
      - -f 
      - /dev/null
    extra_hosts:
      - "host.docker.internal:host-gateway"

once inside the image i do apt update && apt install curl -y and curl host.docker.internal:9323/metrics

if i install ping i see that can ping host.docker.internal, and if i get a quick server going in my localhost with python python3 -m http.server 8080 i can do curl host.docker.internal:8080

If i change the daemon.json to 0.0.0.0:9323 it also works but i dont want to to that for obvious reasons.

Feels to me something in iptables isnt passing 9323 so that docker0 can see it. but i dont know enough of iptables to find my way around it.

Could anyone provide me some help please. Feels like one of those things that its gonna be a duh thing on my side.

Regards

[bluepuma77]

I don’t think this works. When the service on host only listens on 127.0.0.1 (localhost), then you can not connect to it from inside a container. The container connects via a bridge network, so you may be able to use the host bridge IP.

Run hostname -I to see all active IPs. Run docker network inspect bridge to the see the bridge network IP. Then it’s the IP listed in both, maybe 172.17.0.1.

[rpduarte]

Thanks for replying, but i am not sure i understand which bit do you say it wont work or why, could you clarify please?
I have followed the instructions in the link in my post. And i cant connect consistently to port 9323 as long the Daemon is pointing to 127.0.0.1

Adding extra hosts to a container with config i have above gives me access to my localhost from the container, i managed to prove this by running a naff python server outside of docker on 8080 and curl it from inside Ubuntu.

That worked well. And my host was available. Also i can ping host.docker.internal from inside my Ubuntu docker too. I can’t however curl that specific port. I am not sure why. Its almost as if docker didn’t create all the iptables for such to happen. (I am obviously speculating about iptables as i dont really understand how do they work).

Also my container doesn’t run on the default bridge it runs on a compose created bridge network. (I think there are limitations with this approach and default bridge network).

[bluepuma77]

I did not read the link.

When you set the daemon on host to use 127.0.0.1:9323, then you can not access it from within container. (Well, except you use network mode host for the container.)

When you run a Python app on host, then it will usually listen on 0.0.0.0, which is all available IPs, then it is reachable from container, via host.docker.internal:host-gateway.

If you don’t want to use 0.0.0.0 on host, then you need to use the gateway IP to listen to, usually 172.17.0.1. That one can be reached from within container.

[rimelek (Ákos Takács)]

To add more explanation, let’s clarify what the documentation talks about and what you are using. I read the documentation now and I think it is a little confusing.

The documentation sometimes assumes you are using Docker Desktop, but sometimes it shares a command that works on Docker CE (Docker Engine), not on Docker Desktop. You opened the topic in the Docker Engine category and wrote that you were using Ubuntu Server, so you will not get what Docker Desktop offers. Which is accessing ports on localhost on the physical host.

When the documentation first suggests setting 127.0.0.1:9323 in the daemoon json, that is for security reasons, but it will not work with Docker CE. It has nothing to do with IPTables. Every container has its own network namespace, so you cannot refer to a loopback IP and expect accessing ports outside the container as the loopback IP always loops back to the same namespace which means the container’s localhost not the host’s localhost.

Then the documentation suggests adding an exra host on Docker CE, but it doesn’t explain that it will not work when you are using a loopback IP in the daemon json. It would work when using 0.0.0.0, which is definitely insecure on an Ubuntu Server. Fortunately, you don’t have to make it listen on all IP addresses, only on an IP address that the container has access to. Which is the gateway IP address for which you added the extra host. You need to find out the IP address of the gateway and that is what @bluepuma77 showed

@bluepuma77

Run docker network inspect bridge to the see the bridge network IP. Then it’s the IP listed in both, maybe 172.17.0.1.

The actual IP can be different based on settings and maybe Docker versions existing already used subnets.
You can also get the IP from the container like this:

docker run --rm -it  --add-host host.docker.internal:host-gateway curlimages/curl cat /etc/hosts | grep host\.docker\.internal

I actually got this in one of my VMs:

172.16.0.1	host.docker.internal

So you can make the service listen on that IP.

{
        "metrics-addr": "172.16.0.1:9323",
        "experimental" : true
}

But check what the IP is on your machine.

@rpduarte

if i install ping i see that can ping host.docker.internal, and if i get a quick server going in my localhost

@rpduarte

Adding extra hosts to a container with config i have above gives me access to my localhost from the container, i managed to prove this by running a naff python server outside of docker on 8080 and curl it from inside Ubuntu.

I assume you use “localhost” to refer to the physical host machine. But from a network perspective localhost is always the loopback IP address 127.0.0.1 which you cannot access without Docker Desktop or without a little trick forwarding TCP and unix sockets.

Host.docker.internal in production environment

Once again a question that inspired me to try something. Actually I have been thinking for a while about trying to implement a feature like the one in Docker desktop that lets you use host.docker.internal to connect to the localhost of the host machine. I think I have done it. So host.docker.internal is not just a hostname that directly points to the IP address of your host. It will point to the “host gateway” which can be configured but it is 172.17.0.1 by default which is the gateway of the d…

So that is why you could ping host.docker.internal, but the ping command must have returned the gateway IP as well

docker run --rm -it --net hook_default --add-host host.docker.internal:host-gateway curlimages/curl ping host.docker.internal

Output

PING host.docker.internal (172.16.0.1) 56(84) bytes of data.
64 bytes from host.docker.internal (172.16.0.1): icmp_seq=1 ttl=64 time=0.173 ms
64 bytes from host.docker.internal (172.16.0.1): icmp_seq=2 ttl=64 time=0.171 ms

@rpduarte

Also my container doesn’t run on the default bridge it runs on a compose created bridge network. (I think there are limitations with this approach and default bridge network).

It doesn’t matter. The host-gateway is an IP that you can use to access a service running on the host if it is listening on that IP address. It will be accessible from all networks by default. Its default value is the IP of the default docker bridge, but as I mention in the linked topic, it can be changed passing --host-gateway-ip to the Docker daemon which you will not need in this case..

[rpduarte]

Thank you @rimelek and @bluepuma77.

I should have noticed the ip of the hostname now yes it points to docker0 interface. I assumed that it would all end up on the loopback and that’s why the docs pointed at 127.0.0.1. changing the daemon to point to the docker0 IP worked perfectly.

Mystery solved, thank you both.