Docker-compose connection refused between containers on same network

General
1

I’ve started building a docker environment for prometheus a few exporters, and grafana. Once configured and composed up, I notice that prometheus is getting connection refused from multiple exporters. Since I am new to both docker and prometheus, I’ve probably done something unintelligent, but for the life of me, I simply can’t find where I messed it up. I hope some of you can shed light on my screw-up?

The docker-compose.yml

```yml

version: '3.3'

volumes:
    prometheus_data: {}
    grafana_data: {}

networks:
  public:
    driver: bridge
    external: true
  name: pub-net
  labels:
    description: "front-end / public network"
  private:
    driver: bridge
    name: priv-net
    labels:
      description: "back-end / private network"
      purpose: "monitoring"

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    restart: always
    ports:
      - '9090:9090'
    networks:
      - private
    external_links:
      - github
      - node-exporter
      - blackbox

  github:
    tty: true
    stdin_open: true
    build:
      context: .
      dockerfile: ./github/dockerfile
    image: infinityworks/github-exporter
    expose:
      - 9171
    ports:
      - '9171:9171'
    restart: unless-stopped
    environment:
      - REPOS=${github_csv_repo_list}
      - USERS=${github_csv_user_list}
      - GITHUB_TOKEN=${github_personal_access_token} 
    working_dir: /go/src/github.com/infinityworks/github-exporter
    networks:
      - private

  blackbox:
    tty: true
    stdin_open: true
    expose:
      - 9115
    ports:
      - '9115:9115'
    image: prom/blackbox-exporter
    networks:
      - private

  node-exporter:
    privileged: true
    image: prom/node-exporter
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    ports:
      - '9102:9102'
    restart: always
    networks:
      - private```

And here’s the output from docker network inspect

[
    {
        "Name": "dan-prom_private",
        "Id": "9477062a0b46df149225e8bda2176ab5af28b5cee61723a57b155c4fbac3906f",
        "Created": "2020-04-16T13:32:38.900816634-04:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.192.0/20",
                    "Gateway": "192.168.192.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "08937535fd86dab1681f13f31a93b842a8bb778fe4f674198b50d765f257231d": {
                "Name": "dan-prom_blackbox_1",
                "EndpointID": "045fb7834be012c9573589a86fcca7e5d5b05ea25a2a3c2e316bbf0674ee307e",
                "MacAddress": "02:42:c0:a8:c0:07",
                "IPv4Address": "192.168.192.7/20",
                "IPv6Address": ""
            },
            "279d06baa5b73f5de3133435ee93019b81766726b16a3031809d105010d31221": {
                "Name": "dan-prom_node-exporter_1",
                "EndpointID": "08fac68cac44b17b590e8faed2640c5d6bbf489109829356dad500bd9d3b1a1d",
                "MacAddress": "02:42:c0:a8:c0:06",
                "IPv4Address": "192.168.192.6/20",
                "IPv6Address": ""
            },
            "7d544d65afd51ff466a433dfe73bb2ce4061cb54a1b4c43d26d56c9bbd78914b": {
                "Name": "dan-prom_prometheus_1",
                "EndpointID": "875fe15c4ccb28c089025ba053bd91d63ac878a0e277221ca516d093f65f14a2",
                "MacAddress": "02:42:c0:a8:c0:03",
                "IPv4Address": "192.168.192.3/20",
                "IPv6Address": ""
            },
            "895287c8307e6a1351321652fa87d243a0bc7b0d255cba0252f365fe0cbbe680": {
                "Name": "dan-prom_github_1",
                "EndpointID": "44686a51d8675dd93828cecfb2cd9e89e442de5d4e173a862886130df3aceeb3",
                "MacAddress": "02:42:c0:a8:c0:05",
                "IPv4Address": "192.168.192.5/20",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "private",
            "com.docker.compose.project": "dan-prom",
            "com.docker.compose.version": "1.21.2",
            "description": "back-end / private network",
            "purpose": "monitoring"
        }
    }
]

And here’s what I see in Prometheus -

image
image1606×331 53.9 KB

Again, I would appreciate any help understanding why I can’t get these to communicate.

Thank you in advance!

2

You havent shown the prometheus configuration file (luckily, I can guess a lot from the prometheus sources screenshot)
From your docker network inspect, your exporter are available at :

  • dan-prom_blackbox_1:9115
  • dan-prom_node-exporter_1:9102
  • dan-prom_github_1:9171

That’s what prometheus have to collect.
A few sides notes :

  • If you dont plan on querying the exporter manually, then there’s no point at exporting these ports outside of the “dan-prom_private” network (remove the “expose:” and “ports:” sections of your docker-compose file for these services)
  • node-exporter doesnt need to be “privileged”
  • I dont know the “blackbox” or the “github” exporter, But my guess is that both doesnt need a TTY nor an open stdin.
4

The answer is to delete all network configuration and simply use network_mode: host on your sources. Apparently, prometheus can’t communicate very well inside a user defined network…
Here’s a snip of the new format:

blackbox:
    image: prom/blackbox-exporter
    container_name: blackbox
    volumes:
      - ./blackbox/config:/config
    command:
      - '--config.file=/config/blackbox.yml'
    network_mode: host

Also, I’d responded a day or so ago, but the admins here don’t seem to review the flagged posts.