Prometheus isn’t scrapping cadvisor and node-exporter

General
,
1

Hi All,

Need help

When i click on prometheus URL on target page it showing below message.

cadvisor (0/1 up) and node-exporter (0/1 up) are showing in Prometheus URL

Here is the my filename.yml file

version: '3.2'

services:

  prometheus:
       image: prom/prometheus:latest
#       container_name: monitoring_prometheus
       command:
         - '--config.file=/etc/prometheus/prometheus.yml'
         - '--storage.tsdb.path=/prometheus'
         - '--web.console.libraries=/usr/share/prometheus/console_libraries'
         - '--web.console.templates=/usr/share/prometheus/consoles'
       volumes:
         - /Prometheus/alert.rules:/etc/prometheus/alert.rules
         - /Prometheus/container.yml:/etc/prometheus/container.yml
         - /Prometheus/diskusage.yml:/etc/prometheus/diskusage.yml
         - ./prometheus.yml:/etc/prometheus/prometheus.yml
       ports:
         - 9090:9090

  node-exporter:       
       image: prom/node-exporter:latest
#       container_name: monitoring_node_exporter       
       volumes:       
         - /proc:/host/proc:ro       
         - /sys:/host/sys:ro       
         - /:/rootfs:ro       
       ports:
         - 9091:9091
       command:       
         - '--path.procfs=/host/proc'       
         - '--path.sysfs=/host/sys'       
         - '--path.rootfs=/host'       
         - '--collector.filesystem.ignored-mount-points="^(/rootfs|/host|)/(sys|proc|dev|host|etc)($|/)"'       
         - '--collector.filesystem.ignored-fs-types="^(sys|proc|auto|cgroup|devpts|ns|au|fuse\.lxc|mqueue)(fs|)$"'       


  cadvisor:
       image: google/cadvisor:latest
#       container_name: monitoring_cadvisor
       ports:
         - 9092:9092
       volumes:
         - /:/rootfs:ro
         - /var/run:/var/run:rw
         - /sys:/sys:ro
         - /var/lib/docker/:/var/lib/docker:ro


  grafana:
       image: grafana/grafana:latest
#       image: grafana-custom:latest
#       container_name: monitoring_grafana   
       volumes:
         - grafana-data:/var/lib/grafana
#         - ./data/grafana:/var/lib/grafana
#         - ./grafana/provisioning:/etc/grafana/provisioning
#         - ./defaults.ini:/usr/share/grafana/conf/defaults.ini
       ports:
         - 3000:3000
       environment:
         - "GF_SMTP_ENABLED=true" 
         - "GF_SMTP_HOST=smtp.gmail.com:465" 
         - "GF_SMTP_USER=test@gmail.com" 
         - "GF_SMTP_PASSWORD=xxxxxx" 

  alertmanager:
       image: prom/alertmanager
#        privileged: true
       volumes:
         - ./alertmanager.yml:/alertmanager.yml
       command:
         - '--config.file=/alertmanager.yml'
       ports:
         - '9093:9093'
 
  karma-alert-dashboard:
#       image: cloudflare/unsee
       image: lmierzwa/karma:latest
       environment:
#         - "ALERTMANAGER_URIS=default:http://alertmanager:9093"
         - "ALERTMANAGER_URI=http://alertmanager:9093"
       ports:
         - '9094:9094'


volumes:
  grafana-data:

I have deploy 5 docker service through docker stack -c filename.yml test, All services and docker container are UP.

First 3 are connected together and are working fine

Grafana      =  http://SERVERIP:3000/ - 3000

Prometheus = http://SERVERIP:9090/ - 9090
AlertManager = http://SERVERIP:9093/ - 9093

Last 3 are Not connected together and are Not working fine because of port issue in docker container and in docker service also

Karma        =  http://SERVERIP:9094/ - 8080

Node Exporter= http://SERVERIP:9100/ - 9100
CAdvisor = http://SERVERIP:8080/ - 8080

Here is the Screenshot

image
image1248×385 88.4 KB

When i check ports number 3 services are showing different in terminal but in my filename.yml everything is correct can anyone let me what is the issue and where i’m wrong

When i check logs of 3 docker container

  1. docker logs cadvisor
Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
  1. node-exporter is not showing any logs.
  2. Here is the logs of lmierzwa/karma
level=info msg="Version: v0.86-83-gf5e5269f"
level=info msg="Configured Alertmanager source" name=default proxy=false readonly=false uri=http://alertmanager:9093
level=info msg="Initial Alertmanager collection"
level=info msg="Pulling latest alerts and silences from Alertmanager"
level=info msg="Collecting alerts and silences" alertmanager=default
level=info msg="GET request" timeout=40 uri=http://alertmanager:9093/metrics
level=info msg="Upstream version" alertmanager=default version=0.22.2
level=info msg="Got silences" alertmanager=default duration=0.000204419 silences=0
level=info msg="Detecting ticket links in silences" alertmanager=default silences=0
level=info msg="Collected alert groups" alertmanager=default duration=0.000201856 groups=0
level=info msg="Deduplicating alert groups" alertmanager=default groups=0
level=info msg="Processing deduplicated alert groups" alertmanager=default groups=0
level=info msg="Merging autocomplete hints" alertmanager=default hints=0
level=info msg="Collection completed"
level=info msg="Done, starting HTTP server"
level=info msg="Starting HTTP server" address=[::]:8080
level=info msg="Pulling latest alerts and silences from Alertmanager"
level=info msg="Collecting alerts and silences" alertmanager=default
level=info msg="GET request" timeout=40 uri=http://alertmanager:9093/metrics
level=info msg="Upstream version" alertmanager=default version=0.22.2
level=info msg="Got silences" alertmanager=default duration=0.000155633 silences=0