I’m trying to launch filebeat using docker-compose (I intend to add other services later on) but every time I execute the docker-compose.yml file, the filebeat service always ends up with the following error:
filebeat_1 | 2019-08-01T14:01:02.750Z ERROR instance/beat.go:877 Exiting: 1 error: setting 'filebeat.prospectors' has been removed filebeat_1 | Exiting: 1 error: setting 'filebeat.prospectors' has been removed
I discovered the error by accessing the docker-compose logs.
My docker-compose file is as simple as it can be at the moment. It simply calls a filebeat Dockerfile and launches the service immediately after.
Next to my Dockerfile for filebeat I have a simple config file (filebeat.yml), which is copied to the container, replacing the default filebeat.yml.
If I execute the Dockerfile using the docker command, the filebeat instance works just fine: it uses my config file and identifies the “output.json” file as well.
I’m currently using version 7.2 of filebeat and I know that the “filebeat.prospectors” isn’t being used. I also know for sure that this specific configuration isn’t coming from my filebeat.yml file (you’ll find it below).
It seems that, when using docker-compose, the container is accessing another configuration file instead of the one that is being copied to the container, by the Dockerfile, but so far I haven’t been able to figure it out how, why and how can I fix it…
Here’s my docker-compose.yml file:
version: "3.7" services: filebeat: build: "./filebeat" command: filebeat -e -strict.perms=false
The filebeat.yml file:
filebeat.inputs: - paths: - '/usr/share/filebeat/*.json' fields_under_root: true fields: tags: ['json'] output: logstash: hosts: ['localhost:5044']
The Dockerfile file:
FROM docker.elastic.co/beats/filebeat:7.2.0 COPY filebeat.yml /usr/share/filebeat/filebeat.yml COPY output.json /usr/share/filebeat/output.json USER root RUN chown root:filebeat /usr/share/filebeat/filebeat.yml RUN mkdir /usr/share/filebeat/dockerlogs USER filebeat
The output I’m expecting should be similar to the following, which comes from the successful executions I’m getting when I’m executing it as a single container.
The ERROR is expected because I don’t have logstash configured at the moment.
INFO crawler/crawler.go:72 Loading Inputs: 1 INFO log/input.go:148 Configured paths: [/usr/share/filebeat/*.json] INFO input/input.go:114 Starting input of type: log; ID: 2772412032856660548 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1 INFO log/harvester.go:253 Harvester started for file: /usr/share/filebeat/output.json INFO pipeline/output.go:95 Connecting to backoff(async(tcp://localhost:5044)) ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [::1]:5044: connect: cannot assign requested address INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 1 reconnect attempt(s) ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [::1]:5044: connect: cannot assign requested address INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 2 reconnect attempt(s)
Any help or advice will be most welcome!