Filebeat not running using docker-compose: setting 'filebeat.prospectors' has been removed

Docker Hub
1

I’m trying to launch filebeat using docker-compose (I intend to add other services later on) but every time I execute the docker-compose.yml file, the filebeat service always ends up with the following error:

filebeat_1  | 2019-08-01T14:01:02.750Z  ERROR   instance/beat.go:877    Exiting: 1 error: setting 'filebeat.prospectors' has been removed
filebeat_1  | Exiting: 1 error: setting 'filebeat.prospectors' has been removed

I discovered the error by accessing the docker-compose logs.

My docker-compose file is as simple as it can be at the moment. It simply calls a filebeat Dockerfile and launches the service immediately after.

Next to my Dockerfile for filebeat I have a simple config file (filebeat.yml), which is copied to the container, replacing the default filebeat.yml.

If I execute the Dockerfile using the docker command, the filebeat instance works just fine: it uses my config file and identifies the “output.json” file as well.

I’m currently using version 7.2 of filebeat and I know that the “filebeat.prospectors” isn’t being used. I also know for sure that this specific configuration isn’t coming from my filebeat.yml file (you’ll find it below).

It seems that, when using docker-compose, the container is accessing another configuration file instead of the one that is being copied to the container, by the Dockerfile, but so far I haven’t been able to figure it out how, why and how can I fix it…

Here’s my docker-compose.yml file:

version: "3.7"
services:
  filebeat:
    build: "./filebeat"
    command: filebeat -e -strict.perms=false

The filebeat.yml file:

filebeat.inputs:
  - paths:
    - '/usr/share/filebeat/*.json'
    fields_under_root: true
    fields:
      tags: ['json']
output:
  logstash:
    hosts: ['localhost:5044']

The Dockerfile file:

FROM docker.elastic.co/beats/filebeat:7.2.0
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
COPY output.json /usr/share/filebeat/output.json
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
RUN mkdir /usr/share/filebeat/dockerlogs
USER filebeat

The output I’m expecting should be similar to the following, which comes from the successful executions I’m getting when I’m executing it as a single container.

The ERROR is expected because I don’t have logstash configured at the moment.

INFO    crawler/crawler.go:72   Loading Inputs: 1
INFO    log/input.go:148        Configured paths: [/usr/share/filebeat/*.json]
INFO    input/input.go:114      Starting input of type: log; ID: 2772412032856660548
INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
INFO    log/harvester.go:253    Harvester started for file: /usr/share/filebeat/output.json
INFO    pipeline/output.go:95   Connecting to backoff(async(tcp://localhost:5044))
ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [::1]:5044: connect: cannot assign requested address
INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 1 reconnect attempt(s)
ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [::1]:5044: connect: cannot assign requested address
INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 2 reconnect attempt(s)

Any help or advice will be most welcome!

2

I managed to figure out what the problem was. I needed to map the location of the config file and logs directory in the docker-compose file, using the volumes tag:

version: "3.7"
services:
  filebeat:
    build: "./filebeat"
    command: filebeat -e -strict.perms=false
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - ./filebeat/logs:/usr/share/filebeat/dockerlogs

Finally I just had to execute the docker-compose command and everything start working properly:

docker-compose -f docker-compose.yml up -d