Issue with Filebeat service does not run inside container with systemd during docker run

General
1

Hi I have issue with Filebeat service does not run inside container with systemd during docker run.

I want to create a container with systemd init process as PID 1 and
filebeat service should be run as a child to PID 1. The filebeat service
is starting through python script and that would be executed by docker
ENTRYPOINT.

The container is running with /usr/sbin/init process as PID 1 but it is
unable to start the filebeat service, but if i execute the same python
script manually inside the container, filebeat service is running
Steps to reproduce the issue:
docker build -t filebeat_img .docker run -itd --privileged --name filebeat_cont filebeat_imgdocker exec -it filebeat_cont bashps -ef
Describe the results you received:

UID PID PPID C STIME TTY TIME CMD

root 1 0 0 12:55 ? 00:00:00 /usr/sbin/init

root 49 1 0 12:55 ? 00:00:00 /usr/lib/systemd/systemd-journald

root 61 1 1 12:55 ? 00:00:00 /usr/lib/systemd/systemd-udevd

dbus 88 1 0 12:55 ? 00:00:00 /usr/bin/dbus-daemon
–system --address=systemd: --nofork --nopidfile --systemd-activation

root 89 1 0 12:55 ? 00:00:00 /usr/lib/systemd/systemd-logind

root 93 1 0 12:55 tty1 00:00:00 /sbin/agetty --noclear tty1 linux

root 94 1 0 12:55 console 00:00:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 linux

root 108 0 0 12:55 pts/1 00:00:00 bash

root 122 108 0 12:55 pts/1 00:00:00 ps -ef
Describe the results you expected:

UID PID PPID C STIME TTY TIME CMD

root 1 0 0 12:55 ? 00:00:00 /usr/sbin/init

root 49 1 0 12:55 ? 00:00:00 /usr/lib/systemd/systemd-journald

root 61 1 0 12:55 ? 00:00:00 /usr/lib/systemd/systemd-udevd

dbus 88 1 0 12:55 ? 00:00:00 /usr/bin/dbus-daemon
–system --address=systemd: --nofork --nopidfile --systemd-activation

root 89 1 0 12:55 ? 00:00:00 /usr/lib/systemd/systemd-logind

root 93 1 0 12:55 tty1 00:00:00 /sbin/agetty --noclear tty1 linux

root 94 1 0 12:55 console 00:00:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 linux

root 108 0 0 12:55 pts/1 00:00:00 bash

root 142 1 0 12:56 ? 00:00:00
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml
-path.home /usr/share/filebeat -path.config /etc/filebeat

root 149 108 0 12:56 pts/1 00:00:00 ps -ef
Output of docker version:
[root@localhost filebeat_test]# docker -v

Docker version 18.03.0-ce, build 0520e24
Output of docker info:
[root@localhost filebeat_test]# docker info

Containers: 10

Running: 7

Paused: 0

Stopped: 3

Images: 483

Server Version: 18.03.0-ce

Storage Driver: overlay2

Backing Filesystem: xfs

Supports d_type: true

Native Overlay Diff: true

Logging Driver: json-file

Cgroup Driver: cgroupfs

Plugins:

Volume: local

Network: bridge host macvlan null overlay

Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog

Swarm: inactive

Runtimes: runc

Default Runtime: runc

Init Binary: docker-init

containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c

runc version: 4fc53a81fb7c994640722ac585fa9ca548971871

init version: 949e6fa

Security Options:

seccomp

Profile: default

Kernel Version: 3.10.0-693.el7.x86_64

Operating System: CentOS Linux 7 (Core)

OSType: linux

Architecture: x86_64

CPUs: 2

Total Memory: 3.859GiB

Name: localhost.localdomain

ID: QSFU:ODKS:LJGZ:GC34:KXTP:6B7Y:5UMB:Q7WT:V2X3:4K6M:DFLQ:I7WS

Docker Root Dir: /var/lib/docker

Debug Mode (client): false

Debug Mode (server): false

Registry: https://index.docker.io/v1/

Labels:

Experimental: false

Insecure Registries:

127.0.0.0/8

Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled

WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):
Running Docker in a VirtualBox

2

Is your systemd working?
What does systemctl status output (Is your systemd service up and running)?
Can you provide a link to your git repository? Or post your Dockerfile here, too.

If you don’t want to run your container wit --privileged, use the latest docker version and write your Dockerfile like this: https://chaos.expert/agowa338/fedora-systemd-dockerized/blob/dd8621473b625118399100f1b6c920aaf21dfff7/Dockerfile
And execute it with docker run -itv /sys/fs/cgroup:/sys/fs/cgroup:ro agowa338/fedora-systemd /sbin/init

3

Thanks for the response, Iam trying to run filebeat from python.
When i log in to container with just init and run my python script it works fine

Docker:

############################################################
# Dockerfile to nginx and filebeat services
# Based on centos
############################################################

# Set the base image to centos
FROM centos:centos7

# File Author / Maintainer
MAINTAINER TestName

# Update the repository
RUN yum -y update; yum clean all && \
    yum -y install initscripts && yum clean all && \
    yum -y install systemd; yum clean all && \
# Add the EPEL-Release yum repository, which will have NGINX for us
    yum -y install epel-release && \
    mkdir -p /etc/nginx/logs/
# Sets the working directory
WORKDIR filebeat

# Copy filebeat rpm package
ADD filebeat-6.2.3-x86_64.rpm .

#Install filebeat rpm Package
RUN rpm -Uvh --nodeps filebeat-6.2.3-x86_64.rpm

COPY fcgi-server-access-1.log /etc/nginx/logs/
COPY filebeat.yml /etc/filebeat/
COPY start_filebeat.py .

# Environment variable to enable systemd for docker container
ENV container docker

# Creates a mount point for systemd
VOLUME [ "/sys/fs/cgroup" ]

# Run command to enable systemd
ENTRYPOINT python start_filebeat.py && exec /usr/sbin/init

start_filebeat.py
#!/usr/bin/python

import subprocess
import os
command = "service filebeat start"
#comm = subprocess.Popen(command, stdout=subprocess.PIPE,
#                        shell=True, stderr=subprocess.PIPE)
os.system(command)
4

Can you please repost your Dockerfile by using pre-formating (or pastebin)? It is currently very hard to read.

5

I have edited my post and marked “docker file” contents and “start_filebeat.py” with pre-formating. Please provide your inputs.

6

Ok, now were coming closer. You have two issues.
The first one is your systemd stays in ‘degraded’ state after the container is started, as some services fail to start.
For this add the following to your Dockerfile:

RUN systemctl disable console-getty.service && \
    systemctl disable systemd-logind.service && \
    systemctl mask console-getty.service && \
    systemctl mask systemd-logind.service && \
    rm -f /etc/rc.d/init.d/network /run/systemd/generator.late/network.service
VOLUME [ "/sys/fs/cgroup", "/run", "/tmp" ]

the other is an ordering issue in your ENTRYPOINT statement. You try to run a shell command namely service before your init system is up.
This can be fixed by also adding these to your Dockerfile:

RUN systemctl enable filebeat
ENTRYPOINT /usr/sbin/init
7

Thanks for the responses.
With your suggestions systemd is enabled properly and filebeat starts fine.
Wanted to generate the filebeat configuration from python scrip and run filebeat from python script so i ma using python and init as entry point. For now filebeat starts automatically, from python filebeat configuration is updated and filebeat is started again with new configuration. Will check if i need to do this and will post back if more help is required.
Thank you for the help.

8

I just want to note this command in your ps output:

/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat

If you make that the CMD of your container, you won’t need to run init, run the container in privileged mode, mount the host cgroups into the container, or manually update any other systemd configuration.

9

WithRUN systemctl enable filebeat filebeat gets automatically inside container.
Using python script i had updated my configurations and restarting filebeat(stop & start or restart). Filebeat restarts with new configuration.

Facing issue with docker stop and start.
Unable to start the docker container which was stopped(with docker stop).
Container fails to start as it fails to start Filebeat with following info.
Reloading systemd: Failed to get D-Bus connection: Connection refused
[FAILED]
How to start the container with docker start with privileges to star the filebeat.
Please find below docker logs for stop and start.

Docker logs info for docker stop

[  OK  ] Stopped target Timers.
[  OK  ] Removed slice system-getty.slice.
[  OK  ] Stopped target Multi-User System.
         Stopping D-Bus System Message Bus...
[  OK  ] Stopped target Login Prompts.
         Stopping Permit User Sessions...
         Stopping filebeat...
[  OK  ] Stopped D-Bus System Message Bus.
[  OK  ] Stopped Permit User Sessions.
[  OK  ] Stopped target Remote File Systems.
[  OK  ] Stopped filebeat.
[  OK  ] Stopped target Network is Online.
[  OK  ] Stopped target Basic System.
[  OK  ] Stopped target Paths.
[  OK  ] Stopped target Sockets.
[  OK  ] Closed D-Bus System Message Bus Socket.
[  OK  ] Stopped target System Initialization.
         Stopping Load/Save Random Seed...
         Stopping Update UTMP about System Boot/Shutdown...
[  OK  ] Stopped Read and set NIS domainname from /etc/sysconfig/network.
         Stopping Read and set NIS domainname from /etc/sysconfig/network...
[  OK  ] Stopped target Local Encrypted Volumes.
[  OK  ] Stopped Update is Completed.
         Stopping Update is Completed...
[  OK  ] Stopped Rebuild Hardware Database.
         Stopping Rebuild Hardware Database...
[  OK  ] Stopped Rebuild Journal Catalog.
         Stopping Rebuild Journal Catalog...
[  OK  ] Stopped target Slices.
[  OK  ] Stopped Load/Save Random Seed.
[  OK  ] Stopped Update UTMP about System Boot/Shutdown.
[  OK  ] Stopped Create Volatile Files and Directories.
         Stopping Create Volatile Files and Directories...
[  OK  ] Stopped target Local File Systems.
         Unmounting Temporary Directory...
         Unmounting /proc/bus...
         Unmounting /proc/timer_stats...
         Unmounting /proc/asound...
         Unmounting /proc/keys...
         Unmounting /proc/scsi...
         Unmounting /proc/timer_list...
         Unmounting /proc/kcore...
         Unmounting /etc/hosts...
         Unmounting /proc/fs...
         Unmounting /proc/sysrq-trigger...
         Unmounting /etc/hostname...
         Unmounting /proc/irq...
         Unmounting /proc/sched_debug...
         Unmounting /sys/firmware...
         Unmounting /etc/resolv.conf...
[  OK  ] Stopped Configure read-only root support.
         Stopping Configure read-only root support...
[  OK  ] Reached target Shutdown.
Sending SIGTERM to remaining processes...

Docker logs info for docker start

Executing command: service filebeat restart
Reloading systemd:  Failed to get D-Bus connection: Connection refused
                                                           [FAILED]
Restarting filebeat (via systemctl):  Failed to get D-Bus connection: Connection refused
                                                           [FAILED]