Me stuck in a strange phenomenon when using docker and fluentd log driver

kaijietti · June 8, 2023, 10:19am

I will paste a github repo here (GitHub - kaijietti/loop: To reproduce a bug when using docker and fluentd.) which can reproduce this bug. I am not familiar with Docker so i don’t know what’s happening.

kaijietti · June 9, 2023, 11:05am

bump! Need some help.

rimelek · June 9, 2023, 6:11pm

Please describe the actual bug. I don’t expect many people cloning the repository and trying to reproduce the issue which might not happen in different environments. Since you only mentioned the repository can reproduce the issue but not that you described the issue there, it is understandable that you didn’t get the attention of anyone yet. I almost asked for describing the issue too., but even now I don’t completely understand it and I wouldn’t run failing commands and spend time on understanding what is happening.

You mention t0 and t11 but not what that means so we have to find it… I leave this link here for others:

github.com

kaijietti/loop/blob/main/reproduce.sh

# NOTE:
# Must not run this script directly, follow the timeline t0-t11 manually.

# t0 Bootstrap fluentd to collect container logs. [in terminal#1]
docker run -it -p 24224:24224 -v ./test.conf:/fluentd/etc/test.conf -e FLUENTD_CONF=test.conf fluent/fluentd:latest

# t1 Build and Run this program called 'loop' [in another terminal#2]
docker build -t loop:latest .
docker run -it --log-driver=fluentd --log-opt fluentd-address=127.0.0.1:24224 -p 8090:8090 loop:latest

# t5 Now back to terminal#1 and stop the fluentd server

# t6 Observing terminal#2 for about half a minute, you will note that the terminal stop logging 'Hello' message.

# t10 Now try to stop this container or just press CTRL+C in terminal#2, you will note that container is stuck.
# t11 You will fail to stop or rm this container, unless restart docker.service.

Please, share the relevant information directly on the forum next time to make it easier for everyone to help you.

The fact that the loop container doesn’t have any output could be a fluentd driver bug. I don’t know how exactly that driver works, but the container will continue trying to send logs to the stopped fluentd and it is possible that the standard output breaks too.

CTRL+C stops the container only if the process in the container handles the stop signal. I don’t see that in your Go code, but it could be that I just don’t recognize it.

Removing the container means you also need to stop it. If nothing handles the stop signal, Docker normally kills the process in 10 seconds by default but that timeout can be chaanged. If Docker can’t handle that either because it is already in a state where constantly doing something else, that could lead to tthe situation you had, but this is just a guess. Since restarting the Docker daemon helped, You could try to check the daemon logs.

systemctl status docker

and

journalctl -e -u docker