I have a Python REST API server built with FastAPI. Upon response, it spawns a ffmpeg sub-process which is used to capture a video from an RTSP stream provided by an IP camera. Upon another request, the app stops recording the video by sending a SIGTERM signal to the ffmpeg process.
This works just fine outside of Docker, but in a container, the SIGTERM is ignored by the ffmpeg process, thus the video never stops recording. I tried adding the --init option to my container but it didn’t help although my app didn’t have PID 1 anymore.
My Dockerfile:
# convert poetry's pyproject.toml into a requirements.txt file
FROM python:3.9 as requirements-stage
WORKDIR /tmp
RUN pip install poetry
COPY ./pyproject.toml ./poetry.lock* /tmp/
RUN poetry export -f requirements.txt --output requirements.txt --without-hashes
# actual app deployment
FROM python:3.9
RUN apt update && apt install -y usbutils zlib1g libjpeg-dev ffmpeg
WORKDIR /
COPY --from=requirements-stage /tmp/requirements.txt /requirements.txt
RUN pip install --no-cache-dir --upgrade -r /requirements.txt
COPY . /
ENV IS_DOCKERIZED true
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
Is the version of ffmpeg the same as the one on the host? If a process handles signals then it handles outside and inside a container. If it doesn’t handle signals then the default action of the signal will happen unless it is PID 1. I read that there was a bug related to the signal handling of ffmpeg. So my next guess is that the version of the ffmpeg inside the container is buggy.
I have checked it and there seems to be a mismatch between versions of ffmpeg on my host and inside the container, however I’m not sure which bug you are talking about. Would you mind linking an issue?
Sorry, I didn’t share the link since I had no idea if this was related but I should have shared this. I searched for “ffmpeg sigterm does not work” and found this:
I have just tried recording manually with ffmpeg inside the container and it has worked just fine. I think the issue is in communicating signals from my app to the ffmpeg process
Just had an idea about the PIDs. May it be that the app records the host PID when spawning a process and tries to send a signal to it instead of a local PID inside the container?
No, this doesn’t seem to be the case. The app spawns a subprocess using sh. I experimented now: server spawns sh process (PID 24) and it spawns an ffmpeg process (PID 26). The app sends a SIGINT to the PID 24, which seems to be right.
It is not possible as you already found out. Without using the host’s pid namespace the process inside the container doesn’t know about the host pid.
seing these pids makes me wonder if the server process could restart ffmpeg immediately after you send a signal so you see it is still running but it in fact it is running again.
I don’t think it is the case. Video files are saved to a docker volume and I can see how the files keep growing until I kill the container. So the stream seems to be uninterrupted from ffmpeg to the file meaning that sh ignores the signal not relaying it to the ffmpeg process. I still don’t understand, why this only happens inside Docker.
You are probably right, but you can make sure of it by checking the pid before and after trying to stop the process.
What sh you are referring to? If you mean /bin/sh I don’t see that running. If uvicorn has PID 1 then that is responsible for handling signals. If you use the init flag, then “tini” handles the signals. There is no shell which can stop the signals. The only thing I can think of is that uvicorn does somethong differently.
If you are not able to figure it out, I offer my help to test it on my machine if you can give me a test app with the same issue but without your private data and code.
Recently I worked on multiple example projects to demonstrate how signals work with and without Docker containers. I thought I knew everything but I don’t understand this issue. So I would really like to find out what the problem is
Actually that is the PID of the shell which I can see now, thanks. When you wrote
did you send the signal to the shell process?
If you did and you are right, then you should be able terminate ffmpeg by sending the signal to it and not to /bin/sh.
Thank you. This project seems to be larger than I have time for and it is Russian which could be an issue for me. Although, yandex translates it pretty well.
I think I’ve tried sending signals directly to the ffmpeg process, not the shell one.
When I spawn a shell using a library in my code it returns me the PID of the shell. I don’t think I have a reliable way of getting the ffmpeg process PID. Note that there may be many ffmpeg processes running simultaneously.
Only the README is in Russian, everything inside is English. The very code that might be interesting to you is /src/video/camera.py lines 91-116.
I would not have suggested that. I just wanted to know if the shell is the problem. Sorry, I don’t have more ideas. If you can figure it out, please, share the solution. Of course, I will write if I have anything to add to solve this.
Were you ever able to solve this issue? I am running into a similar problem where SIGINT is being ignored by a subprocess spawned via subprocess.Popen only when inside of a docker container
