Docker Community Forums

Share and learn in the Docker community.

Signal sent to the child process from a parent process is ignored inside a docker container

I have a Python REST API server built with FastAPI. Upon response, it spawns a ffmpeg sub-process which is used to capture a video from an RTSP stream provided by an IP camera. Upon another request, the app stops recording the video by sending a SIGTERM signal to the ffmpeg process.

This works just fine outside of Docker, but in a container, the SIGTERM is ignored by the ffmpeg process, thus the video never stops recording. I tried adding the --init option to my container but it didn’t help although my app didn’t have PID 1 anymore.

My Dockerfile:

# convert poetry's pyproject.toml into a requirements.txt file
FROM python:3.9 as requirements-stage
WORKDIR /tmp
RUN pip install poetry
COPY ./pyproject.toml ./poetry.lock* /tmp/
RUN poetry export -f requirements.txt --output requirements.txt --without-hashes

# actual app deployment
FROM python:3.9
RUN apt update && apt install -y usbutils zlib1g libjpeg-dev ffmpeg
WORKDIR /
COPY --from=requirements-stage /tmp/requirements.txt /requirements.txt
RUN pip install --no-cache-dir --upgrade -r /requirements.txt
COPY . /
ENV IS_DOCKERIZED true
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

My docker-compose.yml:

version: "3.7"
services:
  ipfs_node:
    image: ipfs/go-ipfs:v0.8.0
    volumes:
      - ~/ipfs/ipfs_staging:/export
      - ~/ipfs/ipfs_data:/data/ipfs
    ports:
      - "5001:5001"
      - "8080:8080"
      - "4001:4001"

  feecc_io_gateway:
    build:
      context: ./
      dockerfile: Dockerfile
    init: true
    network_mode: "host"
    volumes:
      - "./output:/output"
      - "/dev/usb:/dev/usb"
      - "/etc/timezone:/etc/timezone:ro"
      - "/etc/localtime:/etc/localtime:ro"

What exactly sends that signal? Does it come from outside or inside the same container?

The signal is sent by the main process - the Python web-server.

It comes from the inside. Both processes - the ffmpeg one and the web-server are inside the same container.

Can you send the signal manually from the container to try if that works?

I tried - it didn’t stop the process

Is the version of ffmpeg the same as the one on the host? If a process handles signals then it handles outside and inside a container. If it doesn’t handle signals then the default action of the signal will happen unless it is PID 1. I read that there was a bug related to the signal handling of ffmpeg. So my next guess is that the version of the ffmpeg inside the container is buggy.

I have checked it and there seems to be a mismatch between versions of ffmpeg on my host and inside the container, however I’m not sure which bug you are talking about. Would you mind linking an issue?

Sorry, I didn’t share the link since I had no idea if this was related but I should have shared this. I searched for “ffmpeg sigterm does not work” and found this:

https://trac.ffmpeg.org/ticket/9009

I have just tried recording manually with ffmpeg inside the container and it has worked just fine. I think the issue is in communicating signals from my app to the ffmpeg process

Just had an idea about the PIDs. May it be that the app records the host PID when spawning a process and tries to send a signal to it instead of a local PID inside the container?

No, this doesn’t seem to be the case. The app spawns a subprocess using sh. I experimented now: server spawns sh process (PID 24) and it spawns an ffmpeg process (PID 26). The app sends a SIGINT to the PID 24, which seems to be right.

It is not possible as you already found out. Without using the host’s pid namespace the process inside the container doesn’t know about the host pid.

seing these pids makes me wonder if the server process could restart ffmpeg immediately after you send a signal so you see it is still running but it in fact it is running again.

I don’t think it is the case. Video files are saved to a docker volume and I can see how the files keep growing until I kill the container. So the stream seems to be uninterrupted from ffmpeg to the file meaning that sh ignores the signal not relaying it to the ffmpeg process. I still don’t understand, why this only happens inside Docker.

You are probably right, but you can make sure of it by checking the pid before and after trying to stop the process.

What sh you are referring to? If you mean /bin/sh I don’t see that running. If uvicorn has PID 1 then that is responsible for handling signals. If you use the init flag, then “tini” handles the signals. There is no shell which can stop the signals. The only thing I can think of is that uvicorn does somethong differently.

If you are not able to figure it out, I offer my help to test it on my machine if you can give me a test app with the same issue but without your private data and code.

Recently I worked on multiple example projects to demonstrate how signals work with and without Docker containers. I thought I knew everything but I don’t understand this issue. So I would really like to find out what the problem is :smiley:

I’ve added PID logging to my code. The PID remains the same - 25.

Here’s the output of ps -x --forest

  1. Right after start
# ps -x --forest
    PID TTY      STAT   TIME COMMAND
      8 pts/0    Ss     0:00 bash
     15 pts/0    R+     0:00  \_ ps -x --forest
      1 ?        Ss     0:00 /sbin/docker-init -- uvicorn app:app --host 0.0.0.0 --port 8000
      7 ?        S      0:00 /usr/local/bin/python /usr/local/bin/uvicorn app:app --host 0.0.0.0 --port 8000

  1. After sending a request to start recording the video (I’m using a sample demo RTSP stream here instead of a real IP camera)
# ps -x --forest
    PID TTY      STAT   TIME COMMAND
      8 pts/0    Ss     0:00 bash
     28 pts/0    R+     0:00  \_ ps -x --forest
      1 ?        Ss     0:00 /sbin/docker-init -- uvicorn app:app --host 0.0.0.0 --port 8000
      7 ?        Sl     0:00 /usr/local/bin/python /usr/local/bin/uvicorn app:app --host 0.0.0.0 --port 8000
     25 ?        S      0:00  \_ /bin/sh -c ffmpeg -rtsp_transport tcp -i "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4" -r 25 -c copy -map 0 output/video/f5ba1c43
     27 ?        SL     0:00      \_ ffmpeg -rtsp_transport tcp -i rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4 -r 25 -c copy -map 0 output/video/f5ba1c432d0e497f8
  1. After sending a SIGINT to ffmpeg
# ps -x --forest
    PID TTY      STAT   TIME COMMAND
      8 pts/0    Ss     0:00 bash
     29 pts/0    R+     0:00  \_ ps -x --forest
      1 ?        Ss     0:00 /sbin/docker-init -- uvicorn app:app --host 0.0.0.0 --port 8000
      7 ?        Sl     0:00 /usr/local/bin/python /usr/local/bin/uvicorn app:app --host 0.0.0.0 --port 8000
     25 ?        S      0:00  \_ /bin/sh -c ffmpeg -rtsp_transport tcp -i "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4" -r 25 -c copy -map 0 output/video/f5ba1c43
     27 ?        SL     0:00      \_ ffmpeg -rtsp_transport tcp -i rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4 -r 25 -c copy -map 0 output/video/f5ba1c432d0e497f8

The project is open source, I think you can try it out. Get it from GitHub. However, I don’t think you will be able to use it without a valid DB.

Actually that is the PID of the shell which I can see now, thanks. When you wrote

did you send the signal to the shell process?

If you did and you are right, then you should be able terminate ffmpeg by sending the signal to it and not to /bin/sh.

Thank you. This project seems to be larger than I have time for and it is Russian which could be an issue for me. Although, yandex translates it pretty well.

I think I’ve tried sending signals directly to the ffmpeg process, not the shell one.

When I spawn a shell using a library in my code it returns me the PID of the shell. I don’t think I have a reliable way of getting the ffmpeg process PID. Note that there may be many ffmpeg processes running simultaneously.

Only the README is in Russian, everything inside is English. The very code that might be interesting to you is /src/video/camera.py lines 91-116.

I would not have suggested that. I just wanted to know if the shell is the problem. Sorry, I don’t have more ideas. If you can figure it out, please, share the solution. Of course, I will write if I have anything to add to solve this.