Container not restarting after volume mount fails

General
, ,
1

Hello docker community,

recently I encountered an issue after rebooting a server; it failed to automatically restart a previously running and working container.

The container in question uses a CIFS mount with the local driver and the problem arises if the CIFS server is not reachable during the restart process.

The error in the docker logs is as follows:

level=error msg="failed to start container" container=ca1a13714b040be5478cc41191b1a04bcd63e1f8bcd0a7bec501ee1d419674e4 error="error while mounting volume '/var/lib/docker/volumes/vol_certificates/_data': failed to mount local volume: mount //certificates.url.com/certificates:/var/lib/docker/volumes/vol_certificates/_data, data: vers=3.0,addr=10.10.0.2,username=certificates,password=XXXXXXXXXXXXXXXXX,port=445,sec=ntlmv2i,_netdev,hard: no such file or directory"

Since the restart policy is set to unless-stopped I expected the docker daemon to attempt restarting the container, but it will not do that. The container just stays down.

Inspecting the container gives the following output (excerpt):

[...]
"State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 2,
            "Error": "error while mounting volume '/var/lib/docker/volumes/vol_certificates/_data': failed to mount local volume: mount //certificates.url.com/certificates:/var/lib/docker/volumes/vol_certificates/_data, data: vers=3.0,addr=10.10.0.2,username=certificates,password=XXXXXXXXXXXXXXXXX,port=445,sec=ntlmv2i,_netdev,hard: no such file or directory",
            "StartedAt": "2022-03-09T06:18:46.434234582Z",
            "FinishedAt": "2022-03-09T06:18:49.412527317Z"
}
[...]
"HostConfig": {
[...]
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
[...]

The container is started by a docker-compose.yaml like this:

version: "3.7"
services:
  nginx-proxy:
    image: nginxproxy/nginx-proxy
    container_name: nginx-proxy
    volumes:
      - vol_proxy_dhparam:/etc/nginx/dhparam/
      - vol_certificates:/etc/nginx/certs:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
    restart: unless-stopped

volumes:
  vol_proxy_dhparam:
  vol_certificates:
    driver: local
    driver_opts:
      type: 'cifs'
      o: 'vers=3.0,addr=certificates.url.com,username=${CERTIFICATES_CIFS_USER},password=${CERTIFICATES_CIFS_PASS},port=445,sec=ntlmv2i,_netdev'
      device: '//certificates.url.com/certificates'

Is the restart policy ignored after volume errors and if so, is there a way with integrated docker tools to fix this issue?

Otherwise I will add some functionality on the server to detect the error state and restart the container that way.

Some additional software infos:

Ubuntu 20.04.4 LTS
Docker version 20.10.7, build 20.10.7-0ubuntu5~20.04.2
docker-compose version 1.25.0, build unknown

Best regards and thanks for any input in advance,
Thorsten Fuchs

2

The issue you’re experiencing with the container not restarting might be related to the CIFS volume mount problem. When Docker encounters issues with the volume mount, it may not restart the container because it considers the error significant. The “unless-stopped” policy assumes that the container stops for valid reasons.

To work around this issue and have more control over the restart behavior, you can modify your Docker Compose configuration to use a tool like docker-compose restart in combination with a script that checks the status of the container. Here’s how you can approach this:

  1. Create a simple shell script (e.g., restart-container.sh) to restart the container if it’s not running:
#!/bin/bash

if ! docker ps | grep -q "nginx-proxy"; then
  docker-compose restart nginx-proxy
fi

  1. Run this script in a loop using cron or another scheduling mechanism. This script will check if the container is running and restart it if it’s not.

  2. Make sure to adjust the script and the container name (“nginx-proxy”) to match your specific Docker Compose configuration.

By periodically running this script, you can ensure that the container is restarted if it’s not running, even in the presence of CIFS volume mount errors.

This is the best solution.