Docker Community Forums

Share and learn in the Docker community.

Upgrading kernel version on RHEL7 to 3.10.0-1160.21 breaks Docker with the overlay2 storage driver

Our enterprise patching team updated our system kernels overnight and as a result our Docker services were broken. We have backed out the kernel upgrade and the services are again behaving as expected, so we can be pretty confident the kernel update was the cause.

A typical error would be
open /var/lib/docker/overlay2/abcde12345/some_file: operation not permitted
This would be similar to known issue(s) e.g.

Error response from daemon: open [path to file]: operation not permitted. · Issue #20950 · moby/moby · GitHub

Works under:
RHEL 7
3.10.0-1160.15.2.el7.x86_64 #1 SMP Thu Jan 21 16:15:07 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

Broken under
3.10.0-1160.21

Client: Docker Engine - Community
Version: 20.10.5
API version: 1.41
Go version: go1.13.15
Git commit: 55c4c88
Built: Tue Mar 2 20:33:55 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.5
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 363e9a8
Built: Tue Mar 2 20:32:17 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0

cat /etc/docker/daemon.json

{
“storage-driver”: “overlay2”
}

Configure Docker with the overlay or overlay2 storage driver
It is highly recommended that you use the overlay2 driver if possible, rather than the overlay driver. The overlay driver is not supported for Docker EE.

To configure Docker to use the overlay storage driver your Docker host must be running version 3.18 of the Linux kernel (preferably newer) with the overlay kernel module loaded. For the overlay2 driver, the version of your kernel must be 4.0 or newer.

Before following this procedure, you must first meet all the prerequisites.

The steps below outline how to configure the overlay2 storage driver. If you need to use the legacy overlay driver, specify it instead.

Stop Docker.

$ sudo systemctl stop docker
Copy the contents of /var/lib/docker to a temporary location.

$ cp -au /var/lib/docker /var/lib/docker.bk
If you want to use a separate backing filesystem from the one used by /var/lib/, format the filesystem and mount it into /var/lib/docker. Make sure add this mount to /etc/fstab to make it permanent.

Edit /etc/docker/daemon.json. If it does not yet exist, create it. Assuming that the file was empty, add the following contents.

{
“storage-driver”: “overlay2”
}
Docker does not start if the daemon.json file contains badly-formed JSON.

Start Docker.

$ sudo systemctl start docker
Verify that the daemon is using the overlay2 storage driver. Use the docker info command and look for Storage Driver and Backing filesystem.

$ docker info

Containers: 0
Images: 0
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
<…>
Docker is now using the overlay2 storage driver and has automatically created the overlay mount with the required lowerdir, upperdir, merged, and workdir constructs.

Continue reading for details about how OverlayFS works within your Docker containers, as well as performance advice and information about limitations of its compatibility with different backing filesystems.

You missed the point of this thread. We have been running docker with Overlay2. The kernel upgrade broke that and reverting the kernel fixed the issue.