Share and learn in the Docker community.
Hello,
Something docker is doing is preventing my arch Linux machine from booting. It is talking about remounting as seen in this photo.
could it be going too fast before HDD can get spun up? It won’t accept any input/ssh and the machine has to be shut down and rebooted to work again. All drives/ram have plenty of space still. Not sure what logs to check to get extra information.
I assume your question is: how to delay the start of the docker service, until all required depencencies are available?
Here is how the default systemd docker.service unit looks like:
So basically you will need to figure out which service is responsible to start whatever you suspect is missing when docker is started, and modify the systemd unit to wait for it to be available.
I can only give you the pointer, but can’t help you to identify it.
I never experienced any problems with local disks over the last 12 years. I can see how waiting for network shares, or external hdd’s being mounted could cause problems. I never have issue with network shares either, as I use docker volumes whenever I need to access remote shares. I don’t mount remote shares into the host filesystem.
Yeah, it shouldn’t have any problem mounting, especially since there’s no network shares (until explicitly started after boot), and it’s running on a nvme here. I notice it says
I don’t know if this is a coincidence but here I found this line " overlay Overlay networks connect multiple Docker daemons together." Maybe it’s something to do with a network which brings me to
Sometimes I get this error that hangs shutdowns:
[FAILED] Failed to start Network Manager Wait Online.
^@^@^@^@^@^@^@^@^@^@^@^@
Broadcast message from root@[computer name] on pts/2
The system will power off now!
I thought maybe just a delay for things to get going could help so I added this line below just to throw in a 30s delay
ExecStartPre=/bin/sleep 30
unfortunately the issue still happened, gotta figure out the network manager thing.
I ran
$ nmcli device status
DEVICE TYPE STATE CONNECTION
enp5s0 ethernet connected Wired connection 1
wlan0 wifi connected [snip]
tap-easytether tun connecting (getting IP configuration) EasyTether
tailscale0 tun connected (externally) tailscale0
br-3217fc5f7820 bridge connected (externally) br-3217fc5f7820
br-3fa7f96300fe bridge connected (externally) br-3fa7f96300fe
br-45797add1204 bridge connected (externally) br-45797add1204
br-4e28caf142a4 bridge connected (externally) br-4e28caf142a4
br-5f0446d2d512 bridge connected (externally) br-5f0446d2d512
br-5fb5e76e20cd bridge connected (externally) br-5fb5e76e20cd
br-cfe21c979aa4 bridge connected (externally) br-cfe21c979aa4
lo loopback connected (externally) lo
br-2b5e4fd87ba9 bridge connected (externally) br-2b5e4fd87ba9
br-966b216a8f73 bridge connected (externally) br-966b216a8f73
br-987516d099b6 bridge connected (externally) br-987516d099b6
docker0 bridge connected (externally) docker0
virbr0 bridge connected (externally) virbr0
p2p-dev-wlan0 wifi-p2p disconnected --
veth030b893 ethernet unmanaged --
veth06d9763 ethernet unmanaged --
veth071cc02 ethernet unmanaged --
veth0758f9f ethernet unmanaged --
veth07e1c95 ethernet unmanaged --
veth0af310c ethernet unmanaged --
veth145ea09 ethernet unmanaged --
veth1ccbcc3 ethernet unmanaged --
veth1eb409f ethernet unmanaged --
veth2205a74 ethernet unmanaged --
veth29d40c1 ethernet unmanaged --
veth324e72a ethernet unmanaged --
veth33a78c0 ethernet unmanaged --
veth37afa23 ethernet unmanaged --
veth39bf091 ethernet unmanaged --
veth433bc3a ethernet unmanaged --
veth4887070 ethernet unmanaged --
veth59c0893 ethernet unmanaged --
veth5ca8d12 ethernet unmanaged --
veth664c359 ethernet unmanaged --
veth69d22ac ethernet unmanaged --
veth7a25f7f ethernet unmanaged --
veth81a1d5c ethernet unmanaged --
veth85b6eea ethernet unmanaged --
veth8774839 ethernet unmanaged --
veth88f4508 ethernet unmanaged --
veth8bc5f50 ethernet unmanaged --
veth913d44c ethernet unmanaged --
veth9772d9a ethernet unmanaged --
veth999ae4c ethernet unmanaged --
vethab7ceaf ethernet unmanaged --
vethabe9371 ethernet unmanaged --
vethb0d330f ethernet unmanaged --
vethb151bc6 ethernet unmanaged --
vethb32159e ethernet unmanaged --
vethc7788ff ethernet unmanaged --
vethccadf9a ethernet unmanaged --
vethd5962bc ethernet unmanaged --
vethd59d9af ethernet unmanaged --
vethd82b912 ethernet unmanaged --
vethd834062 ethernet unmanaged --
vethe228ea3 ethernet unmanaged --
vethe2a65b9 ethernet unmanaged --
vethe61f6fd ethernet unmanaged --
vethe761375 ethernet unmanaged --
vethef0d8b4 ethernet unmanaged --
vethf9402bc ethernet unmanaged --
vethf962927 ethernet unmanaged --
vethfa8c262 ethernet unmanaged --
vethfb6e450 ethernet unmanaged --
vethfc2a502 ethernet unmanaged --
should I unmanage some of these you think like in this article? I’m worried about breaking docker networking lol
Then this can’t be the cause for the problem. You should investigate why the filesystem is mounted in read-only mode. This is not caused by Overlay2, but Overlay2 suffers from this situation and can’t work properly.
The overlay network is a swarm cluster scoped network that spans across one or more swarm cluster nodes. Not related to your problem. Overlay2, the one shown in your error message, is a storage driver.
Okay, good to know it’s not networking or something. Is there any way to track volume ID’s across restarts? I’m not sure how I could hunt down what volume/container is causing it if they’re all switching around. I do have some examples of :ro’s
cadvisor:
image: gcr.io/cadvisor/cadvisor:v0.51.0
container_name: cadvisor
networks:
- backend_network
ports:
#changed from 8080:8080 (used)
- 8181:8080
volumes:
- /:/rootfs:ro
- /run:/run:ro
- /sys:/sys:ro
# I moved docker from /var/lib
- /media/server/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
devices:
- /dev/kmsg
privileged: true
restart: unless-stopped
it also has a lot of overlay mentions in the logs, but no errors.
I0711 11:08:16.913655 1 fs.go:135] Filesystem UUIDs: map[143fa65c-2d8c-4a78-9eab-fbc7ec36fa06:/dev/sdc1 1aebcc39-6433-45c2-818b-2e026b4d415d:/dev/sde1 2416966C16963EAC:/dev/sdb2 251B-C491:/dev/nvme1n1p1 3b6e33f3-0d24-4fad-a8dc-e01dffa96c54:/dev/nvme1n1p2 425879F35879E657:/dev/sda2 566d753b-12de-4a9a-90c8-e496675a60d9:/dev/zram0 cad8f977-3dd7-42f7-9f63-cf79a3c92d24:/dev/sdd1 d69c6a60-89f8-4432-b0e3-a8b7673cdbf9:/dev/nvme0n1p1 faee6ae1-b7bc-4837-9ad2-365d648295b3:/dev/nvme2n1p1]
I0711 11:08:16.913727 1 fs.go:136] Filesystem partitions: map[/dev:{mountpoint:/dev major:0 minor:1193 fsType:tmpfs blockSize:0} /dev/nvme0n1p1:{mountpoint:/rootfs/media/server major:259 minor:4 fsType:ext4 blockSize:0} /dev/nvme1n1p2:{mountpoint:/rootfs major:259 minor:2 fsType:ext4 blockSize:0} /dev/nvme2n1p1:{mountpoint:/rootfs/media/240gb major:259 minor:6 fsType:ext4 blockSize:0} /dev/sdc1:{mountpoint:/rootfs/media/8tb major:8 minor:33 fsType:ext4 blockSize:0} /dev/sdd1:{mountpoint:/rootfs/media/16d1 major:8 minor:49 fsType:ext4 blockSize:0} /dev/sde1:{mountpoint:/rootfs/media/16d2 major:0 minor:48 fsType:btrfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:1206 fsType:tmpfs blockSize:0} /rootfs/dev/shm:{mountpoint:/rootfs/dev/shm major:0 minor:26 fsType:tmpfs blockSize:0} /rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev:{mountpoint:/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev major:0 minor:1193 fsType:tmpfs blockSize:0} /rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm:{mountpoint:/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm major:0 minor:1206 fsType:tmpfs blockSize:0} /rootfs/run:{mountpoint:/rootfs/run major:0 minor:24 fsType:tmpfs blockSize:0} /rootfs/run/credentials/systemd-journald.service:{mountpoint:/rootfs/run/credentials/systemd-journald.service major:0 minor:33 fsType:tmpfs blockSize:0} /rootfs/run/credentials/systemd-resolved.service:{mountpoint:/rootfs/run/credentials/systemd-resolved.service major:0 minor:43 fsType:tmpfs blockSize:0} /rootfs/run/user/1000:{mountpoint:/rootfs/run/user/1000 major:0 minor:54 fsType:tmpfs blockSize:0} /rootfs/tmp:{mountpoint:/rootfs/tmp major:0 minor:37 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /run/credentials/systemd-journald.service:{mountpoint:/run/credentials/systemd-journald.service major:0 minor:33 fsType:tmpfs blockSize:0} /run/credentials/systemd-resolved.service:{mountpoint:/run/credentials/systemd-resolved.service major:0 minor:43 fsType:tmpfs blockSize:0} /run/user/1000:{mountpoint:/run/user/1000 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev major:0 minor:1193 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm major:0 minor:1206 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/dev/shm:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/dev/shm major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev major:0 minor:1193 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm major:0 minor:1206 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run major:0 minor:24 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/credentials/systemd-journald.service:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/credentials/systemd-journald.service major:0 minor:33 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/credentials/systemd-resolved.service:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/credentials/systemd-resolved.service major:0 minor:43 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/user/1000:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/user/1000 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/tmp:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/tmp major:0 minor:37 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run major:0 minor:24 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/credentials/systemd-journald.service:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/credentials/systemd-journald.service major:0 minor:33 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/credentials/systemd-resolved.service:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/credentials/systemd-resolved.service major:0 minor:43 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/user/1000:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/user/1000 major:0 minor:54 fsType:tmpfs blockSize:0} overlay_0-105:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0c4242faa778bd99795f328015f12ed643082f1f6547556c5c9979d4510c904f/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/e1d46ad39cdc2bf66dd70769143ff10da57bf2793d6889988920ab2ad1954274/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-117:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/1ec1e178f56048d0c2fe16fb469d3819a8e028793297ab4f428530cbd79b825b/merged major:0 minor:117 fsType:overlay blockSize:0} overlay_0-120:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0ec823ffd07551ecd7779ed7c0a3df81ce7234549ebf30a0eb979b50389c1d63/merged major:0 minor:120 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/bdb18b248cdd9972692a2d8bfd2b80be61e206b8e2239c72541ebfb6feaca9ab/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0dc17f22c7358e865d264197b18f75838b1ba720d85baa94fdff4565dedd3bbb/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-125:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/ba628e3c8ce1b8056409685bfc54cb66c888a21b182c617f9a57575c012861e4/merged major:0 minor:125 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/471738ee490b02cd3d5760e883d747eb20392c1d92356c3cff595371bee2e48a/merged major:0 minor:126 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/cf23595958bfa0cd94bbd6cc088d3463bf309c2777ce58ed1e09874ac578a308/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0ec575b5fedff85893cb19e6dcac1a3626767ebbfd92ea871e291739194bb6da/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/bd88eda418b88a44caf50a4acd55cd47270cf8ea1e0b8ae20062f2556b9b7757/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/30e8967743a6f39d00afcd9454a5d773b13a4fd06de56de9735e28d08b7909bb/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/509312b954387c7eddb66f06cf33e86953ced060fe3ff7805343a3764ae1fa83/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/a9d298e6f61aea7456e178d3d16724855bc98b04de7bfefe4b181a7281698de8/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/3c4b56bea8d183c3acff2645f73ea9d05c055d6081ab8c3bfe4570781591424d/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/6ae25828fc257d8dc7f2a1ceb6209420a4156458d4e55d1d2f02f34fe334dee2/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/81be4eee85044c3c75c05daebe979280d331e92548f8f87d45d8cb0e2cb1746c/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/471a636679419326b6caf8df5da7ccd520dc8c3831a2c7b8681fab9b92d10c76/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/6533e2a44b6373d0a2ebbe4b7fea6fa064a214f057e3a5ac9bc6d6606b859305/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/1eecb4b8c66c10f4fe2fddedda0dda3a2d6703b2bb314a6b81a9c7a2b6ee9459/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/dbb393b38e2fa04348b1863b5fe564e6c5f3ce8730bfaaf9fbb77c08c5d6a8cf/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71df39ef4a02742a63494eeddd1273afab60298fae9565279205ec80881bf559/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/2b582da3c66e079119c076259c6aa852b04968082234fa88e7a5e36d33198c33/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/02a6ba4cd13df928d342a98c0aebb8ff30c84aac1f0e817fd78e6980464e6168/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/84c40f8d175676ff725d7e65def58cf24930eeb35eb3143165fc471d60752c8c/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-165:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0ddb4b2841218ff6f06128ca2021c416d55ac4bb8c76a4b1297ce9dc8b47762d/merged major:0 minor:165 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/5ea85a16f3b940dbaa6f8c61d4348468fb947df349fc4dd76e399bf2d06e9985/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/088397e109b8093163343e5e0aaab7b9139245186a84f9d7d5d1692738de47df/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-170:{mountpoint:/var/lib/docker/overlay2/6c2e21b52d3ef4d5bacf6ab7b765abd4255e051dfbfb117d14a9cb6d754ce52a/merged major:0 minor:170 fsType:overlay blockSize:0} overlay_0-175:{mountpoint:/var/lib/docker/overlay2/738adf3d359d9eb8208afe87fe3c3b8c039d74b56be6bb1a182242c70fb1ebe3/merged major:0 minor:175 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/ major:0 minor:176 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/docker/overlay2/30d42bb32e2bf38a775e5b4742781449a4ab279db8c9ee0a8b7bac576d284865/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/docker/overlay2/d18308abc1d0106506d649bf3d710ef0ed288aa0189411be4647cdf24dbd696e/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/docker/overlay2/5aceae2da6574f3e61e0e37868b038939dd4489401d67b024b68a1f4cfa0e441/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/862b7ccdfd7b6d1a53dacf680263b760558f0074a95e63a35f67ae6b39cf2dd3/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/713a255a42a104f1c554f152d5f2de7b5d2b29cd4db7ea3edeacd3cc8ed4e904/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/48e184b750fe273426cc5f10e40dc6c20b133d3976442ce08148317a8993acaf/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/li
I had tried running nmcli device status but no logs look to appear applicable (maybe they’d been erased because I had to hard shutdown computer?)
It’s not about volumes either.
A container id (and as such it’s folders in /var/lib/docker/containers) does not change when a container stops, unless when the container is created through a swarm service task (If you have no idea what this means, you are most likely not using it). With docker compose, a container id only changes, when the container is recreated, due to configuration changes.
Your initial error message is about the container filesystems, managed by overlay2. There must be a reason the merged folder of the overlay2 filesystem of you containers are re-mounted in read-only mode. You need to figure out and solve that underlying reason.
The only situation where I can see docker doing this by itself is when you specifically create your containers with a read-only filesystem (like it’s done in most enterprise environments). If you didn’t add the property read_only: true to your services, then you are not using read-only containre filesystems.
Just in case it gives anyone any idea, the last time I saw a similar error message, it was on a Kubernetes node. More specifically on a k3s worker. We could not figure it out why it happened, but by the time we saw the logs, the filesystem was not read-only. I think when it happened, I also had problems with data disks that were handled by LongHorn when some of the data disks were modified by “multipathd”. I’m not sure if these two errors were related in my case, but I noticed the logs after the data disk issue.
And there is this issue on the Arch Linux forum
Since you are using Arch, you could try to look for similar issues on an Arch forum as well. I did not search for Arch at all. It was just one of the first results., so that could matter.