Overlay2 preventing boot?

Hello,
Something docker is doing is preventing my arch Linux machine from booting. It is talking about remounting as seen in this photo.

could it be going too fast before HDD can get spun up? It won’t accept any input/ssh and the machine has to be shut down and rebooted to work again. All drives/ram have plenty of space still. Not sure what logs to check to get extra information.

I assume your question is: how to delay the start of the docker service, until all required depencencies are available?

Here is how the default systemd docker.service unit looks like:

So basically you will need to figure out which service is responsible to start whatever you suspect is missing when docker is started, and modify the systemd unit to wait for it to be available.

I can only give you the pointer, but can’t help you to identify it.

I never experienced any problems with local disks over the last 12 years. I can see how waiting for network shares, or external hdd’s being mounted could cause problems. I never have issue with network shares either, as I use docker volumes whenever I need to access remote shares. I don’t mount remote shares into the host filesystem.

1 Like

Yeah, it shouldn’t have any problem mounting, especially since there’s no network shares (until explicitly started after boot), and it’s running on a nvme here. I notice it says

I don’t know if this is a coincidence but here I found this line " overlay Overlay networks connect multiple Docker daemons together." Maybe it’s something to do with a network which brings me to

Sometimes I get this error that hangs shutdowns:

[FAILED] Failed to start Network Manager Wait Online.
^@^@^@^@^@^@^@^@^@^@^@^@
Broadcast message from root@[computer name] on pts/2

The system will power off now!

I thought maybe just a delay for things to get going could help so I added this line below just to throw in a 30s delay

ExecStartPre=/bin/sleep 30

unfortunately the issue still happened, gotta figure out the network manager thing.

I ran

$ nmcli device status
DEVICE           TYPE      STATE                                  CONNECTION         
enp5s0           ethernet  connected                              Wired connection 1 
wlan0            wifi      connected                              [snip]                
tap-easytether   tun       connecting (getting IP configuration)  EasyTether         
tailscale0       tun       connected (externally)                 tailscale0         
br-3217fc5f7820  bridge    connected (externally)                 br-3217fc5f7820    
br-3fa7f96300fe  bridge    connected (externally)                 br-3fa7f96300fe    
br-45797add1204  bridge    connected (externally)                 br-45797add1204    
br-4e28caf142a4  bridge    connected (externally)                 br-4e28caf142a4    
br-5f0446d2d512  bridge    connected (externally)                 br-5f0446d2d512    
br-5fb5e76e20cd  bridge    connected (externally)                 br-5fb5e76e20cd    
br-cfe21c979aa4  bridge    connected (externally)                 br-cfe21c979aa4    
lo               loopback  connected (externally)                 lo                 
br-2b5e4fd87ba9  bridge    connected (externally)                 br-2b5e4fd87ba9    
br-966b216a8f73  bridge    connected (externally)                 br-966b216a8f73    
br-987516d099b6  bridge    connected (externally)                 br-987516d099b6    
docker0          bridge    connected (externally)                 docker0            
virbr0           bridge    connected (externally)                 virbr0             
p2p-dev-wlan0    wifi-p2p  disconnected                           --                 
veth030b893      ethernet  unmanaged                              --                 
veth06d9763      ethernet  unmanaged                              --                 
veth071cc02      ethernet  unmanaged                              --                 
veth0758f9f      ethernet  unmanaged                              --                 
veth07e1c95      ethernet  unmanaged                              --                 
veth0af310c      ethernet  unmanaged                              --                 
veth145ea09      ethernet  unmanaged                              --                 
veth1ccbcc3      ethernet  unmanaged                              --                 
veth1eb409f      ethernet  unmanaged                              --                 
veth2205a74      ethernet  unmanaged                              --                 
veth29d40c1      ethernet  unmanaged                              --                 
veth324e72a      ethernet  unmanaged                              --                 
veth33a78c0      ethernet  unmanaged                              --                 
veth37afa23      ethernet  unmanaged                              --                 
veth39bf091      ethernet  unmanaged                              --                 
veth433bc3a      ethernet  unmanaged                              --                 
veth4887070      ethernet  unmanaged                              --                 
veth59c0893      ethernet  unmanaged                              --                 
veth5ca8d12      ethernet  unmanaged                              --                 
veth664c359      ethernet  unmanaged                              --                 
veth69d22ac      ethernet  unmanaged                              --                 
veth7a25f7f      ethernet  unmanaged                              --                 
veth81a1d5c      ethernet  unmanaged                              --                 
veth85b6eea      ethernet  unmanaged                              --                 
veth8774839      ethernet  unmanaged                              --                 
veth88f4508      ethernet  unmanaged                              --                 
veth8bc5f50      ethernet  unmanaged                              --                 
veth913d44c      ethernet  unmanaged                              --                 
veth9772d9a      ethernet  unmanaged                              --                 
veth999ae4c      ethernet  unmanaged                              --                 
vethab7ceaf      ethernet  unmanaged                              --                 
vethabe9371      ethernet  unmanaged                              --                 
vethb0d330f      ethernet  unmanaged                              --                 
vethb151bc6      ethernet  unmanaged                              --                 
vethb32159e      ethernet  unmanaged                              --                 
vethc7788ff      ethernet  unmanaged                              --                 
vethccadf9a      ethernet  unmanaged                              --                 
vethd5962bc      ethernet  unmanaged                              --                 
vethd59d9af      ethernet  unmanaged                              --                 
vethd82b912      ethernet  unmanaged                              --                 
vethd834062      ethernet  unmanaged                              --                 
vethe228ea3      ethernet  unmanaged                              --                 
vethe2a65b9      ethernet  unmanaged                              --                 
vethe61f6fd      ethernet  unmanaged                              --                 
vethe761375      ethernet  unmanaged                              --                 
vethef0d8b4      ethernet  unmanaged                              --                 
vethf9402bc      ethernet  unmanaged                              --                 
vethf962927      ethernet  unmanaged                              --                 
vethfa8c262      ethernet  unmanaged                              --                 
vethfb6e450      ethernet  unmanaged                              --                 
vethfc2a502      ethernet  unmanaged                              --      

should I unmanage some of these you think like in this article? I’m worried about breaking docker networking lol

Then this can’t be the cause for the problem. You should investigate why the filesystem is mounted in read-only mode. This is not caused by Overlay2, but Overlay2 suffers from this situation and can’t work properly.

The overlay network is a swarm cluster scoped network that spans across one or more swarm cluster nodes. Not related to your problem. Overlay2, the one shown in your error message, is a storage driver.

Okay, good to know it’s not networking or something. Is there any way to track volume ID’s across restarts? I’m not sure how I could hunt down what volume/container is causing it if they’re all switching around. I do have some examples of :ro’s

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.51.0
    container_name: cadvisor
    networks:
      - backend_network
    ports:
        #changed from 8080:8080 (used)
      - 8181:8080
    volumes:
      - /:/rootfs:ro
      - /run:/run:ro
      - /sys:/sys:ro
      # I moved docker from /var/lib
      - /media/server/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
    privileged: true
    restart: unless-stopped

it also has a lot of overlay mentions in the logs, but no errors.

I0711 11:08:16.913655       1 fs.go:135] Filesystem UUIDs: map[143fa65c-2d8c-4a78-9eab-fbc7ec36fa06:/dev/sdc1 1aebcc39-6433-45c2-818b-2e026b4d415d:/dev/sde1 2416966C16963EAC:/dev/sdb2 251B-C491:/dev/nvme1n1p1 3b6e33f3-0d24-4fad-a8dc-e01dffa96c54:/dev/nvme1n1p2 425879F35879E657:/dev/sda2 566d753b-12de-4a9a-90c8-e496675a60d9:/dev/zram0 cad8f977-3dd7-42f7-9f63-cf79a3c92d24:/dev/sdd1 d69c6a60-89f8-4432-b0e3-a8b7673cdbf9:/dev/nvme0n1p1 faee6ae1-b7bc-4837-9ad2-365d648295b3:/dev/nvme2n1p1]

I0711 11:08:16.913727       1 fs.go:136] Filesystem partitions: map[/dev:{mountpoint:/dev major:0 minor:1193 fsType:tmpfs blockSize:0} /dev/nvme0n1p1:{mountpoint:/rootfs/media/server major:259 minor:4 fsType:ext4 blockSize:0} /dev/nvme1n1p2:{mountpoint:/rootfs major:259 minor:2 fsType:ext4 blockSize:0} /dev/nvme2n1p1:{mountpoint:/rootfs/media/240gb major:259 minor:6 fsType:ext4 blockSize:0} /dev/sdc1:{mountpoint:/rootfs/media/8tb major:8 minor:33 fsType:ext4 blockSize:0} /dev/sdd1:{mountpoint:/rootfs/media/16d1 major:8 minor:49 fsType:ext4 blockSize:0} /dev/sde1:{mountpoint:/rootfs/media/16d2 major:0 minor:48 fsType:btrfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:1206 fsType:tmpfs blockSize:0} /rootfs/dev/shm:{mountpoint:/rootfs/dev/shm major:0 minor:26 fsType:tmpfs blockSize:0} /rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev:{mountpoint:/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev major:0 minor:1193 fsType:tmpfs blockSize:0} /rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm:{mountpoint:/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm major:0 minor:1206 fsType:tmpfs blockSize:0} /rootfs/run:{mountpoint:/rootfs/run major:0 minor:24 fsType:tmpfs blockSize:0} /rootfs/run/credentials/systemd-journald.service:{mountpoint:/rootfs/run/credentials/systemd-journald.service major:0 minor:33 fsType:tmpfs blockSize:0} /rootfs/run/credentials/systemd-resolved.service:{mountpoint:/rootfs/run/credentials/systemd-resolved.service major:0 minor:43 fsType:tmpfs blockSize:0} /rootfs/run/user/1000:{mountpoint:/rootfs/run/user/1000 major:0 minor:54 fsType:tmpfs blockSize:0} /rootfs/tmp:{mountpoint:/rootfs/tmp major:0 minor:37 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /run/credentials/systemd-journald.service:{mountpoint:/run/credentials/systemd-journald.service major:0 minor:33 fsType:tmpfs blockSize:0} /run/credentials/systemd-resolved.service:{mountpoint:/run/credentials/systemd-resolved.service major:0 minor:43 fsType:tmpfs blockSize:0} /run/user/1000:{mountpoint:/run/user/1000 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev major:0 minor:1193 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm major:0 minor:1206 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/dev/shm:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/dev/shm major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev major:0 minor:1193 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/dev/shm major:0 minor:1206 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run major:0 minor:24 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/credentials/systemd-journald.service:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/credentials/systemd-journald.service major:0 minor:33 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/credentials/systemd-resolved.service:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/credentials/systemd-resolved.service major:0 minor:43 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/user/1000:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/run/user/1000 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/tmp:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/tmp major:0 minor:37 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run major:0 minor:24 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/credentials/systemd-journald.service:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/credentials/systemd-journald.service major:0 minor:33 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/credentials/systemd-resolved.service:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/credentials/systemd-resolved.service major:0 minor:43 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/user/1000:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/run/user/1000 major:0 minor:54 fsType:tmpfs blockSize:0} overlay_0-105:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0c4242faa778bd99795f328015f12ed643082f1f6547556c5c9979d4510c904f/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/e1d46ad39cdc2bf66dd70769143ff10da57bf2793d6889988920ab2ad1954274/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-117:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/1ec1e178f56048d0c2fe16fb469d3819a8e028793297ab4f428530cbd79b825b/merged major:0 minor:117 fsType:overlay blockSize:0} overlay_0-120:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0ec823ffd07551ecd7779ed7c0a3df81ce7234549ebf30a0eb979b50389c1d63/merged major:0 minor:120 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/bdb18b248cdd9972692a2d8bfd2b80be61e206b8e2239c72541ebfb6feaca9ab/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0dc17f22c7358e865d264197b18f75838b1ba720d85baa94fdff4565dedd3bbb/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-125:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/ba628e3c8ce1b8056409685bfc54cb66c888a21b182c617f9a57575c012861e4/merged major:0 minor:125 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/471738ee490b02cd3d5760e883d747eb20392c1d92356c3cff595371bee2e48a/merged major:0 minor:126 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/cf23595958bfa0cd94bbd6cc088d3463bf309c2777ce58ed1e09874ac578a308/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0ec575b5fedff85893cb19e6dcac1a3626767ebbfd92ea871e291739194bb6da/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/bd88eda418b88a44caf50a4acd55cd47270cf8ea1e0b8ae20062f2556b9b7757/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/30e8967743a6f39d00afcd9454a5d773b13a4fd06de56de9735e28d08b7909bb/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/509312b954387c7eddb66f06cf33e86953ced060fe3ff7805343a3764ae1fa83/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/a9d298e6f61aea7456e178d3d16724855bc98b04de7bfefe4b181a7281698de8/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/3c4b56bea8d183c3acff2645f73ea9d05c055d6081ab8c3bfe4570781591424d/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/6ae25828fc257d8dc7f2a1ceb6209420a4156458d4e55d1d2f02f34fe334dee2/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/81be4eee85044c3c75c05daebe979280d331e92548f8f87d45d8cb0e2cb1746c/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/471a636679419326b6caf8df5da7ccd520dc8c3831a2c7b8681fab9b92d10c76/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/6533e2a44b6373d0a2ebbe4b7fea6fa064a214f057e3a5ac9bc6d6606b859305/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/1eecb4b8c66c10f4fe2fddedda0dda3a2d6703b2bb314a6b81a9c7a2b6ee9459/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/dbb393b38e2fa04348b1863b5fe564e6c5f3ce8730bfaaf9fbb77c08c5d6a8cf/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/71df39ef4a02742a63494eeddd1273afab60298fae9565279205ec80881bf559/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/2b582da3c66e079119c076259c6aa852b04968082234fa88e7a5e36d33198c33/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/02a6ba4cd13df928d342a98c0aebb8ff30c84aac1f0e817fd78e6980464e6168/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/84c40f8d175676ff725d7e65def58cf24930eeb35eb3143165fc471d60752c8c/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-165:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/0ddb4b2841218ff6f06128ca2021c416d55ac4bb8c76a4b1297ce9dc8b47762d/merged major:0 minor:165 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/5ea85a16f3b940dbaa6f8c61d4348468fb947df349fc4dd76e399bf2d06e9985/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/088397e109b8093163343e5e0aaab7b9139245186a84f9d7d5d1692738de47df/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-170:{mountpoint:/var/lib/docker/overlay2/6c2e21b52d3ef4d5bacf6ab7b765abd4255e051dfbfb117d14a9cb6d754ce52a/merged major:0 minor:170 fsType:overlay blockSize:0} overlay_0-175:{mountpoint:/var/lib/docker/overlay2/738adf3d359d9eb8208afe87fe3c3b8c039d74b56be6bb1a182242c70fb1ebe3/merged major:0 minor:175 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/ major:0 minor:176 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/docker/overlay2/30d42bb32e2bf38a775e5b4742781449a4ab279db8c9ee0a8b7bac576d284865/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/docker/overlay2/d18308abc1d0106506d649bf3d710ef0ed288aa0189411be4647cdf24dbd696e/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/docker/overlay2/5aceae2da6574f3e61e0e37868b038939dd4489401d67b024b68a1f4cfa0e441/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/862b7ccdfd7b6d1a53dacf680263b760558f0074a95e63a35f67ae6b39cf2dd3/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/713a255a42a104f1c554f152d5f2de7b5d2b29cd4db7ea3edeacd3cc8ed4e904/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/docker/overlay2/71f89c13a798dcbd40e3cf70580b84ba7b4fc1471ce2a5c8c11bd7931d966408/merged/rootfs/media/server/docker/overlay2/48e184b750fe273426cc5f10e40dc6c20b133d3976442ce08148317a8993acaf/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/li

I had tried running nmcli device status but no logs look to appear applicable (maybe they’d been erased because I had to hard shutdown computer?)

It’s not about volumes either.

A container id (and as such it’s folders in /var/lib/docker/containers) does not change when a container stops, unless when the container is created through a swarm service task (If you have no idea what this means, you are most likely not using it). With docker compose, a container id only changes, when the container is recreated, due to configuration changes.

Your initial error message is about the container filesystems, managed by overlay2. There must be a reason the merged folder of the overlay2 filesystem of you containers are re-mounted in read-only mode. You need to figure out and solve that underlying reason.

The only situation where I can see docker doing this by itself is when you specifically create your containers with a read-only filesystem (like it’s done in most enterprise environments). If you didn’t add the property read_only: true to your services, then you are not using read-only containre filesystems.

Just in case it gives anyone any idea, the last time I saw a similar error message, it was on a Kubernetes node. More specifically on a k3s worker. We could not figure it out why it happened, but by the time we saw the logs, the filesystem was not read-only. I think when it happened, I also had problems with data disks that were handled by LongHorn when some of the data disks were modified by “multipathd”. I’m not sure if these two errors were related in my case, but I noticed the logs after the data disk issue.

And there is this issue on the Arch Linux forum

Since you are using Arch, you could try to look for similar issues on an Arch forum as well. I did not search for Arch at all. It was just one of the first results., so that could matter.