Docker-compose down - root file system is read-only

General Discussions
1

Hello.

I am new here and new to using Docker.

I have a Supermicro dual Xeon server (dual Xeon E5-2697A v4) system, with a Supermicro X10DRi-T4+ motherboard and 256 GB of DDR4-2400 ECC Reg RAM, running Proxmox 7.3-3.

I created an Ubuntu 18.04 LTS VM and I downloaded the Plex Media Server docker image and it was running fine for quite some time.

However, this morning, it looks like that there was an error or some kind of an issue with it, so I was going to try and restart the docker image using docker-compose down and docker-compose up -d.

But when I went to issue the docker-compose down command, it said that it couldn’t remove the volume because it was “read only”?

My docker-compose.yml file is not set up where the Plex Media Server would have a read-only root filesystem (being a Plex Media Server docker, there is no real need for that).

I tried googling the issue, and most of what I found appears to be people asking how to configure their docker to be read only, rather than when you are trying to shutdown a docker container, and it saying that it can’t, because the root file system, somehow, for some reason, is in a read-only state.

If there is additional information that is needed by the team, please let me know what commands to run.

Again, I am new to this, so I am in the process of learning more about it.

Thank you for your help in advance.

My apologies for being such a n00b.

(I’m at work right now, so I am unable to post the exact error message, but I can do that when I get home from work, later on today.)

2

Do you still have the exact error message? I guess it was more than showing “read-only”. The context could be important.

You can mount a volume as read-only, but that would not affect removing the volume. If the volume could not be deleted, that probably means that the filesystem on which the volume is stored became read-only for some reasons. Docker could not prevent it. It just uses the “backing filesystem”

3

Yup, I’m at home now, so here is the exact error message:

ubuntu@plex:~/pms-docker$ sudo docker-compose down
[sudo] password for ubuntu:
Stopping plex ... done
Removing plex .. error

ERROR: for plex  container 88af890b5ddb5b582866bb773ba524b385e4a586d087a8bfff86c5b6c1891e56: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/35a4a4a6ae9bc6670a9781352761fcc39002fd2d90b0cdc1048b4772c7b2a1f1: read-only file system

Thank you.

4

So it is not about the volume. I focused on a volume issue after you mentioned Docker could not remove a volume, but now it seems /var/lib/docker/overlay2 is read-only. Possibly the whole /var/lib/docker folder.

Have you mounted this fodler from a remote share or a partition different from the partition of the host os?

Update:

I realized I had a read-only filesystem issue recently, but I played with a vitual machine on macOS with an experimental feature. It was perfect at the beginning, but somehow I managed to break it and I could not fix it. In my case whe whole root filesystem of the OS was broken and I tried an ARM virtual machine.

Since you are laso running a VM even though a different one, maybe the disk of the virtual machine became read-only.

5

So, being that I am running docker inside an Ubuntu VM inside Proxmox – Proxmox is my storage host, and it communicates to the Ubuntu VM via virtio-fs.

In doing it this way, it bypassed the need for a NFS mount/export (or communication of a network connection) (virtual or not), to the host system. (My homelab is just really mostly me using it. Wife and kids use Plex and doesn’t deal with the sysadmin issues that I deal with, so as far as they’re concerned, this whole process is transparent to them. For my kids, either it works, or it doesn’t. And that’s about the extent that they really care about.) – my point being that rather than going through NFS, virtio-fs enables more “direct” communication with the host system’s file/data storage.)

I ended up rebooting the VM, and that seemed to have cleared things up, but I am still not really sure that I understand why this error happened in the first place, and what I can do to fix it in the future, should it happen again, without needing to reboot said VM.

Not sure.

It shouldn’t have been though because the “OS” disk for the VM has been allocated 1 TiB (for the Plex cache data), and the “backend” data store has about 155 TiB “available” to it.

I would understand it if it was running out of space, and that might be the reason why the file system went to read only, but there is nothing that would indicate that that was the case, nor why the failure/error occurred.

Thank you.

6

Thanks for the detailed explanation.

I whish I could tell you why it happened and that it would not happen again. It doesn’t seem to be a Docker issue, unless Docker did something bad and the OS made the filesystem read-only. I have never found out either why my test VM’s filesystem was read-only but in my case reboot made it worse and I couldn’t even boot the OS so I am glad it helped you.

1 Like
7

Thank you.

Yeah, I mean, being that its for my Plex Media Server at home, it’s not a huge deal, if I have to reboot the VM.

My initial concern was moreso around what happened if I rebooted the VM, and then the docker-compose wouldn’t come back up because there was the defunct or stale/dead docker container that was left.

My “pre-emptive”, alternative plan was that in the event that docker and/or the Plex Media Server container still had issues, was that I would just install Plex Media Server on the Ubuntu VM that was hosting said docker container.

Not the end of the world, but I also figured that since this error/failure occurred, that I also wanted to take and leverage that as a learning opportunity so that I would try and learn and know better in regards to what to do if (or when) it happens again.

Thank you.