Why is my XFS Block IO *insane* compared to my EXT4 Block IO?

General
1

I’ve been “gifted” :wink: a Docker container that’s exhibiting what I think is very strange behavior. The authors tell me there should be very very little write IO. When I deploy the container on a host with EXT4 filesystem(s) this is absolutely the case…I see about 600MB of write IO when the container has completed its job. When I deploy the same container on a host with XFS filesystem(s) things go haywire…at the end of the run docker stats is reporting about 16GB of write IO. And it’s incredibly peaky…almost all (all but about 600MB) at the beginning of its processing run (the same container running on EXT4 writes nothing at this point), within 60 seconds of the container starting (full run takes about 10 minutes…the EXT4 container does it in about 5 due at least in part to the lack of IO wait).

Both hosts are CentOS 7, fully patched as of Friday. Both are running Docker CE 20.10.9. Both are using overlay2. The container is using bind mounts ( :frowning: ) for two directories in both cases. The container is processing the same job in both cases. The Docker hosts are built exactly the same way other than filesystem type and both are very well-resourced and running on SSD-backed FC-connected datastores. EDIT: d_type is set as advised.

Does anyone have any ideas of what might be happening here or why? It was a happy accident that I stumbled on the difference between the filesystem types so I can get acceptable performance from the container, but to “fix” this I’m going to have to explain it.

For those interested…I think I know what is happening, I just don’t know why. When I use inotify to look into the activity in the directory where my containers are, in addition to a lot more entries for the XFS-backed system (other files, etc. being written when I compare the traces), when I look at a representative “same” action I see 5 ops on XFS…there are only 2 for the same action on EXT4. See below:

XFS
Sat Oct 23 17:17:06 2021 CREATE /var/lib/docker/overlay2/3f3c7f372897ee684aeacd045697e425a78e4bb3a81fca1da542cf66136b981a/diff/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o
Sat Oct 23 17:17:06 2021 CREATE /var/lib/docker/overlay2/3f3c7f372897ee684aeacd045697e425a78e4bb3a81fca1da542cf66136b981a/merged/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o
Sat Oct 23 17:17:06 2021 MODIFY /var/lib/docker/overlay2/3f3c7f372897ee684aeacd045697e425a78e4bb3a81fca1da542cf66136b981a/merged/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o
Sat Oct 23 17:17:06 2021 MODIFY /var/lib/docker/overlay2/3f3c7f372897ee684aeacd045697e425a78e4bb3a81fca1da542cf66136b981a/diff/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o
Sat Oct 23 17:17:06 2021 MODIFY /var/lib/docker/overlay2/3f3c7f372897ee684aeacd045697e425a78e4bb3a81fca1da542cf66136b981a/merged/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o

EXT4
Sat Oct 23 18:06:53 2021 CREATE /var/lib/docker/overlay2/d8e52e22e4b66d06974bed795d354ae035e6e06e17f281d678d5aff536d64841/diff/tmp/.wine-0/server-f9-80299/ anonmap.yq1eHS
Sat Oct 23 18:06:53 2021 MODIFY /var/lib/docker/overlay2/d8e52e22e4b66d06974bed795d354ae035e6e06e17f281d678d5aff536d64841/diff/tmp/.wine-0/server-f9-80299/ anonmap.yq1eHS

2

This is probably not the answer but there is a warning related to XFS in the documentation.

The overlay and overlay2 drivers are supported on xfs backing filesystems, but only with d_type=true enabled.

Did you configure the XFS filesystem correctly?

3

Yes, I have checked that and it appears to be OK (I meant to include that in my first post, nice catch!).

4

Without d_type=true, overlay2 wouldn’t be available as storage driver.

Since a bind mount is basicly a mount --bind operation from the host to the container, I would be surprised if docker would add additional weight to the operations - as it is just involved in “wiring things together”. The addional overhead might be explained if temporary data is written into the container’s copy-on-write layer, which is managed by overlay2 and the underlying baking fielsystem. But wouldn’t this affect both baking filesystems?

Thus said, I have no idea why you experience such a huge difference in io operation with XFS:

5

I noticed you added new information to your first question. This is interesting. I am not an expert in this on filesystems, especially not XFS, but it should not write in the “merged” folder if it was really an overlay filesystem. And yet this is happening

CREATE   /diff/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o
CREATE /merged/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o
MODIFY /merged/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o
MODIFY   /diff/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o
MODIFY /merged/tmp/.wine-0/server-3a-1075af8/ anonmap.V0n82o
6

Yes, I added to the original post so that anyone new coming into the post would see the info right away rather than having to read down through any/all new comments.

Yup, it all seems very strange to me too. Can’t make heads or tails of it.

7

Did you ever find a solution to your problem? I’m having similar problems, though my setup isn’t identical to yours. Figured it could be related.