Filesystem of a container become read-only after running an application

General Discussions
1

Hello folks,

I’m having strange problem with file system of a container. I observed this issue in following running containers – opensuse, ubuntu (official) and sles12 all latest versions. Containers are run on VMWare VM host with sles12.

Issue is following: After I run the container /proc/mounts file shows that FS is read/write

cat /proc/mounts

…
/dev/mapper/docker-254:5-1452-701284ae46bd92f54ddaa7914e9043ecb678c450f208dc1074fd2ba1a8865508
/ ext4 rw,relatime,discard,stripe=16,data=ordered
0 0

Then when I run an application that is using massively FS for writing some files on it, file system of the container become read-only and /proc/mounts shows following

cat /proc/mounts

…
/dev/mapper/docker-254:5-1452-701284ae46bd92f54ddaa7914e9043ecb678c450f208dc1074fd2ba1a8865508
/ ext4 ro,relatime,discard,stripe=16,data=ordered
0 0

Then when I type dmesg it shows following

dmesg

…
[ 3422.254633] loop: Write error at byte offset 1476132864,length 4096.
[ 3422.254635] EXT4-fs warning (device dm-7): ext4_end_bio:316: I/O error -28 writing to inode 268149 (offset 54530048 size 983040 starting block 60656)
[ 3424.671444] loop: Write error at byte offset 1476198400,length 4096.
[ 3424.671455] loop: Write error at byte offset 1476263936,length 4096.
[ 3424.671462] loop: Write error at byte offset 1476329472,length 4096.
[ 3424.671468] loop: Write error at byte offset 1476395008,length 4096.
[ 3424.671476] loop: Write error at byte offset 1476460544,length 4096.
[ 3424.671612] loop: Write error at byte offset 1476526080, length 4096.
[ 3424.671670] Aborting journal on device dm-7-8.
[ 3424.671729] EXT4-fs error (device dm-7) in ext4_reserve_inode_write:4894: Journal has aborted
[ 3424.671763] EXT4-fs error (device dm-7) in ext4_dirty_inode:5013: Journal has aborted
[ 3424.671781] EXT4-fs error (device dm-7) in ext4_da_write_end:2815: Journal has aborted
[ 3427.234316] EXT4-fs error (device dm-7): ext4_journal_check_start:56: Detected aborted journal
[ 3427.234324] EXT4-fs (dm-7): Remounting filesystem read-only
[ 3427.234344] EXT4-fs (dm-7): ext4_writepages: jbd2_start: 13312 pages, ino 268150; err -30

Here is used docker version:

#docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.3.3
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.3.3
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

Can someone help me bring container working?

Thank you and Regards

1 Like
2

I found a simple way to reproduce the issue. Execution of the below script turns read/write FS mode
to read-only.

The idea of script is to simulate mass writing of files to FS of the container (in this case it creates 50000 text files each one having 100 lines) on file system. After a while FS of the container stands read-only and script stops working. By the way I observed that similar faire you can reproduce if you just echo some text many times (50000 ) on the console.

Here is script source.

#!/bin/bash
for (( c=1; c<=5000000; c++ ))
do
echo “Some text $c” >> outfile$(( c / 100 )).out
done

Regards,

3

@geletok I am running into the same issue with Docker 1.9.1, have you found a solution to this problem?