Cassandra failed to run with "Insufficient disk space to write" error while using a docker volume

Docker Desktop
1

Expected behavior

Volumes should be used without any issue.

Actual behavior

Seen “Insufficient disk space to write” while using a docker volume.

Information

OS X: version 10.11.5 (build: 15F34)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160524-191032.tar.gz
Most specific failure is: No error was detected
Your unique id is: 5CEDF9D3-C715-4375-8F1E-6D3F5B1FC990

I have the following docker-compose.yaml:

version: '2'
services:
  cassandra:
    image: cassandra:2.1
    volumes:
     - "$PWD/data/cassandra:/var/lib/cassandra"

Steps to reproduce the behavior

  1. docker-compose up -d
  2. Cassandra has failed to start with the following log:

cassandra_1 | INFO 13:05:26 Couldn’t detect any schema definitions in local storage.
cassandra_1 | INFO 13:05:26 To create keyspaces and column families, see ‘help create’ in cqlsh.
cassandra_1 | INFO 13:05:26 Enqueuing flush of local: 794 (0%) on-heap, 0 (0%) off-heap
cassandra_1 | ERROR 13:05:26 Exception encountered during startup
cassandra_1 | java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: FSWriteError in
cassandra_1 | at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2752) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.db.ColumnFamilyStore.truncateBlocking(ColumnFamilyStore.java:2713) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.db.SystemKeyspace.discardCompactionsInProgress(SystemKeyspace.java:278) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:310) [apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: FSWriteError in
cassandra_1 | at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:376) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.db.SystemKeyspace.forceBlockingFlush(SystemKeyspace.java:468) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.db.SystemKeyspace.saveTruncationRecord(SystemKeyspace.java:306) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.db.ColumnFamilyStore$11.run(ColumnFamilyStore.java:2707) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_101]
cassandra_1 | at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2748) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | … 5 common frames omitted
cassandra_1 | Caused by: java.util.concurrent.ExecutionException: FSWriteError in
cassandra_1 | at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_101]
cassandra_1 | at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:1.7.0_101]
cassandra_1 | at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | … 10 common frames omitted
cassandra_1 | Caused by: org.apache.cassandra.io.FSWriteError: java.io.IOException: Insufficient disk space to write 182 bytes
cassandra_1 | at org.apache.cassandra.io.util.DiskAwareRunnable.getWriteDirectory(DiskAwareRunnable.java:32) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:332) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na]
cassandra_1 | at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1127) ~[apache-cassandra-2.1.14.jar:2.1.14]
cassandra_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_101]
cassandra_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_101]
cassandra_1 | at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_101]
cassandra_1 | Caused by: java.io.IOException: Insufficient disk space to write 182 bytes
cassandra_1 | … 8 common frames omitted

2

Issue is still reproducible in 1.11.1-beta13 (build: 7975)

3

What does $PWD expand to? What directory are you running this in? Please see https://beta.docker.com/docs/mac/osxfs/#namespaces for details regarding what names from the host are available unmodified in the guest.

4

it expands to /Users/pnuzhdin/some-project.

5

This issue is due to osxfs currently missing a statfs implementation. See https://forums.docker.com/t/cant-start-elasticsearch-with-mounted-data-directory-f-stat-v-fs-family-of-functions/8705. The Cassandra 2.1 error message is misleading but if you run Cassandra 2.2, you will see:

cassandra_1  | Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Unable to check disk space available to /var/lib/cassandra/commitlog. Perhaps the Cassandra user does not have the necessary permissions
cassandra_1  | org.apache.cassandra.exceptions.ConfigurationException: Unable to check disk space available to /var/lib/cassandra/commitlog. Perhaps the Cassandra user does not have the necessary permissions
cassandra_1  |     at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:522)
cassandra_1  |     at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:119)
cassandra_1  |     at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:491)
cassandra_1  |     at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625)
cassandra_1  | Caused by: java.nio.file.FileSystemException: /var/lib/cassandra/commitlog: Function not implemented
cassandra_1  |     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
cassandra_1  |     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
cassandra_1  |     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
cassandra_1  |     at sun.nio.fs.UnixFileStore.readAttributes(UnixFileStore.java:111)
cassandra_1  |     at sun.nio.fs.UnixFileStore.getTotalSpace(UnixFileStore.java:118)
cassandra_1  |     at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:517)
cassandra_1  |     ... 3 more

and so you are correct and it is exactly the same issue as https://forums.docker.com/t/mounting-volume-with-mongodb/13606/2 which we are actively working on.

Thanks for the report and for participating in the Docker for Mac Beta!

David

1 Like