Can't start elasticsearch with mounted data directory - [f]stat[v]fs family of functions

Docker Desktop for Mac
,
1

Expected behavior

An elasticsearch container with a mounted data volume should start.

Actual behavior

The elasticsearch server fails to start and provides a stack trace in the logs (see below).

Information

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.0-beta6
Running diagnostic tests:
[OK]      docker-cli
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160407-223554.tar.gz.
Your unique id in bugsnag is: BEF8B2D6-1F87-433C-A4EB-DE4E84FC677F
Please quote this in all correspondence.

Steps to reproduce the behavior

Run the following command:

docker run -v "$PWD/esdata":/usr/share/elasticsearch/data elasticsearch

The output is:

[2016-04-08 02:04:58,623][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: your kernel is buggy and you should upgrade
[2016-04-08 02:04:58,879][INFO ][node                     ] [Richard Rider] version[2.3.1], pid[1], build[bd98092/2016-04-04T12:25:05Z]
[2016-04-08 02:04:58,879][INFO ][node                     ] [Richard Rider] initializing ...
[2016-04-08 02:04:59,506][INFO ][plugins                  ] [Richard Rider] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
Exception in thread "main" java.lang.IllegalStateException: Failed to created node environment
Likely root cause: java.nio.file.FileSystemException: /usr/share/elasticsearch/data: Function not implemented
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileStore.readAttributes(UnixFileStore.java:111)
	at sun.nio.fs.UnixFileStore.getTotalSpace(UnixFileStore.java:118)
	at org.elasticsearch.env.ESFileStore.getTotalSpace(ESFileStore.java:198)
	at org.elasticsearch.monitor.fs.FsProbe.getFSInfo(FsProbe.java:59)
	at org.elasticsearch.env.NodeEnvironment.maybeLogPathDetails(NodeEnvironment.java:269)
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:211)
	at org.elasticsearch.node.Node.<init>(Node.java:165)
	at org.elasticsearch.node.Node.<init>(Node.java:140)
	at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.

The above command works properly running Docker under Linux. It also works when omitting the volume flag.

3 Likes
2

Hello ectsue

I just tried to replicate the issue and dont get the same results

here is my test in an attempt to replicate

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.0-beta6
Running diagnostic tests:
[OK] docker-cli
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x

MacBook-Air:ELK bgrissin$ docker volume ls
DRIVER VOLUME NAME
local 08970ae7a768d73264cf90b11ffb43780444a02a6cf663736bccac17cb8c9a7e
local 8bfc5e0d4c4dcd3673cb12d60d36755af18ca84a616aad5f84d6cc4cae611c76
local elk_ucp-elasticsearch-data
local monitorlogging_ucp-elasticsearch-data
local ucp-client-root-ca
local ucp-cluster-root-ca
local ucp-controller-client-certs
local ucp-controller-server-certs
local ucp-elasticsearch-data
local ucp-kv
local ucp-kv-certs
local ucp-node-certs

MacBook-Air:ELK bgrissin$ docker run -d -v ucp-elasticsearch-data:/usr/share/elasticsearch/data -p 9200:9200 -p 9300:9300 elasticsearch:latest elasticsearch -Des.network.host=0.0.0.0
214e7f0782574744424f014864fa6c7f92cf1380a3f4394c8d59298e37cce23d
MacBook-Air:ELK bgrissin$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
214e7f078257 elasticsearch:latest “/docker-entrypoint.s” 3 seconds ago Up 2 seconds 192.168.64.5:9200->9200/tcp, 192.168.64.5:9300->9300/tcp elegant_pasteur

It appears to be an issue with a plugin from the logs

3

more specific log detail in my test

docker run -v ucp-elasticsearch-data:/usr/share/elasticsearch/data -p 9200:9200 -p 9300:9300 elasticsearch:latest elasticsearch -Des.network.host=0.0.0.0
[2016-04-08 11:18:07,503][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: your kernel is buggy and you should upgrade
[2016-04-08 11:18:07,661][INFO ][node ] [Ch’od] version[2.3.1], pid[1], build[bd98092/2016-04-04T12:25:05Z]
[2016-04-08 11:18:07,661][INFO ][node ] [Ch’od] initializing …
[2016-04-08 11:18:08,201][INFO ][plugins ] [Ch’od] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-04-08 11:18:08,231][INFO ][env ] [Ch’od] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda2)]], net usable_space [55.5gb], net total_space [59gb], spins? [possibly], types [ext4]
[2016-04-08 11:18:08,232][INFO ][env ] [Ch’od] heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-04-08 11:18:10,290][INFO ][node ] [Ch’od] initialized
[2016-04-08 11:18:10,290][INFO ][node ] [Ch’od] starting …
[2016-04-08 11:18:10,386][INFO ][transport ] [Ch’od] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
[2016-04-08 11:18:10,403][INFO ][discovery ] [Ch’od] elasticsearch/zpEDq8beSfKatvpNA5eUxA
[2016-04-08 11:18:13,511][INFO ][cluster.service ] [Ch’od] new_master {Ch’od}{zpEDq8beSfKatvpNA5eUxA}{172.17.0.2}{172.17.0.2:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-04-08 11:18:13,544][INFO ][http ] [Ch’od] publish_address {172.17.0.2:9200}, bound_addresses {[::]:9200}
[2016-04-08 11:18:13,545][INFO ][node ] [Ch’od] started
[2016-04-08 11:18:13,575][INFO ][gateway ] [Ch’od] recovered [0] indices into cluster_state

4

Can you run a docker volume ls and also check for space and privs for the mount your using locally?

5

Hi Evan.

Thanks for your bug report. This is a known issue with Docker for Mac.

The [f]stat[v]fs family of functions is not yet supported by the Docker for Mac file-sharing solution, osxfs. We have plans to implement the necessary functionality to support both querying file system metadata and sharing the OS X mount table with bind-mounting containers. This support should arrive within the next 2 to 3 weeks so keep watching the changelog, updating your installation, and trying again once more mount point support lands!

In the meantime, I’ll add your use case to our compatibility test suite to ensure that, once we can support it, we do not regress support.

Thanks for your participation in the beta programme and helping to make Docker for Mac more awesome!

– Dave

7 Likes
6

Thank you very much! I’ll be on the lookout for the updates.

1 Like