Problem
I am using docker-compose to run a number of containers that containerize the components of an API I am working on. One of the containers spins up an Elastic Search cluster, but continues to exit (container logs below), only when it is deployed with a specific image. I am posting this here, because the image works and does not crash for anyone else on my team that uses it, and the image in question is implemented at a production level without a problem, so the issue pertains to my machine and something with docker. I have erased all imaged (docker system prune -a), pullede the desired image again, and still, the container crashes.
System
I am running docker version 3.3 on OS Mojave 10.14.3
The following image causes the container to exit and crash:
docker.elastic.co/elasticsearch/elasticsearch:6.2.2
When I use a different image, such as:
docker.elastic.co/elasticsearch/elasticsearch:6.6.1, the container runs with no problem.
Docker logs on the container show the following:
blockquote
Robert-Dowds-Macbook-Pro:bp-api robertdowd$ docker logs 945326b65bbd
[2019-03-01T00:50:40,677][INFO ][o.e.n.Node ] initializing …
[2019-03-01T00:50:40,824][INFO ][o.e.e.NodeEnvironment ] [Qkskaso] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [38.1gb], net total_space [58.4gb], types [ext4]
[2019-03-01T00:50:40,825][INFO ][o.e.e.NodeEnvironment ] [Qkskaso] heap size [494.9mb], compressed ordinary object pointers [true]
[2019-03-01T00:50:40,914][INFO ][o.e.n.Node ] node name [Qkskaso] derived from node ID [QkskasoATsmSt53BVaEiSQ]; set [node.name] to override
[2019-03-01T00:50:40,914][INFO ][o.e.n.Node ] version[6.2.2], pid[1], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.125-linuxkit/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2019-03-01T00:50:40,914][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.NNlLTdcq, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2019-03-01T00:50:44,675][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [aggs-matrix-stats]
[2019-03-01T00:50:44,675][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [analysis-common]
[2019-03-01T00:50:44,675][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [ingest-common]
[2019-03-01T00:50:44,675][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [lang-expression]
[2019-03-01T00:50:44,676][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [lang-mustache]
[2019-03-01T00:50:44,676][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [lang-painless]
[2019-03-01T00:50:44,676][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [mapper-extras]
[2019-03-01T00:50:44,677][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [parent-join]
[2019-03-01T00:50:44,677][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [percolator]
[2019-03-01T00:50:44,677][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [rank-eval]
[2019-03-01T00:50:44,677][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [reindex]
[2019-03-01T00:50:44,678][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [repository-url]
[2019-03-01T00:50:44,678][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [transport-netty4]
[2019-03-01T00:50:44,678][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded module [tribe]
[2019-03-01T00:50:44,679][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [ingest-geoip]
[2019-03-01T00:50:44,679][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [ingest-user-agent]
[2019-03-01T00:50:44,680][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [x-pack-core]
[2019-03-01T00:50:44,680][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [x-pack-deprecation]
[2019-03-01T00:50:44,680][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [x-pack-graph]
[2019-03-01T00:50:44,681][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [x-pack-logstash]
[2019-03-01T00:50:44,681][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [x-pack-ml]
[2019-03-01T00:50:44,681][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [x-pack-monitoring]
[2019-03-01T00:50:44,681][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [x-pack-security]
[2019-03-01T00:50:44,682][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [x-pack-upgrade]
[2019-03-01T00:50:44,682][INFO ][o.e.p.PluginsService ] [Qkskaso] loaded plugin [x-pack-watcher]
[2019-03-01T00:50:51,047][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/118] [Main.cc@128] controller (64 bit): Version 6.2.2 (Build 61c43548e5e0f2) Copyright (c) 2018 Elasticsearch BV
[2019-03-01T00:50:52,096][ERROR][o.e.g.GatewayMetaState ] [Qkskaso] failed to read local state, exiting…
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:22, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/ul8_KNtgRTiyuD3SkUgZNg/_state/state-22.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:190) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:334) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:87) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.node.Node.(Node.java:424) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.node.Node.(Node.java:246) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.2.jar:6.2.2]
Caused by: java.io.IOException: failed to read [id:22, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/ul8_KNtgRTiyuD3SkUgZNg/_state/state-22.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:327) ~[elasticsearch-6.2.2.jar:6.2.2]
… 15 more
Caused by: java.lang.IllegalArgumentException: Unexpected field [mapping_version]
at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:1259) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1305) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1296) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:202) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:322) ~[elasticsearch-6.2.2.jar:6.2.2]
… 15 more
[2019-03-01T00:50:52,138][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:22, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/ul8_KNtgRTiyuD3SkUgZNg/_state/state-22.st]]; nested: IOException[failed to read [id:22, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/ul8_KNtgRTiyuD3SkUgZNg/_state/state-22.st]]; nested: IllegalArgumentException[Unexpected field [mapping_version]];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.2.jar:6.2.2]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:22, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/ul8_KNtgRTiyuD3SkUgZNg/_state/state-22.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:190) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:334) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:87) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.node.Node.(Node.java:424) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.node.Node.(Node.java:246) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.2.jar:6.2.2]
… 6 more
Caused by: java.io.IOException: failed to read [id:22, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/indices/ul8_KNtgRTiyuD3SkUgZNg/_state/state-22.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:327) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:87) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.node.Node.(Node.java:424) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.node.Node.(Node.java:246) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.2.jar:6.2.2]
… 6 more
Caused by: java.lang.IllegalArgumentException: Unexpected field [mapping_version]
at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:1259) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1305) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1296) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:202) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:322) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:87) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.node.Node.(Node.java:424) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.node.Node.(Node.java:246) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.2.jar:6.2.2]
I am relatively new to using docker, so i’m having trouble understanding why this specific image is only causing problems on my machine.