I’m trying to troubleshoot an issue that occurred yesterday when my entire docker swarm stack got reloaded. I know we can set restart policies for individual services but is there a scenario where docker would attempt to reload the whole stack? Another thing to note, the non-swarm containers weren’t affected.
Here are some logs from that time frame if it helps. I’m not sure how to interpret them and whether they are relevant or not. Could someone help me understand what it means?
Apr 02 08:20:54 rwdkr 9a940efc12a6[5983]: [2019-04-02T12:20:54,191][WARN ][o.e.m.j.JvmGcMonitorService] [esFess1] [gc][young][2589885][236414] duration [13.1s], collections [1]/[1.1s], total [1
Apr 02 08:20:54 rwdkr 9a940efc12a6[5983]: [2019-04-02T12:20:54,233][WARN ][o.e.m.j.JvmGcMonitorService] [esFess1] [gc][2589885] overhead, spent [13.1s] collecting in the last [1.1s]
…
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,“@timestamp”:“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:xpack_main@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed f
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:graph@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed from g
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:spaces@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed from
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:searchprofiler@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status chang
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:ml@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed from gree
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:tilemap@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed from
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:watcher@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed from
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:grokdebugger@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:logstash@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed fro
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:beats_management@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status cha
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:index_management@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status cha
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:index_lifecycle_management@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:"
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,“@timestamp”:“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:rollup@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed from
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:remote_clusters@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status chan
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:cross_cluster_replication@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“S
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:reporting@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status changed fr
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:elasticsearch@6.6.0”,“error”],“pid”:6,“state”:“red”,“message”:“Status change
Apr 02 08:20:56 rwdkr 0738d74b502d[5983]: 2019-04-02 12:20:56.385 INFO (Thread-174446) [ x:ta_portal] o.a.s.h.d.JdbcDataSource Time taken for getConnection(): 50
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:xpack_main@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status changed
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:graph@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status changed from
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:searchprofiler@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status chan
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:ml@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status changed from red
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:tilemap@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status changed fro
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:watcher@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status changed fro
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:grokdebugger@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status change
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:logstash@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status changed fr
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:beats_management@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status ch
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:index_management@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status ch
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:index_lifecycle_management@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,“@timestamp”:“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:rollup@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status changed from
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:remote_clusters@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status cha
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:cross_cluster_replication@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:"
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,“@timestamp”:“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:reporting@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:“Status changed f
Apr 02 08:20:56 rwdkr 3c70f60f6558[5983]: {“type”:“log”,”@timestamp":“2019-04-02T12:20:56Z”,“tags”:[“status”,“plugin:spaces@6.6.0”,“info”],“pid”:6,“state”:“green”,“message”:"Status changed from
…
Apr 02 08:20:56 rwdkr 9a940efc12a6[5983]: [2019-04-02T12:20:56,981][INFO ][o.e.c.r.a.DiskThresholdMonitor] [esFess1] low disk watermark [85%] exceeded on [nKj0wcJaRtOcSpNKpZ9_xg][esFess1][/var/
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.080621082-04:00” level=error msg=“agent: session failed” backoff=100ms error=“rpc error: code = DeadlineExceeded desc = context de
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.088025120-04:00” level=error msg=“agent: session failed” backoff=300ms error=“rpc error: code = NotFound desc = node not registere
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.117782313-04:00” level=info msg=“manager selected by agent for new session: { }” module=node/agent node.id=opipztohvp9yhseerumikqc
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.124932638-04:00” level=info msg=“waiting 171.957839ms before registering session” module=node/agent node.id=opipztohvp9yhseerumikq
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.414488697-04:00” level=info msg=“worker opipztohvp9yhseerumikqccl was successfully registered” method=”(*Dispatcher).register”
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.752999462-04:00” level=warning msg=“failed to deactivate service binding for container app_mail.1.l67fnmsiy04v36k7ehf7vpuh6”
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.755412004-04:00” level=warning msg="failed to deactivate service binding for container app_redis.1.a9ttx3j4wthuseajimqox7ufh
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.755553424-04:00” level=warning msg="failed to deactivate service binding for container app_lucee.1.3jt1wqvx7d6o8yqbfgehotdvj
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.755490012-04:00” level=warning msg=“failed to deactivate service binding for container app_web.1.vhgly0ds428veim1py5b8h3lt”
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.757444153-04:00” level=warning msg="failed to deactivate service binding for container app_tasurvey-batch-manager.1.3vbb2fuk
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.761104039-04:00” level=warning msg="failed to deactivate service binding for container app_kibana.1.nnbmnp5qh4xn7dm3cd5jtue0
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.762203246-04:00” level=warning msg="failed to deactivate service binding for container app_redis-commander.1.l79qpe00f9z2t6q
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.764062474-04:00” level=warning msg=“failed to deactivate service binding for container app_solr.1.r01wl9bmdtwedfcluyo74xh7m”
Apr 02 08:20:57 rwdkr dockerd[5983]: time=“2019-04-02T08:20:57.764114330-04:00” level=warning msg=“failed to deactivate service binding for container app_fess.1.1i9vo752dl2agckzqcyzvctyy”