Containers constantly exiting with Code 0

Open Source Projects Swarm
1

Hey everyone,

I have a really annoying problem while using docker swarm.

My setup right now consists of two physical servers running proxmox.
Server 1 is an i9-12900 with 32GB 1TB SSD.
Server 2 is an i5-8xxx with 32GB and 1 TB SSD.
Both servers have a windows vm running for dns and dhcp. And each of them are running 2 vms for swarm.
Server 1 runs the manager and worker 1.
Server 2 is running worker 2 and 3.

The problem I am encountering right now is that the containers running on node 2 and 3 both on the same physical machine are constantly exiting with exitcode 0. Containers running on manager and worker 1 have no issues. I am unable right now to pin down where the problem is coming from.
The failing containers come back shortly after as if nothing happend.

Perhaps you guys have some idea where i can start looking for a solution.

Cheers,
ThaMighty

2

We need an under the hood look, before we can have ideas.

Since the services seem to run before they exit with code 0, please share the output of docker service logs {service name} for the affected services.

Furthermore, please share the output of docker service ps --no-trunc {service name} for those services.

3

Will do as soon as they stop again. It is a really weird issue. Sometimes they run for a day or two without problems and sometimes they stop 3 to 4 times a day…

4

These are the results from a service called Package allfro/device-mapping-manager · GitHub. I need this to get my zigbee stick working in docker.

Here are the files.

ID                          NAME                                    IMAGE                                                                                   NODE            DESIRED STATE   CURRENT STATE           ERROR                                                                                                                      PORTS
j0c4pb74f1q58omo8z9ek757q   dmm_dmm.5bt2g0ddhgn1hu87u3vvruzup       docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode02   Running         Running 14 hours ago                                                                                                                               
wg7yyobxnlb5mehqnkv8ulq91    \_ dmm_dmm.5bt2g0ddhgn1hu87u3vvruzup   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode02   Shutdown        Shutdown 14 hours ago                                                                                                                              
ms4a18ucjgrvfltisc7wqdph5    \_ dmm_dmm.5bt2g0ddhgn1hu87u3vvruzup   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode02   Shutdown        Shutdown 14 hours ago                                                                                                                              
hizk2m9k0vwt1vv068daghki1    \_ dmm_dmm.5bt2g0ddhgn1hu87u3vvruzup   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode02   Shutdown        Failed 45 hours ago     "No such container: dmm_dmm.5bt2g0ddhgn1hu87u3vvruzup.hizk2m9k0vwt1vv068daghki1"                                           
qapq9d11rhg0zx5tlawj3mljg    \_ dmm_dmm.5bt2g0ddhgn1hu87u3vvruzup   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode02   Shutdown        Failed 2 days ago       "No such container: dmm_dmm.5bt2g0ddhgn1hu87u3vvruzup.qapq9d11rhg0zx5tlawj3mljg"                                           
0ef0palkuleiarkgnpdvfmfwt   dmm_dmm.slatx3cbh2qyuyti0j2yadzwj       docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsmgr01    Running         Running 14 hours ago                                                                                                                               
j0zgmgqhpoovtar89xmum5hh2    \_ dmm_dmm.slatx3cbh2qyuyti0j2yadzwj   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsmgr01    Shutdown        Shutdown 14 hours ago                                                                                                                              
iifmvpmg9x4youxcughz3n2la    \_ dmm_dmm.slatx3cbh2qyuyti0j2yadzwj   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsmgr01    Shutdown        Shutdown 14 hours ago                                                                                                                              
6p1pdq58i91rx97s91p45atus    \_ dmm_dmm.slatx3cbh2qyuyti0j2yadzwj   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsmgr01    Shutdown        Failed 16 hours ago     "No such container: dmm_dmm.slatx3cbh2qyuyti0j2yadzwj.6p1pdq58i91rx97s91p45atus"                                           
0j8l9531ip1bpb5xssqpe4vtd    \_ dmm_dmm.slatx3cbh2qyuyti0j2yadzwj   docker:latest@sha256:7ff986c816ccc8af25c9f560ca0cba45de2ca2ea2d7099c63099f5539e0d0359   vm001dsmgr01    Shutdown        Failed 4 weeks ago      "task: non-zero exit (255)"                                                                                                
rhkmxghf3smxikgz6xmx8wzo0   dmm_dmm.vf668xg4xu9zodhpy75ml86ht       docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode04   Running         Running 2 hours ago                                                                                                                                
iid2k6jq9sf2exbz5fltf9lck    \_ dmm_dmm.vf668xg4xu9zodhpy75ml86ht   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode04   Shutdown        Shutdown 2 hours ago                                                                                                                               
o9uncno0zf8kmjxyjd8uu1hx5    \_ dmm_dmm.vf668xg4xu9zodhpy75ml86ht   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode04   Shutdown        Shutdown 14 hours ago                                                                                                                              
0pqlupj8gwpoozknbjm9u6fi6    \_ dmm_dmm.vf668xg4xu9zodhpy75ml86ht   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode04   Shutdown        Shutdown 14 hours ago                                                                                                                              
gyu87jqooad474lp56rqcoz73    \_ dmm_dmm.vf668xg4xu9zodhpy75ml86ht   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode04   Shutdown        Shutdown 16 hours ago                                                                                                                              
l3f603namm4ca9spv5q9ssjv6   dmm_dmm.vxmcp7cugyswm2w6iobl2pnsp       docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode03   Running         Running 2 hours ago                                                                                                                                
4nlp8qqw6xst8ua5msy7adjb0    \_ dmm_dmm.vxmcp7cugyswm2w6iobl2pnsp   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode03   Shutdown        Shutdown 2 hours ago                                                                                                                               
papyhx434dq2zt829b62cjca9    \_ dmm_dmm.vxmcp7cugyswm2w6iobl2pnsp   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode03   Shutdown        Shutdown 14 hours ago                                                                                                                              
htqtpfkpg9zvh8qiihm6yzu5a    \_ dmm_dmm.vxmcp7cugyswm2w6iobl2pnsp   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode03   Shutdown        Shutdown 45 hours ago                                                                                                                              
z4g9iejqa96p3meph055xkrrr    \_ dmm_dmm.vxmcp7cugyswm2w6iobl2pnsp   docker:latest@sha256:a690693976550aba640859bb3c3c29eb323a4f53f684c99b2a8282b14a22308b   vm001dsnode03   Shutdown        Rejected 45 hours ago   "network sandbox join failed: subnet sandbox join failed for "10.0.1.0/24": error creating vxlan interface: file exists"

logs last 50 lines because of character limitations.

dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:09:07 /var/lib/docker/volumes/warez_jd_download/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:09:07 1baa510ba82d7960c694c495dfca0d40a633f0a44094f2caa23c815de7d1eb05/477419 requested a volume mount for /etc/localtime at /etc/localtime
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:09:07 /etc/localtime is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:09:07 1baa510ba82d7960c694c495dfca0d40a633f0a44094f2caa23c815de7d1eb05/477419 requested a volume mount for /var/lib/docker/volumes/warez_jd_config/_data at /config
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:09:07 /var/lib/docker/volumes/warez_jd_config/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:10:56 The cgroup version for process 478589 is: 2
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:10:56 Checking mounts for process 478589
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:10:56 5a5a9b3f02465ea2bf9cef29a50cd9840ebd5085eb27e1854517d4261e0966a2/478589 requested a volume mount for /var/lib/docker/volumes/warez_jd_config/_data at /config
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:10:56 /var/lib/docker/volumes/warez_jd_config/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:10:56 5a5a9b3f02465ea2bf9cef29a50cd9840ebd5085eb27e1854517d4261e0966a2/478589 requested a volume mount for /var/lib/docker/volumes/warez_jd_download/_data at /output
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:10:56 /var/lib/docker/volumes/warez_jd_download/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:10:56 5a5a9b3f02465ea2bf9cef29a50cd9840ebd5085eb27e1854517d4261e0966a2/478589 requested a volume mount for /etc/localtime at /etc/localtime
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:10:56 /etc/localtime is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:13:35 The cgroup version for process 479815 is: 2
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:13:35 Checking mounts for process 479815
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:13:35 dae4af669cf50585415fa0099857c1bf8a3f167b1ca04aeb43a220a8291e9056/479815 requested a volume mount for /var/lib/docker/volumes/warez_jd_config/_data at /config
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:13:35 /var/lib/docker/volumes/warez_jd_config/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:13:35 dae4af669cf50585415fa0099857c1bf8a3f167b1ca04aeb43a220a8291e9056/479815 requested a volume mount for /var/lib/docker/volumes/warez_jd_download/_data at /output
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:13:35 /var/lib/docker/volumes/warez_jd_download/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:13:35 dae4af669cf50585415fa0099857c1bf8a3f167b1ca04aeb43a220a8291e9056/479815 requested a volume mount for /etc/localtime at /etc/localtime
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/09 20:13:35 /etc/localtime is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:35 The cgroup version for process 537703 is: 2
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:35 Checking mounts for process 537703
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:35 a24ff238ed9b35a005fd231a02dab9a9a2269e2afb3000f7e780c6ebd71bd54f/537703 requested a volume mount for /var/lib/docker/volumes/paperless_redisdata/_data at /data
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:35 /var/lib/docker/volumes/paperless_redisdata/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:42 The cgroup version for process 537869 is: 2
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:42 Checking mounts for process 537869
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:42 dae531f415c1b911b49c21b2c82e83ed66fa52e41bed5989d53f21b6cc605c24/537869 requested a volume mount for /var/lib/docker/volumes/heimautomatisierung_mosquitto_data/_data at /mosquitto/data
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:42 /var/lib/docker/volumes/heimautomatisierung_mosquitto_data/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:42 dae531f415c1b911b49c21b2c82e83ed66fa52e41bed5989d53f21b6cc605c24/537869 requested a volume mount for /var/lib/docker/volumes/heimautomatisierung_mosquitto_log/_data at /mosquitto/log
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:42 /var/lib/docker/volumes/heimautomatisierung_mosquitto_log/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:42 dae531f415c1b911b49c21b2c82e83ed66fa52e41bed5989d53f21b6cc605c24/537869 requested a volume mount for /var/lib/docker/volumes/heimautomatisierung_mosquitto_config/_data at /mosquitto/config
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:51:42 /var/lib/docker/volumes/heimautomatisierung_mosquitto_config/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:52:03 The cgroup version for process 538208 is: 2
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:52:03 Checking mounts for process 538208
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:52:03 55f159e4c6016751c395e5bca156882650f70cf0a903ccb2ba1701c2a648ddd1/538208 requested a volume mount for /var/lib/docker/volumes/unifi_db_data/_data at /data/db
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:52:03 /var/lib/docker/volumes/unifi_db_data/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:52:03 55f159e4c6016751c395e5bca156882650f70cf0a903ccb2ba1701c2a648ddd1/538208 requested a volume mount for /var/lib/docker/volumes/ca57c36558ee045ac38d5a1a6b02fb2d8d104ec3c3860e350b5c8b0a9cb0b816/_data at /data/configdb
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:52:03 /var/lib/docker/volumes/ca57c36558ee045ac38d5a1a6b02fb2d8d104ec3c3860e350b5c8b0a9cb0b816/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:53:17 The cgroup version for process 538656 is: 2
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:53:17 Checking mounts for process 538656
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:53:17 bdae9d6310fbb0d79f85f64d9ed55ea0e20db115a1159d280fac40d4b39bb34a/538656 requested a volume mount for /var/lib/docker/volumes/heimautomatisierung_homeassistant_data/_data at /config
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:53:17 /var/lib/docker/volumes/heimautomatisierung_homeassistant_data/_data is not a device... skipping
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:53:17 bdae9d6310fbb0d79f85f64d9ed55ea0e20db115a1159d280fac40d4b39bb34a/538656 requested a volume mount for /etc/localtime at /etc/localtime
dmm_dmm.0.j0c4pb74f1q5@vm001dsnode02    | 2024/08/10 05:53:17 /etc/localtime is not a device... skipping
error from daemon in stream: Error grabbing logs: rpc error: code = Unknown desc = warning: incomplete log stream. some logs could not be retrieved for the following reasons: node vxmcp7cugyswm2w6iobl2pnsp is not available, node vf668xg4xu9zodhpy75ml86ht is not available
5

Found out that one of my pve hosts had a wrong workgroup that didnt exist anymore since than no more outages