I have a swarm setup with AWS auto-scaling, that means hosts come and go all the time. I’m using an AWS ECR registry, and its tokens expire about 12 hours after generated.
I’m noticing that after I create the service, new nodes in the swarm have their containers started just fine, and it works for the entire day. The next day, I see only a few of the nodes that should be running, and docker reports the service with something like “3/30” saying it notices 30 hosts that should be running but didn’t start the containers.
Then I call
docker service update --wth-registry-auth myservice and suddenly it starts all missing containers, and keeps working for the day, and fails again at night.
Is it correct that docker doesn’t update the token after the
docker service create/update --with-registry-auth commands are issues? If so, is there anything I can do to keep it working?