Correct way to stop/remove swarm services on new deployment, connectivity problem between containers

I have task for removing docker stack (all services on same stack), this removes the services as well, right?

But on after deploying the new stack, some of services are failing to connect to some of the replicated services. I little bit lost how this could happen, even more as if everything really is removed and swarm is deployed again.
My thought was that there might be running some sort of ghost container of the replicated service, that is causing this. But now I don’t know…

Hmmn, if the new deployment start running before the removing is finished, could that end up in above scenario?

Any ideas?

I’ve noticed a lot of problems with services or containers that aren’t removed correctly. In my stack, some error occurred which stopped 70% of the services running correctly. The only way I managed to solve it was delete the entire /var/lib/docker folder, uninstall and reinstall docker and create the swarm again. This could be related to your problem.

A stack remove is an async process. Sometimes it takes longer to remove all its member object, which collides with immediate redeployments using the same stack name.

This is a snippet of a Makefile I use to deploy/remove swarm stacks:

## This method will deploy a docker-compose template with environment substitution
##
## @param 1 Path to the docker-compose template
## @param Docker Swarm stack name
define deploy
        envsubst "$$(env | sed -e 's/=.*//' -e 's/^/\$$/g')" < $(1) \
                |tee debug.yaml \
                |docker stack deploy -c - $(2) && rm debug.yaml;
endef


## This method will remove a swarm stack and wait until all its services and networks are removed
##
## @param 1 Docker Swarm stack to remove
define remove
        docker stack rm $(1); \
        types="service network config secret"; \
        for type in $$types; do \
                until [ -z "$$(docker $$type ls --filter label=com.docker.stack.namespace=$(1) -q)" ]; do \
                        sleep 1; \
                done; \
        done;
endef