When checking the network with ‘docker network inspect --verbose’ we noticed that some services have ghost/dead task listed as task for the service. These ghost-tasks are not visible when checking the service with docker service ps.
Like in here there is service marked with green that is not listed on docker ps.
Also this dead tasks is not visible to docker inspect ^long id of the task^ → returning “Error: No such object:”
Here’s what we have tried:
To remove the service from the stack with: docker service rm ^id^
—> the task remains in network
To use the docker dperny/tasknuke (docker run -v /var/run/docker/swarm/control.sock:/var/run/swarmd.sock dperny/tasknuke ) → id not foud
docker node drain ( docker node update --availability drain) to the worker/host where the dead task claims to be → the task remains in network
Is there any other solutions to try to kill that orphan/dead task?
A bit clueless now to what to try/do. I am thinking about removing the network, but there are few things that come to mind. As the other tests for the services failed, that orphan task might stand in the way when trying to remove the network, as there “is” something still attached in it. And when thinking about cleaning up the whole swarm, makes me worried about the volumes/data.
Docker 17.09 in use, I know it is old, but stuck with it for another reasons. What I have read similar problems have happened with newer versions of docker also.
I would have said that it’s the service task history.
But if it was the case, you should see the Stopped tasks with docker service ps {service name}, which you appearently don’t. Judged by the output of docker network inspect --verbose the task id 2 is used twice, which could indicate a stopped task from the task history. But again, if it’s the case it should have been listed with docker service ps {service name}.
When it commes to docker ps: it is no surprise that without the -aargument stopped/exited containers are not listed.
Your docker version is now out of support for years. I can not confirm that I see this error, nor have I ever seen it in the couple of years of using swarm. It could be very well a bug of that old version.
I am not sure how to help you. I hope you do find a solution for your problem.
Thank you once again for your quick response.
You are right about the task-history, the one is not visible in there, so I believe limiting task history wont do nothing as it is only length of the “history”. And not about running task, as it seems from the network point of view. That is why I was hoping that node drain would somehow make an effect to that task as well, but like remove service, it affected to only the tasks listed for the service.
And yes, ‘docker ps -a’ did/does not list anything more on the node, which is listed as a host of the dead task.
…So frustrating as it is visible on the network, but no way to edit/kill it away, cleaning the whole network seems to only way out
