Docker compose, two services using same image: first fails with “no such image”, second runs normally

General
,
1

Hi all,

TL;DR: I have two almost identical services in my compose file except for the name of the service and the published ports. When deploying with docker stack deploy..., why does the first service fail with a no such image error, while the second service using the same image runs perfectly fine?

Full: I have a docker-compose file with two Apache Tomcat services pulling the same image from my private git repository. The only difference between the two services in my docker-compose.yml is the name of the service ( *_dev vs. *_prod ) and the published ports. I deploy this docker-compose file on my swarm using the Gitlab CI with the gitlab-ci.yml . For the deployment of my docker-compose in this gitlab-ci.yml I use two commands:

...
   script:
     - docker pull $REGISTRY:$TAG
     - docker stack deploy -c docker-commpose.yml webapp1 --with registry-auth
...

(I use a docker pull [image] command to have the image on the right node, since my --with-registry-auth is not working properly, but this is not my problem currently).

Now the strange thing is that for the first service, I obtain a No such image: error and the service is stopped, while for the second service everything seems to run perfectly fine. Both services are on the same worker node. This is what I get if I docker ps :

:~$ docker service ps webapp1_tomcat_dev
ID          NAME                       IMAGE          NODE           DESIRED STATE       CURRENT STATE             ERROR                              PORTS
xxx1        webapp1_tomcat_dev.1       url/repo:tag   worker1 node   Shutdown            Rejected 10 minutes ago   "No such image: url/repo:tag@xxx…"
xxx2         \_ webapp1_tomcat_dev.1   url/repo:tag   worker1 node   Shutdown            Rejected 10 minutes ago   "No such image: url/repo:tag@xxx…"
:~$ docker service ps webapp1_tomcat_prod
ID          NAME                    IMAGE          NODE           DESIRED STATE       CURRENT STATE            ERROR               PORTS
xxx3        webapp1_tomcat_prod.1   url/repo:tag   worker1 node   Running             Running 13 minutes ago

I have used the --no-trunc obtain to see that the IMAGE used by *_prod and *_dev is identical.

The restart_policy in my docker-compose explains why the first service fails three minutes after the second service started. Here is my docker-compose:

version: '3.2'

services:
  tomcat_dev:
    image: url/repo:tag
    deploy:
      restart_policy:
        condition: on-failure
        delay: 60s
        window: 120s
        max_attempts: 1
    ports:
      - "8282:8080"

  tomcat_prod:
    image: url/repo:tag
    deploy:
      restart_policy:
        condition: on-failure
        delay: 60s
        window: 120s
        max_attempts: 1
    ports:
      - "8283:8080"

Why does the first service fail with a no such image error? Is it for example just not possible to have two services, that use the same image, work on the same worker node?

(I cannot simply scale-up one service, since I need to upload files to the webapp which are different for production and development - e.g. dev vs prod licenses - and hence I need two distinct services)