Pulling and Tagging Latest Images

I am building a CI/CD pipeline. My organization is using two container management systems. ShinyProxy and Portainer. I am able to pull and deploy images with Portainer using webhooks and a stack, however ShinyProxy is unable to find the images that Portainer is pulling. It seems that the tags are not getting updated. The current version of Docker is 20.10.6+azure.

Is there any way to specify what an image name should be in a compose.yml? It is confusing to me that a container management system that allows you to abstractly mount just about anything to a container. Renaming directories in the mounts for example, would not allow for abstracting away from the SHA tags in a docker environment. Are developers really going through and updating SHA tags any time they need to redeploy a part of a stack? I’m sure ShinyProxy isn’t the only application that launches docker containers. It could possibly be a ShinyProxy issue, are they supposed to query the docker api to find what is latest? It could be a Portainer issue, are they pulling in a weird way and not tagging the images with their version of compose. It just seems there should be some way in a docker stack file to specify that what the services image tag should be abstracting away that SHA for any downstream processes that really don’t need to know what the SHA is, so the services don’t break.

Obviously you are aware that using the ‘latest’ tag for a docker image is seen by many as an anti-pattern (because it is mutable, although so are any tags that are created primarily for human readability). When an image is pulled, the tag which docker uses is the hash, hence why you see that in the console output. If you want to use a tag that you feel is immutable (whether that’s strictly the case or not) such as a git commit hash you can do so. As for hard coding that into a compose file, that can be ok in some circumstances, but I would expect that if you are building and/or using images within a CI pipeline you would want that to be dynamic. So in your compose file you might represent the tag as an environment variable which you set at an earlier stage in the pipeline.

The point is if I have a container, that interacts with the container systems, spawning containers or services. Updating the sha tags in the docker-compose, breaks the functionality of the image in the container(which has to be aware of the Sha tags at this point). There are a couple of systems that do this off the top of my head ShinyProxy, JupyterHub

There is clearly a solution to that problem, which is solved in many other instances using a compose or stack file to map things outside the container to things inside the container. Why should sha tags be any different?

There are probably other issues that docker is navigating. Like swarm namespaces and what not, but I have to say. It’s not a good look. What is wrong with in the compose/stack file saying. Image A:Sha@XXXX is named mylatest:boondogle, then ensuring that that mylatest:boondogle is available on the all the swarm nodes tagged as such prior to launching of the compose or stack file, so that process in a stack or compose can interact with each other?

Is that even on the road map?

I assume you are looking for something in this direction: Docker Tagging: Best practices for tagging and versioning docker images – Steve Lasker

The thing is having multiple services having different digests for the same tag is exactly what I want. When there are numerous applications, I don’t want to have to restart the proxy server to update the digests every time someone publishes a change to one of the apps. Yes, on occasion, I might want to purge some of the running containers for an app, not hard to do.

A tag can not point to different digests at a point in time. With each image that pushes a tag, the digest will point to the last pushed images digest.

You could map the behavior of how dyndns services update a dns record:

  • image tag ~= domain name
  • image digest ~= current ip

The same way as each update of a dyndns domain will update the a-record to point to the current ip, and as such only resolve the current ip, a tag will resolve to the image digest of the last pushed image for that tag.

I still feel explicit examples would have prevented a lot of ambiguity here.

So, I am having an issue very similar to this with an instance of ShinyProxy. See how ShinyProxy has an application.yml file in addition to the stack file. Using tags in the ShinyProxy never seem to get the updated latest images when I update a stack. Am I supposed to go through and edit the tags and build the docker image for ShinyProxy every time one of the apps it is running has a new container?

Failed to start container when using docker swarm (1 manager + 1 worker) · Issue #214 · openanalytics/shinyproxy · GitHub