How to deploy docker-compose file on Linux deployment server from Jenkins build job?

My Jenkins build job compiles the code, then creates two docker images (each a separate piece of the application) and publishes them.

Then it comes time to deploying them to on-premise Linux servers. I have a docker-compose yaml file that supposed to get everything up and running.

My problem is, I am looking for a way to do three things:

  1. Need to place the compose file in the deployment server(s) which are not under Jenkins control, allowing perhaps some daemon to process it , as docker images are already uploaded to docker repo. I have root access to the deployment servers, ssh credentials are in Jenkins, and I am able to install things on the deployment servers. So, I tried to do that with scp’ing using a special service user, but it does not have permissions to create a directory (eg: /app ) - I tried working around by adding that user to the root group ( usermod -aG root username ) - still same error.

  2. Once the compose file is in, I assume I will be able to deploy (as that same user is in docker group too) with daemon flag, but what about scenario where the app is already running, and I want to just deploy an update (essentially pull newer version of the images), and how to make sure docker will always keep the deployed stack always running, even if something crashes?

  3. If the running container(s) crashes persistently, how to avoid infinite loop, and get notified when say, it tried to restart it 5 times already?