Docker Swarm: How to detect automatic rollback?

Docker Engine Swarm
1

When deploying with docker stack deploy we can provide rollback_config in our docker-stack.yml file. That will give us an automatic rollback for our services.

But there’s a catch. In my CI/CD pipeline, the Deployment job will always be successful, and it’s not wrong in this case because my orders were: “deploy this new version if it fails, rollback!”.

So I’m curious to see how you guys deal with this in your particular setup.

  • Add more steps to deployment to further check if there was a rollback?
  • Monitor using another tool to check and notify the status of the deployment?

Right now I’m working with Gitlab CI, Docker Swarm, Docker remote connection (TLS).

I’ve done this with Drone.io too, but it was basically the same setup.

2
3

Hi @douglasmiranda1 , I am facing the same issue with Bitbucket, did you get any solutions for the same?

Please help here

Thanks
Vivek

4

This post comes up on search results when searching for how to notify on rollback. None of the solutions would work in our env, were too complex, or were 404s so I came up with a simple solution (with the help of ChatGPT).

We have three node swarm clusters running on dedicated servers. I wrote the following script and installed it as a systemd service on one of the Swarm nodes. This will need the jq package installed:

#!/bin/bash
set -euo pipefail

docker events --filter type=service --filter Action=update --format '{{json .}}' | while read -r event; do
  service_name=$(echo "${event}" | jq -r '.Actor.Attributes.name // empty')
  updatestate_new=$(echo "${event}" | jq -r '.Actor.Attributes."updatestate.new" // empty')

  # echo "service: \"${service_name}\", updatestate: \"${updatestate_old}\" -> \"${updatestate_new}\""

  if [[ "${updatestate_new}" == rollback_* ]]; then
    echo "[ERROR]: ${service_name}: ${updatestate_new}"
  fi
done

This emits the [ERROR] log line which gets detected by our log monitoring system which then emails us. One could also put in a echo "${service_name}: ${updatestate_new}" | mail -s "${service_name}: ${updatestate_new}" root or a call to Pushover or whatever.

The systemd service file:

[Unit]
Description=Rollback Monitor
After=network.target

[Service]
ExecStart=/some/path/rollback-monitor.sh
Restart=always
RestartSec=1s
Type=exec

[Install]
WantedBy=multi-user.target

Then systemctl daemon-reload && systemctl enable rollback-monitor && systemctl start rollback-monitor.

Test it by doing:

docker service create --name my-service -p 8080:80 nginx:alpine

docker service update --env-add FOO2=bar2 my-service

docker service rollback my-service

docker service rm my-service

These are the relevant rollback events:

{"Type":"service","Action":"update","Actor":{"ID":"ua3aiwwn24d0isudpb5a4fni8","Attributes":{"name":"my-service","updatestate.new":"rollback_started","updatestate.old":"completed"}},"scope":"swarm","time":1735016641,"timeNano":1735016641235629426}

{"Type":"service","Action":"update","Actor":{"ID":"ua3aiwwn24d0isudpb5a4fni8","Attributes":{"name":"my-service","updatestate.new":"rollback_completed","updatestate.old":"rollback_started"}},"scope":"swarm","time":1735016650,"timeNano":1735016650084353705}

With the first event '.Actor.Attributes."updatestate.new" is rollback_started, the second it’s rollback_complete. I’m alerting right now on ALL rollback events in the case that a rollback fails or something else happens. There isn’t any documentation I can find on all the possible events so I’m being cautious. This will lead to at least two alerts but I’m OK with that as long as I get notified that a rollback happens.

This only has to be deployed on one of the nodes in the Swarm. I do this with Ansible:

- name: systemd | Push out rollback-monitor systemd config
  ansible.builtin.template:
    src: rollback-monitor.service.j2
    dest: /etc/systemd/system/rollback-monitor.service
    mode: '0644'
  when: inventory_hostname == groups["docker_servers"][0]
  notify: systemctl daemon-reload
  tags: systemd

- name: systemd | Enable and start rollback-monitor
  ansible.builtin.service:
    name: rollback-monitor
    state: started
    enabled: true
  when: inventory_hostname == groups["docker_servers"][0]
  tags: systemd

I had also considered writing a small Swarm service that has access to the Docker socket, runs docker events in the container itself, and alerts on rollbacks, but this solution is good enough for now.

Hope this helps someone down the road.