Duplicate builds again

Hello. I am one of the committers of Apache Airflow open-source project and we have some troubles with duplicate builds for DockerHub auto-builds.

We have a problem that apparently all our builds (setup via auto-build) seem to be happening twice. We have those build rules now for nightly-master DockerHub TAG:

Docker Tag          Trigger (GitHub TAG)  Dockerfile

master-python3.6    nightly-master        Dockerfile
master-python3.6-ci nightly-master        Dockerfile.ci
master-python3.7    nightly-master        Dockerfile
master-python3.7-ci nightly-master        Dockerfile.ci

But when the tag is pushed (via Github Actions in our nightly build), 8 builds get triggered at the DockerHub - all of the builds above are repeated twice and they are all for the same commit. Here is a dump from last night:

  1. Build in ‘nightly-master:/Dockerfile.ci’ (2fa2c4c6) master-python3.6-ci 2fa2c4c 4 hours ago
  2. Build in ‘nightly-master’ (2fa2c4c6) master-python3.7 2fa2c4c 4 hours ago
  3. Build in ‘nightly-master:/Dockerfile.ci’ (2fa2c4c6) master-python3.7-ci 2fa2c4c 4 hours ago
  4. Build in ‘nightly-master’ (2fa2c4c6) master-python3.6 2fa2c4c 4 hours ago
  5. Build in ‘nightly-master:/Dockerfile.ci’ (2fa2c4c6) master-python3.6-ci 2fa2c4c 4 hours ago
  6. Build in ‘nightly-master’ (2fa2c4c6) master-python3.7 2fa2c4c 4 hours ago
  7. Build in ‘nightly-master:/Dockerfile.ci’ (2fa2c4c6) master-python3.7-ci 2fa2c4c 4 hours ago
  8. Build in ‘nightly-master’ (2fa2c4c6) master-python3.6 2fa2c4c 4 hours ago

Some details about “special” things in our setup

We are using a custom build and hook push.

Since we are using the same scripts for DockerHub and local build our build hook not only builds but also pushes the images to DockerHub. We are pushing a bit more than the target image - we also build a special “manifest” image to check faster if we need to pull CI image and also we are pushing a “build” segment for production image that is used to make the production images smaller. Therefore our push hook does not do anything - we just echo “Skip pushing the image. All images were built and pushed in the build hook already!”. All the builds triggered seem to use those push/build hooks correctly.

We are using image caching

Also - we are using image caching (and this seems to work as well - when our build scripts are starting the most recent image is available for us and we are using it as a cache for the build).

We also have a branch (for our 1.10 airflow line) - v-1-10-stable. I pushed it manually yesterday and we had the same issue. We have 8 builds configured there (4 python versions x 2 builds - CI + Production) and after the push 16 builds were triggered (all builds were duplicated).

Some links that you might find useful:

Has anyone experienced anything like this and maybe Docker Support can also take a look? I sent it to the support but have no confirmation so far.