Docker Community Forums

Share and learn in the Docker community.

Auto test rebuilds image


#1

I have a simple image being built from a GitHub repo (with caching) and have set up auto-test to run a simple test script. It’s now working, but I’ve noticed in the logs that the test stage triggers a complete re-build (from scratch, no caching) of the image, as opposed to using the image that was just built.

Why does it rebuild the image? And how can I get it to use the one it just built?

Automated Build settings:
[Branch] master, latest, docker/Dockerfile, /, Autobuild [on], Build Caching [on]

Contents of docker/docker-compose.test.yml

version: '3'
services:
  sut:
    build:
      context: ..
      dockerfile: docker/Dockerfile
    command: matpower_docker_tests.sh

(Gary Forghetti) #2

Yes, it appears like caching is not being done. I’m checking with a contact I have in Docker Hub support.


(Gary Forghetti) #3

Hi. Sorry for the delay in getting back to you.
Finally got an answer from my Docker Hub contact and then I had to try out a few scenarios.
I replaced the build: . statement in my docker-compose.yml file with an image: statement.
That way the test does not do a 2nd build.

I was also told that the “1st” build will use caching. You should see “Using cache”.
Apparently if you trigger a 2nd build in the docker-compose.yml file, it does not use caching.
But no need to do a build in the docker-compose.yml file because the push does a build.

version: '3.3'

services:

  sut:
    image: gforghetti/helloworld
    command: java -cp /HelloWorld HelloWorld

Also in case you weren’t aware, the autobuild generates the docker image tag from the GitHub version or tag.

I added a 2nd build rule to my Docker Hub Repo for tags. That way I can push up a tag and the Docker Hub autobuild and test use the GitHub tag as the docker image tag.

Building in Docker Cloud's infrastructure...
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '192.30.253.112' to the list of known hosts.
Switched to a new branch '1.0.3'
KernelVersion: 4.4.0-1060-aws
Components: [{u'Version': u'18.03.1-ee-3', u'Name': u'Engine', u'Details': {u'KernelVersion': u'4.4.0-1060-aws', u'Os': u'linux', u'BuildTime': u'2018-08-30T18:42:30.000000000+00:00', u'ApiVersion': u'1.37', u'MinAPIVersion': u'1.12', u'GitCommit': u'b9a5c95', u'Arch': u'amd64', u'Experimental': u'false', u'GoVersion': u'go1.10.2'}}]
Arch: amd64
BuildTime: 2018-08-30T18:42:30.000000000+00:00
ApiVersion: 1.37
Platform: {u'Name': u''}
Version: 18.03.1-ee-3
MinAPIVersion: 1.12
GitCommit: b9a5c95
Os: linux
GoVersion: go1.10.2
Starting build of index.docker.io/gforghetti/helloworld:1.0.3...
Step 1/6 : FROM java:latest
---> d23bdf5b1b1b
Step 2/6 : ARG BUILDDIR="/HelloWorld"
---> Running in fa3e9335dc62
Removing intermediate container fa3e9335dc62
---> 0d2f8f670da2
Step 3/6 : WORKDIR ${BUILDDIR}
Removing intermediate container dfa759041b7c
---> cc09bab775ee
Step 4/6 : CMD java -cp /HelloWorld HelloWorld
---> Running in c182e643b875
Removing intermediate container c182e643b875
---> 330d7cbe370d
Step 5/6 : COPY ./HelloWorld.java ${BUILDDIR}
---> b0f3282b288e
Step 6/6 : RUN javac HelloWorld.java && rm -f HelloWorld.java
---> Running in ab83ebfb96e5
Removing intermediate container ab83ebfb96e5
---> c77e2a9a5749
Successfully built c77e2a9a5749
Successfully tagged gforghetti/helloworld:1.0.3
Starting Test in docker-compose.test.yml...
sut uses an image, skipping
Creating network "b8vttbcxatv8q8uf9wk7d2w_default" with the default driver
Pulling sut (gforghetti/helloworld:)...
latest: Pulling from gforghetti/helloworld
Digest: sha256:aed25dc964883b19ac1225f3ff10b0515b6310b4468d3b580b36e4aeb08dd1a0
Status: Downloaded newer image for gforghetti/helloworld:latest
Creating b8vttbcxatv8q8uf9wk7d2w_sut_1 ...
Creating b8vttbcxatv8q8uf9wk7d2w_sut_1 ... done
Hello World!
Removing b8vttbcxatv8q8uf9wk7d2w_sut_1 ...
Removing b8vttbcxatv8q8uf9wk7d2w_sut_1 ... done Going to remove b8vttbcxatv8q8uf9wk7d2w_sut_1
Tests in docker-compose.test.yml succeeded
Pushing index.docker.io/gforghetti/helloworld:1.0.3...
Done!
Build finished


#4

Thanks.

This solution does avoid rebuilding the image for the test. But there’s a fundamental problem. It doesn’t run the test on the image it just built, but rather on the last image that was pushed with that tag. Since the push of the auto-built image only happens after the test completes successfully, the test runs on the previous commit, not the current one.

Isn’t there a straightforward way to specify in the docker-compose.test.yml that you want to use the image just created by the auto-build?


(Gary Forghetti) #5

Ah, yes you are right.

Successfully built c77e2a9a5749
Successfully tagged gforghetti/helloworld:1.0.4
Starting Test in docker-compose.test.yml...
sut uses an image, skipping
Creating network "bqqi8drzgv8j6qfwiksuxdd_default" with the default driver
Pulling sut (gforghetti/helloworld:)...
latest: Pulling from gforghetti/helloworld
Digest: sha256:aed25dc964883b19ac1225f3ff10b0515b6310b4468d3b580b36e4aeb08dd1a0
Status: Downloaded newer image for gforghetti/helloworld:latest
Creating bqqi8drzgv8j6qfwiksuxdd_sut_1 ...
Creating bqqi8drzgv8j6qfwiksuxdd_sut_1 ... done
Hello World!
Removing bqqi8drzgv8j6qfwiksuxdd_sut_1 ...
Removing bqqi8drzgv8j6qfwiksuxdd_sut_1 ... done Going to remove bqqi8drzgv8j6qfwiksuxdd_sut_1
Tests in docker-compose.test.yml succeeded
Pushing index.docker.io/gforghetti/helloworld:1.0.4...
Done!

Let me dig some more…


(Gary Forghetti) #6

Okay this seems to be working. Added the environment variable CACHE_TAG to the image: statement in the docker-compose.test.yml file.

version: '3.3'

services:

  sut:
    image: gforghetti/helloworld:${CACHE_TAG}
    command: java -cp /HelloWorld HelloWorld
Building in Docker Cloud's infrastructure...
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '192.30.253.112' to the list of known hosts.
Switched to a new branch '1.0.5'
Pulling cache layers for index.docker.io/gforghetti/helloworld:1.0.4...
Done!
KernelVersion: 4.4.0-1060-aws
Components: [{u'Version': u'18.03.1-ee-3', u'Name': u'Engine', u'Details': {u'KernelVersion': u'4.4.0-1060-aws', u'Os': u'linux', u'BuildTime': u'2018-08-30T18:42:30.000000000+00:00', u'ApiVersion': u'1.37', u'MinAPIVersion': u'1.12', u'GitCommit': u'b9a5c95', u'Arch': u'amd64', u'Experimental': u'false', u'GoVersion': u'go1.10.2'}}]
Arch: amd64
BuildTime: 2018-08-30T18:42:30.000000000+00:00
ApiVersion: 1.37
Platform: {u'Name': u''}
Version: 18.03.1-ee-3
MinAPIVersion: 1.12
GitCommit: b9a5c95
Os: linux
GoVersion: go1.10.2
Starting build of index.docker.io/gforghetti/helloworld:1.0.5...
Step 1/6 : FROM java:latest
---> d23bdf5b1b1b
Step 2/6 : ARG BUILDDIR="/HelloWorld"
---> Using cache
---> 788ab16bc0e8
Step 3/6 : WORKDIR ${BUILDDIR}
---> Using cache
---> 94ba121d8ab5
Step 4/6 : CMD java -cp /HelloWorld HelloWorld
---> Using cache
---> f6bf9975aada
Step 5/6 : COPY ./HelloWorld.java ${BUILDDIR}
---> Using cache
---> 7e48808c5867
Step 6/6 : RUN javac HelloWorld.java && rm -f HelloWorld.java
---> Using cache
---> c77e2a9a5749
Successfully built c77e2a9a5749
Successfully tagged gforghetti/helloworld:1.0.5

Notice no pull for the docker image in the Test

Starting Test in docker-compose.test.yml...
sut uses an image, skipping
Creating network "bqqez6ipiec6avggqcyhyrq_default" with the default driver
Creating bqqez6ipiec6avggqcyhyrq_sut_1 ...
Creating bqqez6ipiec6avggqcyhyrq_sut_1 ... done
Hello World!
Removing bqqez6ipiec6avggqcyhyrq_sut_1 ...
Removing bqqez6ipiec6avggqcyhyrq_sut_1 ... done Going to remove bqqez6ipiec6avggqcyhyrq_sut_1
Tests in docker-compose.test.yml succeeded
Pushing index.docker.io/gforghetti/helloworld:1.0.5...
Done!
Build finished

#7

Bingo!

Looks like that’s exactly what I was looking for. Thanks!!