Registry responded to head request with "404 Not Found"

ochstobi · January 20, 2023, 12:54pm

Hello,

I update my docker containers automatically with watchtower.
Since some days I get notifications on one image that says:

Could not do a head request for "koenkk/zigbee2mqtt:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

What is this error, and how to fix this? Is it a problem with the image from koenkk/zigbee2mqtt or is it a problem with watchtower?
The problem is not only for me there are a lot of people having trouble and nobody knows exactly why this is happening since some days.

Problem with head request to docker image

opened 10:56AM - 16 Jan 23 UTC

ochstobi

problem

### What happened? Hello, since the last release there seems to be a problem t…o get information about the docker image. I update the zigbee2mqtt container automatically via watchtower and there is an error since the last update of zigbee2mqtt. The log says: `time="2023-01-13T23:53:08Z" level=warning msg="Could not do a head request for \"koenkk/zigbee2mqtt:latest\", falling back to regular pull." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" time="2023-01-13T23:53:08Z" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"` ### What did you expect to happen? _No response_ ### How to reproduce it (minimal and precise) Run watchtower to update zigbee2mqtt ### Zigbee2MQTT version 1.29.2 ### Adapter firmware version 20200925 ### Adapter zzh! ### Debug log Log before the update: ``` time="2023-01-11T19:53:08Z" level=debug msg="Trying to load authentication credentials." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" time="2023-01-11T19:53:08Z" level=debug msg="No credentials for koenkk found" config_file=/config.json time="2023-01-11T19:53:08Z" level=debug msg="Got image name: koenkk/zigbee2mqtt:latest" time="2023-01-11T19:53:08Z" level=debug msg="Checking if pull is needed" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" time="2023-01-11T19:53:08Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/" time="2023-01-11T19:53:08Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized" time="2023-01-11T19:53:08Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io time="2023-01-11T19:53:08Z" level=debug msg="Setting scope for auth token" image=koenkk/zigbee2mqtt scope="repository:koenkk/zigbee2mqtt:pull" time="2023-01-11T19:53:08Z" level=debug msg="No credentials found." time="2023-01-11T19:53:08Z" level=debug msg="Parsing image ref" host=index.docker.io image=koenkk/zigbee2mqtt normalized="docker.io/koenkk/zigbee2mqtt:latest" tag=latest time="2023-01-11T19:53:08Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/koenkk/zigbee2mqtt/manifests/latest" time="2023-01-11T19:53:09Z" level=debug msg="Found a remote digest to compare with" remote="sha256:62363f6d96ec7046a1d62212a43cb39b6efb4abbdcf16e885a1777b4004e73a6" time="2023-01-11T19:53:09Z" level=debug msg=Comparing local="sha256:62363f6d96ec7046a1d62212a43cb39b6efb4abbdcf16e885a1777b4004e73a6" remote="sha256:62363f6d96ec7046a1d62212a43cb39b6efb4abbdcf16e885a1777b4004e73a6" time="2023-01-11T19:53:09Z" level=debug msg="Found a match" time="2023-01-11T19:53:09Z" level=debug msg="No pull needed. Skipping image." time="2023-01-11T19:53:09Z" level=debug msg="No new images found for /Zigbee2MQTT" ``` Log after the last update: ``` time="2023-01-13T23:53:07Z" level=debug msg="Trying to load authentication credentials." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" time="2023-01-13T23:53:07Z" level=debug msg="No credentials for koenkk found" config_file=/config.json time="2023-01-13T23:53:07Z" level=debug msg="Got image name: koenkk/zigbee2mqtt:latest" time="2023-01-13T23:53:07Z" level=debug msg="Checking if pull is needed" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" time="2023-01-13T23:53:07Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/" time="2023-01-13T23:53:07Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized" time="2023-01-13T23:53:07Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io time="2023-01-13T23:53:07Z" level=debug msg="Setting scope for auth token" image=koenkk/zigbee2mqtt scope="repository:koenkk/zigbee2mqtt:pull" time="2023-01-13T23:53:07Z" level=debug msg="No credentials found." time="2023-01-13T23:53:07Z" level=debug msg="Parsing image ref" host=index.docker.io image=koenkk/zigbee2mqtt normalized="docker.io/koenkk/zigbee2mqtt:latest" tag=latest time="2023-01-13T23:53:07Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/koenkk/zigbee2mqtt/manifests/latest" time="2023-01-13T23:53:08Z" level=warning msg="Could not do a head request for \"koenkk/zigbee2mqtt:latest\", falling back to regular pull." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" time="2023-01-13T23:53:08Z" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" time="2023-01-13T23:53:08Z" level=debug msg="Pulling image" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" time="2023-01-13T23:53:10Z" level=debug msg="No new images found for /Zigbee2MQTT" ```

github.com/containrrr/watchtower

Could not do a head request

opened 05:53AM - 29 Dec 22 UTC

modem7

Type: Bug Priority: Medium Status: Available

### Describe the bug Issues with the crowdsec image. ```yaml crowdsec: … image: crowdsecurity/crowdsec:latest container_name: Crowdsec ``` Error: ```text Could not do a head request for "crowdsecurity/crowdsec:latest", falling back to regular pull. Reason: invalid character 'S' looking for beginning of value ``` ### Steps to reproduce Watchtower config: ```yaml watchtower: image: containrrr/watchtower container_name: Watchtower hostname: Watchtower networks: - isonet labels: - autoheal=true environment: TZ: $TZ WATCHTOWER_CLEANUP: "true" WATCHTOWER_REMOVE_VOLUMES: "false" WATCHTOWER_INCLUDE_STOPPED: "true" WATCHTOWER_NO_STARTUP_MESSAGE: "true" WATCHTOWER_TIMEOUT: 30s WATCHTOWER_SCHEDULE: "0 0 6 * * *" # Everyday at 04:30 WATCHTOWER_NOTIFICATIONS: shoutrrr WATCHTOWER_NOTIFICATION_URL: "telegram://$TGRAM_BOT_TOKEN@telegram?channels=$TGRAM_CHAT_ID" WATCHTOWER_NOTIFICATIONS_LEVEL: info DOCKER_HOST: unix:///var/run/docker.sock volumes: - /var/run/docker.sock:/var/run/docker.sock logging: driver: "local" options: max-size: 10m max-file: "3" restart: always mem_limit: 250m mem_reservation: 60m ``` ### Expected behavior To update. ### Screenshots ![image](https://user-images.githubusercontent.com/4349962/209908753-a43771e7-36f1-48bd-8021-0bf4dcc95b28.png) ### Environment - Ubuntu 22.04 - x86 - Docker version 20.10.22, build 3a2c30b - [Crowdsec](https://hub.docker.com/r/crowdsecurity/crowdsec) ### Your logs ```text time="2022-12-26T06:00:35Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no time="2022-12-27T06:00:39Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no time="2022-12-28T06:00:04Z" level=warning msg="Could not do a head request for \"crowdsecurity/crowdsec:latest\", falling back to regular pull." container=/Crowdsec image="crowdsecurity/crowdsec:latest" time="2022-12-28T06:00:04Z" level=warning msg="Reason: invalid character 'S' looking for beginning of value" container=/Crowdsec image="crowdsecurity/crowdsec:latest" time="2022-12-28T06:00:15Z" level=info msg="Found new linuxserver/bookstack:latest image (f532d54a179c)" time="2022-12-28T06:00:23Z" level=info msg="Found new linuxserver/sonarr:latest image (10d61a66d74e)" time="2022-12-28T06:00:51Z" level=info msg="Found new linuxserver/netbootxyz:latest image (2d07183b1077)" time="2022-12-28T06:00:53Z" level=info msg="Stopping /Netbootxyz (14c6cda3588b) with SIGTERM" time="2022-12-28T06:01:34Z" level=info msg="Stopping /Sonarr (d19f614e614d) with SIGTERM" time="2022-12-28T06:01:45Z" level=info msg="Stopping /Bookstack (7a1565f59d9e) with SIGTERM" time="2022-12-28T06:01:51Z" level=info msg="Creating /Bookstack" time="2022-12-28T06:01:52Z" level=info msg="Creating /Sonarr" time="2022-12-28T06:01:53Z" level=info msg="Creating /Netbootxyz" time="2022-12-28T06:01:59Z" level=info msg="Removing image 455a86d2b778" time="2022-12-28T06:01:59Z" level=info msg="Removing image f4e288f594a6" time="2022-12-28T06:02:00Z" level=info msg="Removing image 4866c5508f26" time="2022-12-28T06:02:00Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=3 notify=no ``` ### Additional context Unsure if related to https://github.com/containrrr/watchtower/issues/1050 Doing a manual run does not trigger any issues. ```text docker run --rm \ -v /var/run/docker.sock:/var/run/docker.sock \ containrrr/watchtower \ --debug --run-once \ Crowdsec time="2022-12-29T05:56:32Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized." time="2022-12-29T05:56:33Z" level=debug msg="Making sure everything is sane before starting" time="2022-12-29T05:56:33Z" level=info msg="Watchtower 1.5.1" time="2022-12-29T05:56:33Z" level=info msg="Using no notifications" time="2022-12-29T05:56:33Z" level=info msg="Only checking containers which name matches \"Crowdsec\"" time="2022-12-29T05:56:33Z" level=info msg="Running a one time update." time="2022-12-29T05:56:33Z" level=debug msg="Checking containers for updated images" time="2022-12-29T05:56:33Z" level=debug msg="Retrieving running containers" time="2022-12-29T05:56:35Z" level=debug msg="Trying to load authentication credentials." container=/Crowdsec image="crowdsecurity/crowdsec:latest" time="2022-12-29T05:56:35Z" level=debug msg="No credentials for crowdsecurity found" config_file=/config.json time="2022-12-29T05:56:35Z" level=debug msg="Got image name: crowdsecurity/crowdsec:latest" time="2022-12-29T05:56:35Z" level=debug msg="Checking if pull is needed" container=/Crowdsec image="crowdsecurity/crowdsec:latest" time="2022-12-29T05:56:35Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/" time="2022-12-29T05:56:35Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized" time="2022-12-29T05:56:35Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io time="2022-12-29T05:56:35Z" level=debug msg="Setting scope for auth token" image=crowdsecurity/crowdsec scope="repository:crowdsecurity/crowdsec:pull" time="2022-12-29T05:56:35Z" level=debug msg="No credentials found." time="2022-12-29T05:56:36Z" level=debug msg="Parsing image ref" host=index.docker.io image=crowdsecurity/crowdsec normalized="docker.io/crowdsecurity/crowdsec:latest" tag=latest time="2022-12-29T05:56:36Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/crowdsecurity/crowdsec/manifests/latest" time="2022-12-29T05:56:36Z" level=debug msg="Found a remote digest to compare with" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66" time="2022-12-29T05:56:36Z" level=debug msg=Comparing local="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66" time="2022-12-29T05:56:36Z" level=debug msg="Found a match" time="2022-12-29T05:56:36Z" level=debug msg="No pull needed. Skipping image." time="2022-12-29T05:56:36Z" level=debug msg="No new images found for /Crowdsec" time="2022-12-29T05:56:36Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=0 notify=no time="2022-12-29T05:56:36Z" level=info msg="Waiting for the notification goroutine to finish" notify=no ```

Thanks in advance.

foorschtbar · January 21, 2023, 9:11am

Same problem here (other images)

clsferguson · January 23, 2023, 1:07am

Also encountering this.

http://docker.io/itzg/minecraft-bedrock-server:latest

This image and zigbee2mqtt

mjv1337 · January 23, 2023, 6:19am

I’m also experiencing this issue with zigbee2mqtt and dozzle now. Other images update just fine. Very peculiar.

23-01-2023 5:58:16
time="2023-01-23T05:58:16+01:00" level=warning msg="Could not do a head request for \"amir20/dozzle:latest\", falling back to regular pull." container=/dozzle image="amir20/dozzle:latest"
23-01-2023 5:58:16
time="2023-01-23T05:58:16+01:00" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/dozzle image="amir20/dozzle:latest"
23-01-2023 5:58:18
time="2023-01-23T05:58:18+01:00" level=warning msg="Could not do a head request for \"koenkk/zigbee2mqtt:latest\", falling back to regular pull." container=/zigbee2mqtt image="koenkk/zigbee2mqtt:latest"
23-01-2023 5:58:18
time="2023-01-23T05:58:18+01:00" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/zigbee2mqtt image="koenkk/zigbee2mqtt:latest"
23-01-2023 5:58:50
time="2023-01-23T05:58:50+01:00" level=info msg="Session done" Failed=0 Scanned=40 Updated=0 notify=no
23-01-2023 6:58:05
time="2023-01-23T06:58:05+01:00" level=warning msg="Could not do a head request for \"amir20/dozzle:latest\", falling back to regular pull." container=/dozzle image="amir20/dozzle:latest"
23-01-2023 6:58:05
time="2023-01-23T06:58:05+01:00" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/dozzle image="amir20/dozzle:latest"
23-01-2023 6:58:07
time="2023-01-23T06:58:07+01:00" level=warning msg="Could not do a head request for \"koenkk/zigbee2mqtt:latest\", falling back to regular pull." container=/zigbee2mqtt image="koenkk/zigbee2mqtt:latest

iameli · January 23, 2023, 4:11pm

Encountering this as well for some subset of our repos.

Failing manifest queries:

livepeer/data:main
livepeer/go-livepeer:master
livepeer/orch-tester:latest
livepeer/task-runner:main
livepeerci/api:dev
livepeerci/api:master

Successful manifest queries:

grafana/grafana-image-renderer:latest
livepeer/data:dev
livepeer/go-livepeer:yf-sender-nonce-debug
livepeer/lpms-runner:master
livepeer/monitoring:latest
livepeer/monitoring:latest-staging
livepeer/pricing-api:latest
livepeer/record-tester:v0.12.28
livepeer/task-runner:dev
livepeer/webrtmp-server:latest
livepeerci/bgpm:0.6.0
livepeerci/bgpm:0.8.0
livepeerci/catalyst:canary
livepeerci/catalyst:main
livepeerci/kube-nodes-connected:latest
livepeerci/mistserver:static-catalyst-updates-drone-amd64
livepeerci/sthc:1.3.0
livepeerci/stream-monitor:3.3.0

IDGI.

whipcracker · January 24, 2023, 7:55am

I’ve started having this problem with 2 x containers out of the 30 running.

Netdata and Autoheal are experiencing the same fault.

I’ve tried logging back into Docker Hub via CLI, adding :latest tags to the image in the compose script, but still nothing.

Using Pihole, which uses 1.1.1.1 and 8.8.8.8 as upstream, otherwise no probs with anything else?

Anyone else found a solution?

Cheers!

ochstobi · January 24, 2023, 8:19am

Hello,

the problem was figured out here: Could not do a head request · Discussion #1529 · containrrr/watchtower · GitHub
It is a problem with the new manifest of docker buildx. You can either disable this warning on watchtower or the maintainers of the images can fix it `docker manifest` does not work with output from buildx v0.10 · Issue #1509 · docker/buildx · GitHub.