Docker 19.03 cant see changes in volume

General Discussions
1

Hi,

I have a container service (watchtower) that automatically updates my containers. To be able to check for updates, I must have a valid token. For that, I have a cronjob that automatically updates the auth token in a config.json file. This has been working fine for about 10 months, the updated token has been fetched by the container that uses it.

After upgrading to from 18.09 to 19.03, the container can no longer detect changes in the config.json file. It only successfully gets updated when starting the container, but when trying to manually invalidate the token with vim or nano, the changes arenā€™t reflected.

The command used for setting up the volumes in docker-compose is:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock
  - /root/.docker/config.json:/config.json

Are there any changes in 19.03 that broke this behavior?

2

Probably a security fix to prevent the container user from accessing your root owned directories.
I believe un less you set some permissions flag your container is not allowed to do work with the host directories.
YOu should add the host directories to a GROUP that corresponds to a GID that the container user is apart of.
Iff container user is 1001 and group 1001 then the host directory needs a chown for a group with the same ID.
the command id should display something like this
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),108(lxd),114(netdev)
On the host:
chown :<group with id 1000> /root/.docker/config.json
This is speculation.
let me know iff that is helpful.
Cheers,
Z

3

Good idea, though if it works after (re-)starting the container, it is highly unlikely that permissions are the problem.

The behavior would make sense for configs and secrets as a deployed key never get updated, unless the key is changes.

Though for bind-mounts it doesnā€™t make sense that changed content is not visible inside the vm. It would make sense that it doesnā€™t get updated when the file is replaced, as the file handle used for the mount changes for the new file on the host, but the container still uses the old handle.At least this is the behavior for directoriesā€¦ I am not 100% sure if the behavior is the same for single files.

5

Thanks for your answers.
@zacharyhdb actually your solution might just be what was the reason.
At least Iā€™m able to reproduce the issue in this way:

Give config.json the same GID as user that runs container.
-Can see that the auth token works-
Use nano to edit the token (like changing a 1 to 1 2)
-Can see that the logs gives error messages about invalid token-

Will try to let this run for a while, and see if it works even when the cronjob updates the token.

1 Like
6

As written in my previous reply, I can manage to recreate this manually by changing the GID of the file, and when ā€œinvalidatingā€ the file with nano the change is reflected into the container using the file.

However, when running a docker command to refresh the token, the link breaks.

I read somewhere that this could be fixed by creating a symbolic link, and when starting the container mount the symbolic link instead of the file. In my case this breaks the link. Also when using vim to change the file, the link is brokenā€¦

7

I would suggest, creating a non root userā€¦ to run the containerā€¦
Then adjust your bind-mounts in a non root directory.
also make sure the crontab job is non-root user as well.
IFF that does not workā€¦ Please post some more details on the process and outputs.
GOod luck.

8

This has been solved. Thank you for your replies.

The solution;
Watchtower made is possible to read config.json file from a specified folder, instead of /.
This solution enables mounting the folder instead of file, which might have been broken in some way with the 19.03 update.

Watchtower issue:

Watchtower commit with solution: