The log-opts in the /etc/docker/daemon.json is not relaoded

General Discussions
1

Recently I want to use loki-log-driver to ship logs to the loki server. The docker-deamon is controlled by systemd.

The /etc/docker/daemon.json is given below

{
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://loki-log-service:3100/loki/api/v1/push",
    "mode": "non-blocking",
    "loki-retries": "2",
    "loki-external-labels": "container_name={{.Name}}"
  }
}

I changed loki-external-labels to "container_name={{.Name}},job={{.Config.Labels.__meta_kubernetes_pod_container_name}}", saved the change, and executed systemctl restart docker. However the new docker daemon didn’t have the modifed value of loki-external-labels .

docker-version

Client: Docker Engine - Community
 Version:           20.10.0
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        7287ab3
 Built:             Tue Dec  8 18:59:27 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:57:25 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
2

Your containers should have those log-opts too. Have you recreated the containers from which you want to send the logs with the new labels?

3

The commands below work as expected sometimes.

sudo systemctl daemon-reload
sudo systemctl restart docker
4

This is just restarting Docker. I mentioned “recreating” containers, not restarting. Every container has it’s own log settings. The daemon config just contains the default which is copied to new containers, but it will not change existing containers.