Cant communicate with logging driver loki, unknown authority

I’m attempting to use Grafana Loki as a logging driver. the plugin is installed and working, it attempts a connection and returns:

Mar 30 00:54:32 sarin dockerd[9169]: time="2025-03-30T00:54:32Z" level=info msg="level=error ts=2025-03-30T00:54:32.85691463Z caller=client.go:360 container_id=95f9965fb5195273006156ff0d194c6e85dc86733e63afb0b2b17070636d91a8 component=client host=loki.sarin.lan msg=\"final error sending batch\" status=-1 error=\"Post \\\"https://loki.sarin.lan/loki/api/v1/push\\\": x509: certificate signed by unknown authority\"" plugin=80d59480c3ce07d340ac059d28700f38e9086c2af9c5f39b70aa40218bf5a4bf

using This link I’ve installed and verified the existence of ma CA-cert.

verified it again using curl --verbose
< * SSL certificate verify ok.

as far as I’m aware I’ve properly installed and verified that my server recognizes the cert as valid. and the docker docs here seem to suggest that’s all i should have needed to do.

I would either like docker to recognize the cert as valid, or just ignore the “insecure” cert and proceed with the connection.

Docker does nothing with your cert. The documentation you linked contains multiple suggestions for different use cases. Which one did you do? You need to configure the container, not your host operating system. The CA certs are stored in a CA bundle file, but it could be located in different folders in different distributions, but it is usually at /etc/ssl/certs as the guide you linked first suggests. It is also important that it is up to the command or library to use that bundle or not, so when you use a specific software like Grafana, you need to check its documentation first and its community forum if there is any.

https://community.grafana.com/t/adding-a-custom-ca-root-chain-in-trusted-certificates-in-grafana-docker/18059/2

If you don’t want to run the update command (which could also be different in different distributions) as the first page suggests, you can actually copy the original bundle file out from the image, append your CA cert at the end and mount the cert bundle file back when you start the container.

grafana connects to loki with no problem, I believe this to be a docker issue because the error i listed from the previous post is from this command: journalctl -f -u docker.service.

I believe docker is refusing to send log data to grafana Loki because of the unknown authority. ive verified that the cert is properly installed with the two following commands.

ls /etc/ssl/certs/ | grep minica
minica.pem

cat /etc/ssl/certs/ca-certificates.crt | grep GAWl1DQAkXhRFVMd2w
3n5iUapczFumLiOcjeZoxP5lGAWl1DQAkXhRFVMd2w==

I’ve restarted docker and when that didn’t work rebooted the host and it still does not work.

Then I misunderstood your issue, sorry for that. Now that I read your first post again, it is more obvious you were writing about a logging driver. The officially supported Docker logging drivers are here:

https://docs.docker.com/engine/logging/configure/#supported-logging-drivers

You can still ask about others, but these are what most of the users will know about and if you use something else, it is important to share what you are using. In this case I assume it is this one made and supported By Grafana Labs:

https://grafana.com/docs/loki/latest/send-data/docker-driver/

That page also mentions known issues, but not related to certificates. I have not used this logging driver yet, so I don’t know if the issue is related to that or Docker directly.

Please, share the output of the following commands. You can delete anything from the output that you would not share:

docker version
docker info

I also assume you installed the official Docker CE, but it would help if you could also share the output of the following commands:

dpkg -l 'docker*' | grep '^ii'
snap list docker

Please, also share how you configured the CA cert on the host. even though curl strusts it, someone could catch something that is missing compared to the guide from the documentation.

If we can’t help, an alternative way could be using the journald logging driver or the default json driver with proper log rotation and limits, and use promtail for example to read the json files. That way you don’t have to use that logging driver. Promtail is also linked in the Grafana documentation where it writes about the known issues.

To avoid this issue, use the Promtail Docker target or Docker service discovery.

I used the Ubuntu docs to install my Cert listed in the first post. I followed that nearly exactly the only difference being I used a CA I’ve made previously instead of making a new one.
The driver you linked is the driver that I am using.
Thank you for your continued assistance and patience.

output for docker version:

Client: Docker Engine - Community
 Version:           28.0.4
 API version:       1.48
 Go version:        go1.23.7
 Git commit:        b8034c0
 Built:             Tue Mar 25 15:07:11 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.4
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.7
  Git commit:       6430e49
  Built:            Tue Mar 25 15:07:11 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.26
  GitCommit:        753481ec61c7c8955a23d6ff7bc8e4daed455734
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

output for docker info:

Client: Docker Engine - Community
 Version:    28.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.22.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.34.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 21
  Running: 20
  Paused: 0
  Stopped: 1
 Images: 36
 Server Version: 28.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 753481ec61c7c8955a23d6ff7bc8e4daed455734
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.12-8-pve
 Operating System: Ubuntu 22.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 32GiB
 Name: sarin
 ID: 4e62221c-453a-4f6b-a1bb-d890abb138cc
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

output for dpkg -l 'docker*' | grep '^ii':

ii  docker-buildx-plugin      0.22.0-1~ubuntu.22.04~jammy   amd64        Docker Buildx cli plugin.
ii  docker-ce                 5:28.0.4-1~ubuntu.22.04~jammy amd64        Docker: the open-source application container engine
ii  docker-ce-cli             5:28.0.4-1~ubuntu.22.04~jammy amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras 5:28.0.4-1~ubuntu.22.04~jammy amd64        Rootless support for Docker.
ii  docker-compose            1.29.2-1                      all          define and run multi-container Docker applications with YAML
ii  docker-compose-plugin     2.34.0-1~ubuntu.22.04~jammy   amd64        Docker Compose (V2) plugin for the Docker CLI.

snap is not installed.

Okay, try this documentation:

It is mainly for Docker registries, but you can try if this method helps with the logging driver as well.

It uses a different trust store

Unfortunately, the linked documentation didn’t solve the issue.

# tree /etc/docker/
/etc/docker/
|-- certs.d
|   `-- loki.sarin.lan
|       |-- ca.crt
|       |-- client.cert
|       `-- client.key
`-- daemon.json

I’ve also tried using loki.sarin.lan:443 just to see what happens there.

To be honest I never even used plugins except the SSHFS plugin which is still mentioned in the engine plugin documentation, but that plugin is already archived since 2022. So I wasn’t sure how plugins were running, but plugins are running in containers. There is a debugging guide in the documentation that explains how you can list plugin containers. Then if you know how to use runc, you can also find everything about the plugin, including how it sees the network and what certificates it is using.

The below command (as root) would for example list the ca-certificates folder inside the plugin’s container

nsenter --all -t $(
  runc --root /run/docker/runtime-runc/plugins.moby list \
  | grep $(\
      docker plugin list --no-trunc \
      | grep loki \
      | awk '{print $1}' \
  ) | awk '{print $2}' \
) \
-- ls -la /usr/share/ca-certificates
# or
# -- ls -la /usr/local/share/ca-certificates
# or
# -- ls -la /etc/ca-certificates

But you can also get a shell in it:

nsenter --all -t $(
  runc --root /run/docker/runtime-runc/plugins.moby list \
  | grep $(\
      docker plugin list --no-trunc \
      | grep loki \
      | awk '{print $1}' \
  ) | awk '{print $2}' \
) \
-- sh

To shorten it, you can add it to a function

function loki() {
  nsenter --all -t $(
    runc --root /run/docker/runtime-runc/plugins.moby list \
    | grep $(\
      docker plugin list --no-trunc \
      | grep loki \
      | awk '{print $1}' \
    ) | awk '{print $2}' \
  ) -- "$@";
}

Then just use

loki ls -la /usr/local/share/ca-certificates

curl is not in the “loki” plugin container, only wget, but you can install curl in it after you got a shell first or using the function directly:

loki apk add curl

Then you can test what the plugin sees. I don’t know exactly how the CA certificates should be added, but the ca bundle file can be read this way:

loki cat /etc/ssl/certs/ca-certificates.crt

Other files in the same folder are symbolic links to the files in the previously mentioned folder.

Grafana also shared the source code of this plugin

In the config.go file we can find the parameters if not in any documentation (I couldn’t)

So it looks like it supports settingthe CA file. At this point I’m still not sure how the plugin would see it, but you can try to set the “loki-tls-ca-file” config parameter as a log option to an absolute path as described here for other options:

1 Like