Syslog driver not work

General Discussions
1

First here is my setup

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy
$ cat /etc/docker/daemon.json 
{
    "log-driver": "syslog",
    "log-opts": {
        "tag": "{{.ImageName}}/{{.Name}}/{{.ID}}"
    }
}
$ docker --version 
Docker version 20.10.12, build 20.10.12-0ubuntu4

Before using docker let’s do some logging tests

$ logger abcHOST
$ journalctl -b --output cat | grep abcHOST
abcHOST

OK, but inside docker…

$ docker run --rm bash logger abcDOCKER
$ journalctl -b --output cat | grep abcDOCKER || echo RECORD NOT FOUND
RECORD NOT FOUND

Why?

2

The syslog driver is sending the logs to syslog, not to the journal. To see the logs from inside the container, you can try running journalctl -b -u docker.service to see the logs from the docker service or you can add "journald" as the "log-driver" in the daemon.json file and then run journalctl -b to see the logs from the containers.

To make the change, you can edit the daemon.json file and update the "log-driver" field to "journald" like this:

{
    "log-driver": "journald",
    "log-opts": {
        "tag": "{{.ImageName}}/{{.Name}}/{{.ID}}"
    }
}

Then, restart the Docker daemon for the changes to take effect:

sudo systemctl restart docker

After making this change, the logs from inside the container should appear in the journal when you run journalctl -b.

Refer: Journald logging driver | Docker Documentation

3 
~$ docker run --rm bash logger abcDOCKER
~$ journalctl -b -u docker.service --output cat | grep abcDOCKER || echo RECORD NOT FOUND
RECORD NOT FOUND
4

If you use logger inside a container that will not go the the logging driver. That driver handles only the container’s standard output (stdout) and error stream (stderr). You can try the driver this way:

Standard output

docker run --rm bash echo "stdoutDOCKER"

Standard error:

docker run --rm bash -c 'echo "stderrDOCKER" > /dev/stderr'
5 
$ cat test2 
set -eu

for suffix in journald syslog; do
    cat daemon-$suffix.json
    sudo ln -sf $PWD/daemon-$suffix.json /etc/docker/daemon.json
    sudo systemctl restart docker

    echo docker run...
    docker run --rm bash echo "$(date) stdDOCKERout"
    docker run --rm bash -c 'echo "$(date) stdDOCKERerr" > /dev/stderr'

    echo journalctl grep...
    (journalctl --output cat -b | grep stdDOCKER || echo RECORD NOT FOUND) | tail -n2
done

$ test2
{
    "log-driver": "journald",
    "log-opts": {
        "tag": "{{.ImageName}}/{{.Name}}/{{.ID}}"
    }
}
docker run...
mar 10 gen 2023, 12:59:47, CET stdDOCKERout
Tue Jan 10 11:59:48 UTC 2023 stdDOCKERerr
journalctl grep...
mar 10 gen 2023, 12:59:47, CET stdDOCKERout
Tue Jan 10 11:59:48 UTC 2023 stdDOCKERerr
{
    "log-driver": "syslog",
    "log-opts": {
        "tag": "{{.ImageName}}/{{.Name}}/{{.ID}}"
    }
}
docker run...
mar 10 gen 2023, 12:59:53, CET stdDOCKERout
Tue Jan 10 11:59:54 UTC 2023 stdDOCKERerr
journalctl grep...
mar 10 gen 2023, 12:59:53, CET stdDOCKERout
Tue Jan 10 11:59:54 UTC 2023 stdDOCKERerr

Ok !!!
It even seems to work both with journald driver and with syslog driver.
So where is the difference?

6

I didn’t know about this, but after trying the logging drivers again and searching for the relationship between journald and syslog I found a post that confirms that I suspected after playing with the drivers based on your message.

So it looks like it is not the logging driver that sends the logs to syslog and journald. Using the logger command had the same result. I haven’t tried other distributions but at least on Ubuntu, logs sent to syslog can appear in the journal and logs sent to journald can appear in sysllog as well.

7

???
That is?
Didn’t you say that?

That driver handles only the container’s standard output (stdout) and error stream (stderr).

8

Both are true. The logging driver handles only the container’s stdout and stderr. In my last message I was writing about logger on the host. So this is a host OS level setting. Logs will appear in the journal and also in syslog. Logger can’t send logs directly to the host from the container.

9

So both drivers (syslog and journald) are equivalent?

10

No. As I mentioned, this behavior has nothing to do with the log drivers. These are two different methods and how they send the logs to eachother depends on the operating system and the configurations.

We saw how it worked on Ubuntu by default and I also checked an old Centos server, where I tried the journald driver and I saw a strange entry like this: 6bcc8bc59370[17379]: [4B blob data].

Blob data instead of the text I have sent, but I found the text in the syslog. After that I tried the syslog driver and I saw the logs in both syslog and the journal as normal text.

So on Centos it looks like syslog (not the driver, but syslog itself) sends the logs to the journal as well or journld reads to logs from syslog, but when I use the journald driver, I can’t send logs directly to the journal from containers either because of a driver bug or the configuration of journald on CentOS.

Since it works for you, I suggest choose journald if you want to make sure the logs will be sent to the journal and syslog if you prefer syslog. Syslog would also support sending logs to a remote syslog server this is why it has different parameters than journald.

journald parameters: Journald logging driver | Docker Documentation
syslog parameters: Syslog logging driver | Docker Documentation

If you are intrested in how they are programed, you can find the sourcecode here:
journald driver: moby/journald.go at ffb2c1fb4a3f481f707cd8d7f23d2bdb174b90a3 · moby/moby · GitHub
syslog driver: moby/syslog.go at ffb2c1fb4a3f481f707cd8d7f23d2bdb174b90a3 · moby/moby · GitHub

Update:

Oh by the way, if you want to test sending logs from the host to journald and syslog, logger, the command that you tried sends the logs to syslog and the following command sends it to journald

echo "mylogmessage" | systemd-cat
11

???
syslog and journald drivers do not have nothing to do with the log drivers?
???
o_O

OK.

Which command?

To me he also sends them to syslog

$ echo "mylogmessage" | systemd-cat
$ grep mylogmessage /var/log/syslog 
Jan 12 09:02:59 USER-HOST [7633]: mylogmessage

By the way

If you use logger inside a container that will not go the the logging driver. That driver handles only the container’s standard output (stdout) and error stream (stderr).

Where is it documented?

PS

Please, if you can, answer in an orderly way to one question at a time, otherwise we don’t understand each other.

12

I don’t know what I could react to question marks and I don’t understand the sentence between them. Please, read my answers again and if you have questions try to ask a more specific question.

You just quoted the part after mentioning the command to which you replied with “Ok” before that.

Although I have read that in the manual of the logger command on CentOS and it looks like Ubuntu has a different logger command which mentions journald so I guess I was wrong about this.

13

syslog and journald are drivers? Yes or no?

Bah… to me mentions system log

$ logger --help

Usage:
 logger [options] [<message>]

Enter messages into the system log.

OK!!!
A last question

~$ grep log-driver /etc/docker/daemon.json 
    "log-driver": "journald",

~$ echo '<OUTPUT>' && docker run --rm bash echo stdDOCKERout && echo '</OUTPUT>'
<OUTPUT>
stdDOCKERout
</OUTPUT>

~$ journalctl --output cat -b | grep stdDOCKER
stdDOCKERout

Would it be possible to direct the output only on the Journal, avoiding that it is also visuped on the output stream (inside the <OUTPUT>…</OUTPUT> tags)?

14

This is not a yes or know question… syslog and journald are logging servers, but Docker has logging drivers with the same name. The syslog logging driver sends the logs to Syslog and the journald logging driver sends the logs to journald.

As I already mentioned it is something that you have to configure on the host operating system. Docker can’t help you with that. I don’t know how you could configure it if it is possible at all.