Docker Community Forums

Share and learn in the Docker community.

Docker caching logs

Hello,

I have some problem with caching docker logs from containers. When I use:
“gelf-address”: “tcp://xxxxxxxx”,
“cache-max-size”: “25m”,
“cache-max-file”: “6”
Cache parameters doesn’t work, every time I have 5 files, 20m size. Any ideas why this isn’t working?

ls -l /var/lib/docker/containers/61fdb6e55784853db2ee40a2d0b7a76952fe930ae40eaaf42ed81f76a77ea7e7/

razem 6044
drwx------. 2 root root 6 12-01 14:38 checkpoints
-rw-------. 1 root root 6537 12-01 14:38 config.v2.json
-rw-r-----. 1 root root 998446 12-01 16:06 container-cached.log
-rw-r-----. 1 root root 1287207 12-01 15:22 container-cached.log.1.gz
-rw-r-----. 1 root root 1290002 12-01 15:15 container-cached.log.2.gz
-rw-r-----. 1 root root 1289806 12-01 15:07 container-cached.log.3.gz
-rw-r-----. 1 root root 1291118 12-01 14:59 container-cached.log.4.gz
-rw-r–r–. 1 root root 2190 12-01 14:38 hostconfig.json
-rw-r–r–. 1 root root 29 12-01 14:38 hostname
-rw-r–r–. 1 root root 1817 12-01 14:38 hosts
drwx–x—. 3 root root 17 12-01 14:38 mounts
-rw-r–r–. 1 root root 122 12-01 14:38 resolv.conf

docker version

Client: Docker Engine - Community
Version: 20.10.11
API version: 1.41
Go version: go1.16.9
Git commit: dea9396
Built: Thu Nov 18 00:38:53 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.11
API version: 1.41 (minimum version 1.12)
Go version: go1.16.9
Git commit: 847da18
Built: Thu Nov 18 00:37:17 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0

BR,
Mat

Where did you configure those logging parameters? If you want to change those parameters in the daemon.json for an already running container, that will not work. It affects only the newly created containers.

Yes, I configure it in daemon.json, after that I recreate container. After creating the containers, the parameters do not work.

Additionally I attach container parameters:

docker inspect api.evoting

        "ContainerIDFile": "",
        "LogConfig": {
            "Type": "gelf",
            "Config": {
                "cache-max-file": "6",
                "cache-max-size": "25m",
                "gelf-address": "tcp://xx.xx.xx.xx:xxxx"
            }
        },

The system ignores these parameters.
-rw-r-----. 1 root root 1287207 12-01 15:22 container-cached.log.1.gz
-rw-r-----. 1 root root 1290002 12-01 15:15 container-cached.log.2.gz
-rw-r-----. 1 root root 1289806 12-01 15:07 container-cached.log.3.gz
-rw-r-----. 1 root root 1291118 12-01 14:59 container-cached.log.4.gz

Could you share the output of the following command?

docker info

Let’s see if your global configuration was successful.

Hello,
This is output:

# docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
scan: Docker Scan (Docker Inc., v0.8.0)

Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 45
Server Version: 20.10.11
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: gelf
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-1160.31.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.9 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.62GiB
Name: dltkdpwprd01.prd.dlt.kdpw.pl
ID: ZQDB:SQH3:YJ7T:E3XT:DAQQ:JO3B:K5SF:UNUR:GK5P:7B4X:4LLD:IITC
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

I couldn’t try it yet but your config looks good.

Can anyone help me? Please.

I finally tried it and I could not set it correctly either. I tried some debugging and in the meantime I corrupted my Docker daemon every time. The configuration seems right but the cached log file is growing beyond the limit. When I wanted to generate at least 25 megabytes log quickly, I managed to generate more logs than Docker could write to the file, making my container undeletable due to the ongoing write operations. Then restarted the Docker daemon and I tried an other way and docker container inspect test did not respond while it worked on other containers. Of course, the container become undeletable again.

I think I stop trying to create large log files for now :slight_smile:

Back to the original problem: based on the documentation and our observations it seems to be a bug for me. It is also possible that we miss something but I don’t have more ideas.

Is there any way I can report this problem as a software bug?

BR,
Mateusz

You can report it on GitHub. I am not entirely sure what repository is the best for this issue.