Hi all,
My team and I manage a fleet on edge compute devices (for robots) where we deploy our software through a suite of docker services. The edge device is a nvidia jetson orin agx and for the most part, things have been fine but every now and then we get a very concerning failure that causes all services to die (because the docker service exits and restarts) but we are not always certain what the trigger was.
Recently, we noticed that trying to run docker logs <container_name>
sometimes triggers the same error (and this is on a platform that turns on/off frequently with services are sometimes 10+ days old with relatively heavy logging (or atleast heavy in my opinion). Additionally, our logging approach, while it works, is fairly inefficient and most likely a major factor in this issue - our logging driver is configured to use syslog and we have rsyslog scoop these (plus dockerd) logs and write them to disk which get retroactively uploaded using an inhouse solution. Can high volume logging impair the daemon?
This has become a safety concern for our platforms because we are not 100% sure whats causing it and unsure how to resolve it once and for all.
In all cases we have observed this issue, the below log was seen in the logs after failure
panic: runtime error: invalid memory address or nil pointer dereference
We have read through some other cases here on the forums, but most of them appear to be permissions related to an extent or on deployment specific while ours seem to occur after running for long periods time
does anyone have an idea what could be the root cause of this issue? any recommendations on how we could optimize or improve our docker configurations to mitigate this issue and prevent it from happening?
some info on our docker config:
- docker: v24.0.7, build 24.0.7-0ubuntu2~20.04.1
- docker-compose: v1.29.2 (on some) and v2.32.1 (on others)
- we use nvidia container runtime (on both arm and x86 platforms)
- everything else is using the stock configuration afaik
any assistance, advice, or insights are greatly appreciated!