Hello,
we have a problem with missing memory in Linux. Except for the operating system we are runnung all our software in docker containers (somewhere between 30 and 50 containers).
After a reboot everything is fine. However over time (a few days) we are leaking memory. Summing up all memory used by all processes (most of them in docker) (easiest way is using procrank) is much lower than used memory shown in free command. Also summing up memory shown in docker stats is much lower than used (an about the same as summed up by procrank).
If we then restart docker with systemctl restart docker, the missing memory is back. So available memory is much higher and summed up memory is again about the same as used memory shown in free command.
I could not find a known issue similar to this problem but maybe someone knows something.
Thanks
Edit:
Memory usage stated by different tools
before after
procrank 2085114K 2754244K
free used 3055332K 2861756K
free swap 1951740K 721156K
docker stats sum 2120000K 2707000K
After restarting docker, a bit less memory is used, much less in swap but more in RAM. But most importantly before restarting docker there is a gap of about 1GB between the sum used by processes (calculated with procrank and also with docker stats) and after the restart it is only about 100MB. And we are wondering where this memory went in the first place and why restarting docker releases this lost memory.
As i cannot upload files yet, here is the output of the different tools before and after restarting docker:
procrank before restarting docker
PID Vss Rss Pss Uss cmdline
2593 3549868K 496828K 495720K 495480K ./app
6212 4552156K 85596K 85500K 85408K java
328976 606748K 78888K 77280K 77048K python
3278 1431120K 73776K 71914K 71584K python
1042568 2293996K 68680K 67218K 67096K python
2360 1535172K 65284K 63564K 63360K python
3497 786952K 53032K 50893K 50440K python
4848 421744K 52104K 50274K 49964K python
547 2997332K 47588K 47565K 47564K /usr/bin/dockerd
4401 490776K 48576K 47143K 47008K python
7166 580740K 46456K 45396K 45356K python
4236 415900K 44748K 43695K 43652K python
3798 417556K 44516K 43592K 43560K python
3498 1280036K 40736K 38359K 37684K ./app
4602 418312K 37280K 36356K 36324K python
3724 523536K 35964K 35090K 35056K python
2970 405504K 34880K 33787K 33740K python
3361 1206304K 34260K 31922K 31288K ./app
1405375 134904K 31092K 30200K 29500K python3
2678 503964K 30924K 28986K 27980K python
4004 497812K 27304K 25735K 25532K python
4380 499548K 27668K 25602K 24588K python
1044632 877844K 25760K 23936K 23284K python
6264 42004K 24804K 23767K 23312K python
3649 175396K 23716K 23712K 23712K redis-server
1035856 499168K 24432K 23472K 23432K python
4493 334856K 23996K 22907K 22860K python
2790 1143784K 23440K 21155K 20536K ./app
2612 1143784K 22380K 20075K 19452K ./app
2203 689864K 22056K 19808K 19360K python
503 1568952K 19756K 19748K 19748K /usr/bin/containerd
2601 408464K 16900K 16006K 15972K python
1981 413248K 16164K 15196K 15156K python
657110 624072K 16752K 14471K 14024K python
5358 336448K 14616K 13607K 13560K python
497 399572K 13676K 12490K 11708K /usr/bin/python3
432500 878244K 14128K 12358K 11716K python
4741 455120K 12860K 11955K 11740K ./app
3579 453148K 13328K 11884K 11500K ./app
3210 152928K 11324K 11216K 11112K python
1413366 23648K 10260K 9619K 9368K python
6524 481740K 10552K 9493K 9444K python
657322 556120K 10628K 8898K 8200K python
6719 556120K 10456K 8778K 8084K python
6780 487816K 9744K 8420K 8308K python
5050 1541832K 10624K 8339K 7892K python
247 361332K 9612K 8309K 7472K /lib/systemd/systemd-journald
2107 689864K 10152K 7957K 7532K python
4683 450824K 8216K 7350K 7144K ./app
1414740 641220K 8144K 7158K 6776K fish
2779 450404K 7888K 7140K 6952K ./app
1413406 708648K 6748K 6733K 6732K -fish
2096 36524K 6904K 5829K 5344K python
3876 32196K 6308K 5278K 4824K python
6726 144688K 4928K 4820K 4716K python
1 169320K 5300K 3538K 2256K /sbin/init
3192 713584K 5812K 3425K 3372K /usr/bin/containerd-shim-runc-v2
3561 713584K 5788K 3401K 3348K /usr/bin/containerd-shim-runc-v2
2266 713328K 5844K 3398K 3340K /usr/bin/containerd-shim-runc-v2
3319 713584K 5984K 3373K 3296K /usr/bin/containerd-shim-runc-v2
6660 713584K 5804K 3358K 3300K /usr/bin/containerd-shim-runc-v2
2032 713328K 5800K 3354K 3296K /usr/bin/containerd-shim-runc-v2
3671 713392K 5772K 3326K 3268K /usr/bin/containerd-shim-runc-v2
4257 713136K 5932K 3321K 3244K /usr/bin/containerd-shim-runc-v2
4638 713584K 5764K 3318K 3260K /usr/bin/containerd-shim-runc-v2
5313 713584K 5696K 3309K 3256K /usr/bin/containerd-shim-runc-v2
2683 713328K 5968K 3299K 3216K /usr/bin/containerd-shim-runc-v2
2455 713328K 5900K 3289K 3212K /usr/bin/containerd-shim-runc-v2
657301 713328K 5636K 3288K 3236K /usr/bin/containerd-shim-runc-v2
6487 713328K 5732K 3286K 3228K /usr/bin/containerd-shim-runc-v2
2483 713328K 5892K 3281K 3204K /usr/bin/containerd-shim-runc-v2
4794 713328K 5884K 3273K 3196K /usr/bin/containerd-shim-runc-v2
6672 713328K 5660K 3273K 3220K /usr/bin/containerd-shim-runc-v2
4119 713328K 5708K 3262K 3204K /usr/bin/containerd-shim-runc-v2
1918 713648K 5612K 3252K 3200K /usr/bin/containerd-shim-runc-v2
2562 713328K 5636K 3249K 3196K /usr/bin/containerd-shim-runc-v2
2920 713584K 5636K 3249K 3196K /usr/bin/containerd-shim-runc-v2
4965 713648K 5636K 3249K 3196K /usr/bin/containerd-shim-runc-v2
7146 713328K 5580K 3247K 3196K /usr/bin/containerd-shim-runc-v2
3447 713328K 5852K 3241K 3164K /usr/bin/containerd-shim-runc-v2
6756 713328K 5852K 3241K 3164K /usr/bin/containerd-shim-runc-v2
2682 713584K 5612K 3225K 3172K /usr/bin/containerd-shim-runc-v2
3114 713584K 5608K 3221K 3168K /usr/bin/containerd-shim-runc-v2
4324 713584K 5604K 3217K 3164K /usr/bin/containerd-shim-runc-v2
2158 713328K 5576K 3216K 3164K /usr/bin/containerd-shim-runc-v2
1044612 713584K 5600K 3213K 3160K /usr/bin/containerd-shim-runc-v2
3417 713328K 5816K 3205K 3128K /usr/bin/containerd-shim-runc-v2
3763 713328K 5640K 3194K 3136K /usr/bin/containerd-shim-runc-v2
4552 713328K 5636K 3190K 3132K /usr/bin/containerd-shim-runc-v2
1035833 713392K 5536K 3188K 3136K /usr/bin/containerd-shim-runc-v2
657089 713328K 5560K 3173K 3120K /usr/bin/containerd-shim-runc-v2
3901 713328K 5556K 3169K 3116K /usr/bin/containerd-shim-runc-v2
2039 713328K 5528K 3168K 3116K /usr/bin/containerd-shim-runc-v2
4208 713328K 5540K 3153K 3100K /usr/bin/containerd-shim-runc-v2
3409 713392K 5820K 3151K 3068K /usr/bin/containerd-shim-runc-v2
3839 713328K 5568K 3122K 3064K /usr/bin/containerd-shim-runc-v2
4478 713328K 5496K 3109K 3056K /usr/bin/containerd-shim-runc-v2
432478 713584K 5472K 3085K 3032K /usr/bin/containerd-shim-runc-v2
2435 713392K 5468K 3081K 3028K /usr/bin/containerd-shim-runc-v2
6238 713584K 5460K 3073K 3020K /usr/bin/containerd-shim-runc-v2
328956 713584K 5404K 3056K 3004K /usr/bin/containerd-shim-runc-v2
1413381 170644K 4140K 3036K 1980K (sd-pam)
1042548 713584K 5380K 2993K 2940K /usr/bin/containerd-shim-runc-v2
1042642 12232K 2972K 2968K 2968K ./tunnel
1413380 19184K 2324K 2308K 2308K /lib/systemd/systemd
1413404 14180K 2504K 1807K 1332K sshd: alpr@pts/0
459 224556K 1872K 1666K 1640K /usr/sbin/rsyslogd
1416158 3292K 2344K 1467K 1140K ./procrank
6323 19172K 1308K 1296K 1296K /lib/systemd/systemd
1043344 19324K 1340K 1203K 1184K /lib/systemd/systemd-networkd
1413376 13912K 1484K 1030K 592K sshd: alpr [priv]
1042703 573792K 1020K 1013K 1012K -fish
1414738 12660K 1320K 982K 892K sudo
446 243056K 1080K 930K 880K /usr/lib/accountsservice/accounts-daemon
510 75000K 992K 599K 544K /usr/sbin/ntpd
456 32968K 596K 584K 580K /usr/bin/python3
448 7772K 560K 548K 548K /usr/bin/dbus-daemon
1042658 13932K 500K 496K 496K sshd: root@pts/1
1414739 11768K 504K 492K 492K su
523 236672K 488K 476K 476K /usr/lib/policykit-1/polkitd
440 24036K 576K 393K 360K /lib/systemd/systemd-resolved
455 81936K 820K 339K 220K /usr/sbin/irqbalance
461 16980K 464K 325K 296K /lib/systemd/systemd-logind
279 21452K 340K 309K 308K /lib/systemd/systemd-udevd
447 10316K 804K 259K 132K /usr/sbin/cron
514 12292K 168K 160K 160K sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
465 10476K 116K 112K 112K /bin/bash
1042633 16800K 96K 92K 92K dnsmasq
1042641 2388K 88K 84K 84K /bin/sh
509 9332K 64K 60K 60K /sbin/agetty
6324 169288K 132K 46K 4K (sd-pam)
273 2496K 8K 4K 4K bpfilter_umh
------ ------ ------
2085114K 2060664K TOTAL
RAM: 3670912K total, 139124K free, 13388K buffers, 345756K cached, 249288K shmem, 266776K slab
free before restarting docker
total used free shared buff/cache available
Mem: 3670912 3055332 176220 249288 439360 145368
Swap: 1951740 1951740 0
docker stats before restarting docker
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
26709ca3dfad om_payment_cardcomplete_1 0.32% 25.36MiB / 3.501GiB 0.71% 380MB / 156MB 1.21GB / 0B 16
3734e4ed6c50 om_service_updater_1 0.13% 22.89MiB / 3.501GiB 0.64% 332MB / 22.4MB 6.79GB / 0B 7
ebe8ae226e35 om_io_modbus2_2 1.71% 9.605MiB / 3.501GiB 0.27% 8.26GB / 3.53GB 1.28GB / 4.1kB 8
c1f71c6edeae om_service_trafficlight2_4 0.50% 14.89MiB / 3.501GiB 0.42% 7.26GB / 548MB 733MB / 0B 5
7499fc2f2515 om_payment_cardcomplete_2 0.40% 12.79MiB / 3.501GiB 0.36% 985MB / 411MB 1.67GB / 0B 16
bcda056a19da om_service_relay2_1 0.08% 75.02MiB / 3.501GiB 2.09% 1.36GB / 117MB 10.2GB / 4.1kB 7
1a8a784fbb29 om_controller_freeflow2_1 0.49% 71.09MiB / 3.501GiB 1.98% 1.77GB / 1.43GB 14.1GB / 11.8GB 23
19862dc0bc6e om_service_dsgvo2_1 0.12% 14.88MiB / 3.501GiB 0.42% 925MB / 129MB 9.88GB / 367MB 6
757a191de560 om_service_enforcement_1 0.09% 44.71MiB / 3.501GiB 1.25% 1.11GB / 221MB 6.86GB / 382MB 6
289e63f28699 om_service_misc_1 1.85% 48.96MiB / 3.501GiB 1.37% 9.25GB / 1.35GB 6.82GB / 4.1kB 7
345b039b2da7 om_service_databox_1 0.06% 33.8MiB / 3.501GiB 0.94% 925MB / 118MB 5.58GB / 10.2MB 7
f3436f0e3b45 om_alpr_nn_2 2.63% 34.03MiB / 3.501GiB 0.95% 1.27TB / 11.7GB 1.57GB / 119MB 17
2a4db9e14299 om_alpr_nn_1 2.57% 31.51MiB / 3.501GiB 0.88% 1.5TB / 11.3GB 1.03GB / 110MB 17
ecc0277e9d95 om_alpr_nn_4 3.24% 21.3MiB / 3.501GiB 0.59% 845GB / 28.4GB 1.4GB / 109MB 17
597c3cc43cc4 om_alpr_nn_3 3.53% 19.96MiB / 3.501GiB 0.56% 944GB / 30.1GB 1.54GB / 101MB 17
4d7c00e670b5 om_service_eventlog2_1 0.19% 43.12MiB / 3.501GiB 1.20% 1.47GB / 208MB 9.95GB / 21.9GB 6
58da41445c55 om_fancy_assistant2_1 0.54% 69.7MiB / 3.501GiB 1.94% 3.74GB / 280MB 19.1GB / 32.8kB 35
800c8d08a7c8 om_service_stats2_1 0.01% 32.86MiB / 3.501GiB 0.92% 945MB / 5.62GB 45.5GB / 4.1kB 6
85967174ec17 om_alpr_worker_1 5.00% 737.5MiB / 3.501GiB 20.57% 880MB / 74.3MB 7.94GB / 0B 40
32ff60cb09d8 om_alpr_tracking_1 0.27% 12.29MiB / 3.501GiB 0.34% 40.5GB / 5.81GB 572MB / 16.4kB 8
313465b59bbe om_alpr_tracking_2 0.30% 10.89MiB / 3.501GiB 0.30% 40.5GB / 6.13GB 1.11GB / 16.4kB 8
ffac8921de04 om_alpr_tracking_3 0.27% 7.668MiB / 3.501GiB 0.21% 40.4GB / 4.97GB 837MB / 16.4kB 8
2c7c9c7ade0b om_alpr_tracking_4 0.28% 7.309MiB / 3.501GiB 0.20% 40.5GB / 4.97GB 869MB / 16.4kB 8
fe25745ab4e9 om_io_modbus2_1 1.38% 9.34MiB / 3.501GiB 0.26% 15.1GB / 6.55GB 1.35GB / 4.1kB 8
3d568b264f8f om_service_doorreader_1 0.36% 7.738MiB / 3.501GiB 0.22% 1.07GB / 115MB 5.63GB / 4.1kB 7
8d5ead4cf25c om_access_oebb_1 0.03% 48.91MiB / 3.501GiB 1.36% 1.04GB / 77.3MB 6.62GB / 1.04GB 6
b80d9b997dfb om_oebb_api_1 0.02% 21.77MiB / 3.501GiB 0.61% 1.07GB / 110MB 7.12GB / 0B 5
319969a79a63 om_access_arivo_1 0.03% 35.73MiB / 3.501GiB 1.00% 1.1GB / 82.4MB 6.42GB / 1.44MB 6
f8b63d0400c8 om_validation_arivo2_1 0.12% 15.59MiB / 3.501GiB 0.43% 929MB / 106MB 5.35GB / 856kB 6
841c5c71ee05 om_access_whitelist_1 0.02% 42.29MiB / 3.501GiB 1.18% 1.03GB / 71.3MB 6.15GB / 201kB 6
7e8b839902ad om_service_setup_1 0.00% 8.32MiB / 3.501GiB 0.23% 0B / 0B 6.21GB / 4.1kB 7
81ac1a1511a7 om_validation_oebb_1 0.32% 113.1MiB / 3.501GiB 3.15% 1.18GB / 101MB 20.5GB / 60.9MB 38
e80c3c014f8c om_service_trafficlight2_2 0.40% 8.84MiB / 3.501GiB 0.25% 13.3GB / 1.01GB 840MB / 0B 5
23e2909734ad om_service_trafficlight2_3 0.56% 20.75MiB / 3.501GiB 0.58% 13.3GB / 1.01GB 739MB / 0B 5
09f8db49a8f8 om_medium_http_1 0.01% 12.16MiB / 3.501GiB 0.34% 884MB / 55.8MB 5.08GB / 0B 5
d96ae05c1586 om_io_display2_2 0.26% 25.29MiB / 3.501GiB 0.71% 1.36GB / 2.19GB 4.4GB / 430kB 7
f2120a584111 om_io_display2_1 0.27% 28.68MiB / 3.501GiB 0.80% 1.37GB / 2.43GB 3.07GB / 430kB 7
3dc1cf8f08ee om_fancy_sentry_1 0.00% 23.27MiB / 3.501GiB 0.65% 307GB / 3.77GB 1.56GB / 0B 9
a8dbfd65d0aa om_service_logger_1 0.00% 3.867MiB / 3.501GiB 0.11% 1.01GB / 252MB 2.71GB / 4.1kB 7
d37f6580704d om_service_deeplog_1 0.32% 6.27MiB / 3.501GiB 0.17% 53.4GB / 3.82GB 64.4GB / 0B 8
1592d9f1f39c om_service_camoperator_1 1.79% 63.67MiB / 3.501GiB 1.78% 8.85GB / 4.58GB 6.4GB / 70.4MB 21
cdb234206b0e om_service_trafficlight2_1 0.62% 8.469MiB / 3.501GiB 0.24% 13.3GB / 1.01GB 757MB / 0B 5
660116d74a99 om_rpc_client_1 0.14% 48.88MiB / 3.501GiB 1.36% 2.4GB / 1.48GB 9.1GB / 5.52GB 9
60187c214c10 om_service_shootback_1 1.60% 15.56MiB / 3.501GiB 0.43% 1.02GB / 168MB 452MB / 0B 17
183d2d3538ff om_infra_redis_1 0.12% 138MiB / 3.501GiB 3.85% 3.89GB / 306GB 2.04GB / 72.9GB 4
1c04f6df217c om_service_broker_1 0.85% 11.91MiB / 3.501GiB 0.33% 83.5GB / 402GB 261MB / 0B 3
procrank after restarting docker
PID Vss Rss Pss Uss cmdline
1420175 3484332K 1030104K 1027713K 1026824K ./app
1420610 167204K 142432K 142428K 142428K redis-server
1422351 4550296K 94168K 93486K 93016K java
1418960 1126116K 58484K 54630K 54376K python
1419447 496264K 56120K 52065K 51420K python
1420334 605592K 50280K 47330K 47224K python
1417524 2914200K 48312K 45733K 45484K /usr/bin/dockerd
1421760 2135016K 49168K 45727K 45480K python
1422581 144708K 44168K 43702K 43240K python
1420508 433248K 46824K 43307K 43148K python
1422594 555528K 44208K 38814K 36676K python
1422673 555608K 44160K 38772K 36636K python
1419515 500468K 43316K 38630K 37040K python
1422538 481488K 41948K 38544K 38372K python
1418385 498912K 40960K 37019K 36060K python
1420358 487700K 38624K 35642K 35532K python
1417885 559144K 39896K 35216K 34092K python
1418602 496528K 38216K 34947K 34804K python
1419502 499932K 39144K 34671K 33140K python
1420031 408676K 38344K 34577K 34340K python
1420860 409192K 36312K 32638K 32420K python
1419932 1517764K 35596K 32378K 32252K python
1418631 481756K 34892K 31194K 30896K python
1418217 1209320K 34004K 30096K 28448K ./app
1419394 408172K 33128K 29607K 29432K python
1419931 408564K 33008K 29532K 29368K python
1420908 335804K 32224K 29496K 29400K python
1419966 149484K 29708K 29242K 28780K python
1419273 407980K 32448K 29080K 28936K python
1420841 332468K 31012K 28078K 27964K python
1418621 409348K 30252K 27429K 27328K python
1419712 404348K 30288K 27272K 27156K python
1422589 39700K 27968K 26810K 25956K python
1418368 872748K 28916K 25755K 25116K python
1419805 872744K 28676K 25117K 24488K python
1420401 402108K 28264K 24088K 23560K python
1418792 402108K 28216K 24054K 23536K python
1418026 402108K 27436K 23947K 23468K python
1417801 402108K 27392K 23918K 23432K python
503 1568952K 21616K 21592K 21592K /usr/bin/containerd
1420494 928132K 19156K 16371K 15684K ./app
1418194 862596K 17352K 14549K 13844K ./app
1420597 31444K 15296K 14156K 13320K python
247 377848K 16092K 13126K 12024K /lib/systemd/systemd-journald
1419225 31140K 13232K 12612K 12296K python
1418027 862596K 15328K 12482K 11724K ./app
497 399572K 12372K 10304K 9852K /usr/bin/python3
1405375 134904K 9980K 7962K 7508K python3
1417119 30068K 10524K 6738K 5988K /usr/bin/python3
1420522 446300K 9040K 6385K 5420K ./app
1422210 1818868K 7588K 4738K 3916K -fish
1418594 446304K 5764K 4188K 3884K ./app
1418367 446300K 5440K 4134K 3864K ./app
1418205 446300K 5352K 4105K 3852K ./app
1 169212K 7040K 3750K 2692K /sbin/init
1417833 713008K 5316K 3557K 2204K /usr/bin/containerd-shim-runc-v2
1417740 712752K 5288K 3529K 2176K /usr/bin/containerd-shim-runc-v2
456 32968K 5524K 3414K 2888K /usr/bin/python3
1422178 19188K 7464K 3097K 2336K /lib/systemd/systemd
1423059 180572K 5720K 2765K 1924K -fish
1422501 712688K 2980K 2569K 2560K /usr/bin/containerd-shim-runc-v2
1422528 712688K 2888K 2466K 2456K /usr/bin/containerd-shim-runc-v2
1421700 712688K 2820K 2409K 2400K /usr/bin/containerd-shim-runc-v2
1422453 712688K 2784K 2373K 2364K /usr/bin/containerd-shim-runc-v2
1422498 712432K 2732K 2321K 2312K /usr/bin/containerd-shim-runc-v2
1422973 13928K 5888K 2220K 1616K sshd: root@pts/4
1423298 12836K 4720K 2214K 1904K sudo
1420838 712432K 2616K 2205K 2196K /usr/bin/containerd-shim-runc-v2
1422650 711024K 2592K 2181K 2172K /usr/bin/containerd-shim-runc-v2
1417197 18808K 5264K 2115K 1740K /lib/systemd/systemd-networkd
6323 19172K 4104K 1770K 1404K /lib/systemd/systemd
1418883 712688K 2140K 1729K 1720K /usr/bin/containerd-shim-runc-v2
1420533 712432K 1948K 1537K 1528K /usr/bin/containerd-shim-runc-v2
1422634 9864K 1536K 1532K 1532K ./tunnel
1420268 711024K 1900K 1489K 1480K /usr/bin/containerd-shim-runc-v2
446 243056K 3272K 1487K 1128K /usr/lib/accountsservice/accounts-daemon
1420200 711280K 1884K 1473K 1464K /usr/bin/containerd-shim-runc-v2
1423205 16264K 5348K 1470K 900K /lib/systemd/systemd-hostnamed
1420026 711024K 1852K 1441K 1432K /usr/bin/containerd-shim-runc-v2
1419638 711280K 1848K 1437K 1428K /usr/bin/containerd-shim-runc-v2
1422094 13908K 5352K 1412K 484K sshd: alpr [priv]
1419131 712432K 1808K 1397K 1388K /usr/bin/containerd-shim-runc-v2
1420466 711024K 1784K 1373K 1364K /usr/bin/containerd-shim-runc-v2
1419650 711280K 1780K 1369K 1360K /usr/bin/containerd-shim-runc-v2
1419683 711280K 1780K 1369K 1360K /usr/bin/containerd-shim-runc-v2
1419376 712688K 1776K 1365K 1356K /usr/bin/containerd-shim-runc-v2
1420197 712688K 1760K 1349K 1340K /usr/bin/containerd-shim-runc-v2
1418313 712432K 1752K 1341K 1332K /usr/bin/containerd-shim-runc-v2
1420223 712432K 1744K 1333K 1324K /usr/bin/containerd-shim-runc-v2
1420056 712432K 1740K 1329K 1320K /usr/bin/containerd-shim-runc-v2
1419889 712688K 1720K 1309K 1300K /usr/bin/containerd-shim-runc-v2
1419162 712688K 1700K 1289K 1280K /usr/bin/containerd-shim-runc-v2
1420282 711024K 1700K 1289K 1280K /usr/bin/containerd-shim-runc-v2
1422054 15548K 4144K 1265K 904K ssh
1419327 711280K 1660K 1249K 1240K /usr/bin/containerd-shim-runc-v2
1419274 712432K 1648K 1237K 1228K /usr/bin/containerd-shim-runc-v2
1420696 712432K 1632K 1221K 1212K /usr/bin/containerd-shim-runc-v2
1417991 711280K 1616K 1205K 1196K /usr/bin/containerd-shim-runc-v2
1418179 712432K 1608K 1197K 1188K /usr/bin/containerd-shim-runc-v2
1422179 170536K 3104K 1171K 508K (sd-pam)
1419740 712688K 1564K 1153K 1144K /usr/bin/containerd-shim-runc-v2
1420595 711280K 1560K 1149K 1140K /usr/bin/containerd-shim-runc-v2
1417951 710768K 1528K 1117K 1108K /usr/bin/containerd-shim-runc-v2
1418288 712432K 1516K 1105K 1096K /usr/bin/containerd-shim-runc-v2
1418589 712432K 1508K 1097K 1088K /usr/bin/containerd-shim-runc-v2
1418526 712688K 1460K 1049K 1040K /usr/bin/containerd-shim-runc-v2
1418134 712432K 1456K 1045K 1036K /usr/bin/containerd-shim-runc-v2
1418290 711280K 1456K 1045K 1036K /usr/bin/containerd-shim-runc-v2
1418476 712688K 1436K 1025K 1016K /usr/bin/containerd-shim-runc-v2
1418109 712688K 1432K 1021K 1012K /usr/bin/containerd-shim-runc-v2
1423308 3300K 2424K 996K 928K ./procrank
1419413 711280K 1384K 973K 964K /usr/bin/containerd-shim-runc-v2
459 224556K 1704K 968K 940K /usr/sbin/rsyslogd
1418532 711280K 1364K 953K 944K /usr/bin/containerd-shim-runc-v2
440 24036K 2864K 928K 688K /lib/systemd/systemd-resolved
1418449 712688K 1328K 917K 908K /usr/bin/containerd-shim-runc-v2
448 7772K 2076K 869K 708K /usr/bin/dbus-daemon
1419739 711024K 1260K 849K 840K /usr/bin/containerd-shim-runc-v2
523 236672K 1996K 840K 528K /usr/lib/policykit-1/polkitd
1422205 14044K 3484K 806K 124K sshd: alpr@pts/0
461 16972K 2656K 660K 456K /lib/systemd/systemd-logind
279 21452K 1912K 632K 572K /lib/systemd/systemd-udevd
510 75000K 1340K 523K 488K /usr/sbin/ntpd
455 81936K 1464K 320K 248K /usr/sbin/irqbalance
514 12292K 1668K 304K 156K sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
1417118 10484K 1536K 257K 176K SCREEN
1421897 16800K 412K 235K 140K dnsmasq
447 10316K 952K 162K 132K /usr/sbin/cron
1422053 2616K 1504K 160K 96K /bin/sh
1422629 2388K 296K 157K 88K /bin/sh
465 10476K 64K 60K 60K /bin/bash
6324 169288K 128K 42K 0K (sd-pam)
509 9332K 24K 20K 20K /sbin/agetty
273 2496K 84K 18K 16K bpfilter_umh
------ ------ ------
2754244K 2710892K TOTAL
RAM: 3670912K total, 138044K free, 20380K buffers, 533848K cached, 287776K shmem, 245052K slab
free after restarting docker
total used free shared buff/cache available
Mem: 3670912 2861756 136304 284224 672852 260628
Swap: 1951740 721156 1230584
docker stats after restarting docker
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
26709ca3dfad om_payment_cardcomplete_1 9.35% 26.33MiB / 3.501GiB 0.73% 122kB / 83.6kB 29MB / 0B 16
3734e4ed6c50 om_service_updater_1 0.07% 33.77MiB / 3.501GiB 0.94% 221kB / 43.9kB 45MB / 4.1kB 7
ebe8ae226e35 om_io_modbus2_2 2.25% 36.88MiB / 3.501GiB 1.03% 157kB / 64.6kB 12.5MB / 4.1kB 8
c1f71c6edeae om_service_trafficlight2_4 0.83% 23.41MiB / 3.501GiB 0.65% 180kB / 34.6kB 16.2MB / 0B 5
7499fc2f2515 om_payment_cardcomplete_2 0.37% 23.39MiB / 3.501GiB 0.65% 90.6kB / 56kB 5.92MB / 0B 16
bcda056a19da om_service_relay2_1 0.32% 44.65MiB / 3.501GiB 1.25% 188kB / 24.4kB 81.7MB / 4.1kB 7
1a8a784fbb29 om_controller_freeflow2_1 4.04% 53.89MiB / 3.501GiB 1.50% 78.2kB / 32.1kB 36.8MB / 24.6kB 20
19862dc0bc6e om_service_dsgvo2_1 0.16% 26.22MiB / 3.501GiB 0.73% 54kB / 30.2kB 36.5MB / 0B 6
757a191de560 om_service_enforcement_1 0.00% 40.19MiB / 3.501GiB 1.12% 51kB / 27.2kB 44.1MB / 4.1kB 6
289e63f28699 om_service_misc_1 0.82% 31.2MiB / 3.501GiB 0.87% 90kB / 58.4kB 32.2MB / 4.1kB 7
345b039b2da7 om_service_databox_1 0.09% 32.82MiB / 3.501GiB 0.92% 46.7kB / 24.2kB 37.2MB / 8.19kB 7
f3436f0e3b45 om_alpr_nn_2 3.57% 16.55MiB / 3.501GiB 0.46% 51.5MB / 92.9kB 10.8MB / 4.1kB 14
2a4db9e14299 om_alpr_nn_1 33.71% 23.59MiB / 3.501GiB 0.66% 91.8MB / 470kB 26.6MB / 4.1kB 17
ecc0277e9d95 om_alpr_nn_4 4.50% 15.27MiB / 3.501GiB 0.43% 7.11MB / 315kB 2.56MB / 4.1kB 14
597c3cc43cc4 om_alpr_nn_3 0.00% 15.67MiB / 3.501GiB 0.44% 7.69MB / 324kB 28.9MB / 4.1kB 14
4d7c00e670b5 om_service_eventlog2_1 0.00% 33.8MiB / 3.501GiB 0.94% 74.7kB / 29.7kB 32.3MB / 930kB 6
58da41445c55 om_fancy_assistant2_1 0.00% 48.5MiB / 3.501GiB 1.35% 192kB / 47.3kB 68.8MB / 32.8kB 35
800c8d08a7c8 om_service_stats2_1 0.00% 24.99MiB / 3.501GiB 0.70% 44.6kB / 26kB 56MB / 4.1kB 6
85967174ec17 om_alpr_worker_1 0.00% 1.224GiB / 3.501GiB 34.98% 53.9kB / 30.1kB 211MB / 0B 40
32ff60cb09d8 om_alpr_tracking_1 0.00% 6.703MiB / 3.501GiB 0.19% 650kB / 121kB 5.3MB / 16.4kB 8
313465b59bbe om_alpr_tracking_2 1.34% 4.219MiB / 3.501GiB 0.12% 652kB / 89.1kB 995kB / 16.4kB 8
ffac8921de04 om_alpr_tracking_3 1.46% 4.32MiB / 3.501GiB 0.12% 653kB / 89.1kB 2MB / 16.4kB 8
2c7c9c7ade0b om_alpr_tracking_4 1.27% 4.867MiB / 3.501GiB 0.14% 653kB / 90.1kB 7.1MB / 16.4kB 8
fe25745ab4e9 om_io_modbus2_1 2.68% 37.7MiB / 3.501GiB 1.05% 158kB / 66.4kB 18.9MB / 4.1kB 8
3d568b264f8f om_service_doorreader_1 0.84% 32.95MiB / 3.501GiB 0.92% 44.2kB / 17kB 32.8MB / 4.1kB 7
8d5ead4cf25c om_access_oebb_1 0.00% 28.76MiB / 3.501GiB 0.80% 46.9kB / 21.8kB 29.7MB / 0B 6
b80d9b997dfb om_oebb_api_1 0.00% 26.4MiB / 3.501GiB 0.74% 29.4kB / 13.3kB 30.2MB / 0B 5
319969a79a63 om_access_arivo_1 0.10% 32.11MiB / 3.501GiB 0.90% 46.7kB / 22.1kB 32.4MB / 4.1kB 6
f8b63d0400c8 om_validation_arivo2_1 0.86% 28.52MiB / 3.501GiB 0.80% 46.1kB / 23.8kB 30.4MB / 4.1kB 6
841c5c71ee05 om_access_whitelist_1 0.00% 31.97MiB / 3.501GiB 0.89% 50.7kB / 24.3kB 33.6MB / 4.1kB 6
7e8b839902ad om_service_setup_1 0.42% 35.54MiB / 3.501GiB 0.99% 0B / 0B 30.7MB / 4.1kB 7
81ac1a1511a7 om_validation_oebb_1 27.80% 143.5MiB / 3.501GiB 4.00% 47.4kB / 20.4kB 112MB / 0B 39
e80c3c014f8c om_service_trafficlight2_2 0.00% 22.82MiB / 3.501GiB 0.64% 180kB / 34kB 1.74MB / 0B 5
23e2909734ad om_service_trafficlight2_3 1.33% 23.27MiB / 3.501GiB 0.65% 182kB / 31.9kB 9.98MB / 0B 6
09f8db49a8f8 om_medium_http_1 2.26% 31MiB / 3.501GiB 0.86% 26.5kB / 10.7kB 31.7MB / 0B 5
d96ae05c1586 om_io_display2_2 0.79% 32.57MiB / 3.501GiB 0.91% 65.3kB / 58kB 18.5MB / 430kB 7
f2120a584111 om_io_display2_1 0.98% 36.37MiB / 3.501GiB 1.01% 72.8kB / 181kB 21.9MB / 430kB 7
3dc1cf8f08ee om_fancy_sentry_1 0.00% 25.86MiB / 3.501GiB 0.72% 2.78MB / 31.7kB 23MB / 0B 9
a8dbfd65d0aa om_service_logger_1 0.00% 40.68MiB / 3.501GiB 1.13% 16.7kB / 11.7kB 18.3MB / 4.1kB 7
d37f6580704d om_service_deeplog_1 0.00% 13.63MiB / 3.501GiB 0.38% 815kB / 81.2kB 5.15MB / 0B 8
1592d9f1f39c om_service_camoperator_1 1.90% 30.6MiB / 3.501GiB 0.85% 158kB / 84.5kB 27.5MB / 4.1kB 21
cdb234206b0e om_service_trafficlight2_1 0.00% 22.61MiB / 3.501GiB 0.63% 175kB / 31kB 4.24MB / 0B 5
660116d74a99 om_rpc_client_1 0.00% 35.9MiB / 3.501GiB 1.00% 61.6kB / 51.7kB 30.8MB / 53.2kB 8
60187c214c10 om_service_shootback_1 0.00% 11.78MiB / 3.501GiB 0.33% 88.1MB / 47.4kB 6.35MB / 0B 8
183d2d3538ff om_infra_redis_1 0.00% 130.4MiB / 3.501GiB 3.64% 38.9kB / 2.78MB 167MB / 0B 4
1c04f6df217c om_service_broker_1 6.27% 27.12MiB / 3.501GiB 0.76% 2.2MB / 6.64MB 5.75MB / 0B 3
Edit 2:
Restarting all docker container fixes the problem.
In one test we had to restart all containers and only after restarting the last container (om_alpr_worker_1) the problem was fixed.
In a next test we restarted om_alpr_worker_1 first, but this did not fix the problem. In the second test after restarting all but 4 containers, the problem was fixed.
So our conclusion is that stuff running inside the containers cannot be the problem