Memory "lost" in Linux but systemctl restart docker fixes problem (temporarily)

Hello,

we have a problem with missing memory in Linux. Except for the operating system we are runnung all our software in docker containers (somewhere between 30 and 50 containers).
After a reboot everything is fine. However over time (a few days) we are leaking memory. Summing up all memory used by all processes (most of them in docker) (easiest way is using procrank) is much lower than used memory shown in free command. Also summing up memory shown in docker stats is much lower than used (an about the same as summed up by procrank).

If we then restart docker with systemctl restart docker, the missing memory is back. So available memory is much higher and summed up memory is again about the same as used memory shown in free command.

I could not find a known issue similar to this problem but maybe someone knows something.

Thanks

Edit:
Memory usage stated by different tools

                  before         after
procrank          2085114K       2754244K
free used         3055332K       2861756K
free swap         1951740K        721156K
docker stats sum  2120000K       2707000K

After restarting docker, a bit less memory is used, much less in swap but more in RAM. But most importantly before restarting docker there is a gap of about 1GB between the sum used by processes (calculated with procrank and also with docker stats) and after the restart it is only about 100MB. And we are wondering where this memory went in the first place and why restarting docker releases this lost memory.

As i cannot upload files yet, here is the output of the different tools before and after restarting docker:

procrank before restarting docker

  PID       Vss      Rss      Pss      Uss  cmdline
 2593  3549868K  496828K  495720K  495480K  ./app
 6212  4552156K   85596K   85500K   85408K  java
328976   606748K   78888K   77280K   77048K  python
 3278  1431120K   73776K   71914K   71584K  python
1042568  2293996K   68680K   67218K   67096K  python
 2360  1535172K   65284K   63564K   63360K  python
 3497   786952K   53032K   50893K   50440K  python
 4848   421744K   52104K   50274K   49964K  python
  547  2997332K   47588K   47565K   47564K  /usr/bin/dockerd
 4401   490776K   48576K   47143K   47008K  python
 7166   580740K   46456K   45396K   45356K  python
 4236   415900K   44748K   43695K   43652K  python
 3798   417556K   44516K   43592K   43560K  python
 3498  1280036K   40736K   38359K   37684K  ./app
 4602   418312K   37280K   36356K   36324K  python
 3724   523536K   35964K   35090K   35056K  python
 2970   405504K   34880K   33787K   33740K  python
 3361  1206304K   34260K   31922K   31288K  ./app
1405375   134904K   31092K   30200K   29500K  python3
 2678   503964K   30924K   28986K   27980K  python
 4004   497812K   27304K   25735K   25532K  python
 4380   499548K   27668K   25602K   24588K  python
1044632   877844K   25760K   23936K   23284K  python
 6264    42004K   24804K   23767K   23312K  python
 3649   175396K   23716K   23712K   23712K  redis-server
1035856   499168K   24432K   23472K   23432K  python
 4493   334856K   23996K   22907K   22860K  python
 2790  1143784K   23440K   21155K   20536K  ./app
 2612  1143784K   22380K   20075K   19452K  ./app
 2203   689864K   22056K   19808K   19360K  python
  503  1568952K   19756K   19748K   19748K  /usr/bin/containerd
 2601   408464K   16900K   16006K   15972K  python
 1981   413248K   16164K   15196K   15156K  python
657110   624072K   16752K   14471K   14024K  python
 5358   336448K   14616K   13607K   13560K  python
  497   399572K   13676K   12490K   11708K  /usr/bin/python3
432500   878244K   14128K   12358K   11716K  python
 4741   455120K   12860K   11955K   11740K  ./app
 3579   453148K   13328K   11884K   11500K  ./app
 3210   152928K   11324K   11216K   11112K  python
1413366    23648K   10260K    9619K    9368K  python
 6524   481740K   10552K    9493K    9444K  python
657322   556120K   10628K    8898K    8200K  python
 6719   556120K   10456K    8778K    8084K  python
 6780   487816K    9744K    8420K    8308K  python
 5050  1541832K   10624K    8339K    7892K  python
  247   361332K    9612K    8309K    7472K  /lib/systemd/systemd-journald
 2107   689864K   10152K    7957K    7532K  python
 4683   450824K    8216K    7350K    7144K  ./app
1414740   641220K    8144K    7158K    6776K  fish
 2779   450404K    7888K    7140K    6952K  ./app
1413406   708648K    6748K    6733K    6732K  -fish
 2096    36524K    6904K    5829K    5344K  python
 3876    32196K    6308K    5278K    4824K  python
 6726   144688K    4928K    4820K    4716K  python
    1   169320K    5300K    3538K    2256K  /sbin/init
 3192   713584K    5812K    3425K    3372K  /usr/bin/containerd-shim-runc-v2
 3561   713584K    5788K    3401K    3348K  /usr/bin/containerd-shim-runc-v2
 2266   713328K    5844K    3398K    3340K  /usr/bin/containerd-shim-runc-v2
 3319   713584K    5984K    3373K    3296K  /usr/bin/containerd-shim-runc-v2
 6660   713584K    5804K    3358K    3300K  /usr/bin/containerd-shim-runc-v2
 2032   713328K    5800K    3354K    3296K  /usr/bin/containerd-shim-runc-v2
 3671   713392K    5772K    3326K    3268K  /usr/bin/containerd-shim-runc-v2
 4257   713136K    5932K    3321K    3244K  /usr/bin/containerd-shim-runc-v2
 4638   713584K    5764K    3318K    3260K  /usr/bin/containerd-shim-runc-v2
 5313   713584K    5696K    3309K    3256K  /usr/bin/containerd-shim-runc-v2
 2683   713328K    5968K    3299K    3216K  /usr/bin/containerd-shim-runc-v2
 2455   713328K    5900K    3289K    3212K  /usr/bin/containerd-shim-runc-v2
657301   713328K    5636K    3288K    3236K  /usr/bin/containerd-shim-runc-v2
 6487   713328K    5732K    3286K    3228K  /usr/bin/containerd-shim-runc-v2
 2483   713328K    5892K    3281K    3204K  /usr/bin/containerd-shim-runc-v2
 4794   713328K    5884K    3273K    3196K  /usr/bin/containerd-shim-runc-v2
 6672   713328K    5660K    3273K    3220K  /usr/bin/containerd-shim-runc-v2
 4119   713328K    5708K    3262K    3204K  /usr/bin/containerd-shim-runc-v2
 1918   713648K    5612K    3252K    3200K  /usr/bin/containerd-shim-runc-v2
 2562   713328K    5636K    3249K    3196K  /usr/bin/containerd-shim-runc-v2
 2920   713584K    5636K    3249K    3196K  /usr/bin/containerd-shim-runc-v2
 4965   713648K    5636K    3249K    3196K  /usr/bin/containerd-shim-runc-v2
 7146   713328K    5580K    3247K    3196K  /usr/bin/containerd-shim-runc-v2
 3447   713328K    5852K    3241K    3164K  /usr/bin/containerd-shim-runc-v2
 6756   713328K    5852K    3241K    3164K  /usr/bin/containerd-shim-runc-v2
 2682   713584K    5612K    3225K    3172K  /usr/bin/containerd-shim-runc-v2
 3114   713584K    5608K    3221K    3168K  /usr/bin/containerd-shim-runc-v2
 4324   713584K    5604K    3217K    3164K  /usr/bin/containerd-shim-runc-v2
 2158   713328K    5576K    3216K    3164K  /usr/bin/containerd-shim-runc-v2
1044612   713584K    5600K    3213K    3160K  /usr/bin/containerd-shim-runc-v2
 3417   713328K    5816K    3205K    3128K  /usr/bin/containerd-shim-runc-v2
 3763   713328K    5640K    3194K    3136K  /usr/bin/containerd-shim-runc-v2
 4552   713328K    5636K    3190K    3132K  /usr/bin/containerd-shim-runc-v2
1035833   713392K    5536K    3188K    3136K  /usr/bin/containerd-shim-runc-v2
657089   713328K    5560K    3173K    3120K  /usr/bin/containerd-shim-runc-v2
 3901   713328K    5556K    3169K    3116K  /usr/bin/containerd-shim-runc-v2
 2039   713328K    5528K    3168K    3116K  /usr/bin/containerd-shim-runc-v2
 4208   713328K    5540K    3153K    3100K  /usr/bin/containerd-shim-runc-v2
 3409   713392K    5820K    3151K    3068K  /usr/bin/containerd-shim-runc-v2
 3839   713328K    5568K    3122K    3064K  /usr/bin/containerd-shim-runc-v2
 4478   713328K    5496K    3109K    3056K  /usr/bin/containerd-shim-runc-v2
432478   713584K    5472K    3085K    3032K  /usr/bin/containerd-shim-runc-v2
 2435   713392K    5468K    3081K    3028K  /usr/bin/containerd-shim-runc-v2
 6238   713584K    5460K    3073K    3020K  /usr/bin/containerd-shim-runc-v2
328956   713584K    5404K    3056K    3004K  /usr/bin/containerd-shim-runc-v2
1413381   170644K    4140K    3036K    1980K  (sd-pam)
1042548   713584K    5380K    2993K    2940K  /usr/bin/containerd-shim-runc-v2
1042642    12232K    2972K    2968K    2968K  ./tunnel
1413380    19184K    2324K    2308K    2308K  /lib/systemd/systemd
1413404    14180K    2504K    1807K    1332K  sshd: alpr@pts/0
  459   224556K    1872K    1666K    1640K  /usr/sbin/rsyslogd
1416158     3292K    2344K    1467K    1140K  ./procrank
 6323    19172K    1308K    1296K    1296K  /lib/systemd/systemd
1043344    19324K    1340K    1203K    1184K  /lib/systemd/systemd-networkd
1413376    13912K    1484K    1030K     592K  sshd: alpr [priv]
1042703   573792K    1020K    1013K    1012K  -fish
1414738    12660K    1320K     982K     892K  sudo
  446   243056K    1080K     930K     880K  /usr/lib/accountsservice/accounts-daemon
  510    75000K     992K     599K     544K  /usr/sbin/ntpd
  456    32968K     596K     584K     580K  /usr/bin/python3
  448     7772K     560K     548K     548K  /usr/bin/dbus-daemon
1042658    13932K     500K     496K     496K  sshd: root@pts/1
1414739    11768K     504K     492K     492K  su
  523   236672K     488K     476K     476K  /usr/lib/policykit-1/polkitd
  440    24036K     576K     393K     360K  /lib/systemd/systemd-resolved
  455    81936K     820K     339K     220K  /usr/sbin/irqbalance
  461    16980K     464K     325K     296K  /lib/systemd/systemd-logind
  279    21452K     340K     309K     308K  /lib/systemd/systemd-udevd
  447    10316K     804K     259K     132K  /usr/sbin/cron
  514    12292K     168K     160K     160K  sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
  465    10476K     116K     112K     112K  /bin/bash
1042633    16800K      96K      92K      92K  dnsmasq
1042641     2388K      88K      84K      84K  /bin/sh
  509     9332K      64K      60K      60K  /sbin/agetty
 6324   169288K     132K      46K       4K  (sd-pam)
  273     2496K       8K       4K       4K  bpfilter_umh
                           ------   ------  ------
                          2085114K  2060664K  TOTAL

RAM: 3670912K total, 139124K free, 13388K buffers, 345756K cached, 249288K shmem, 266776K slab


free before restarting docker

              total        used        free      shared  buff/cache   available
Mem:        3670912     3055332      176220      249288      439360      145368
Swap:       1951740     1951740           0


docker stats before restarting docker

CONTAINER ID   NAME                         CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
26709ca3dfad   om_payment_cardcomplete_1    0.32%     25.36MiB / 3.501GiB   0.71%     380MB / 156MB     1.21GB / 0B       16
3734e4ed6c50   om_service_updater_1         0.13%     22.89MiB / 3.501GiB   0.64%     332MB / 22.4MB    6.79GB / 0B       7
ebe8ae226e35   om_io_modbus2_2              1.71%     9.605MiB / 3.501GiB   0.27%     8.26GB / 3.53GB   1.28GB / 4.1kB    8
c1f71c6edeae   om_service_trafficlight2_4   0.50%     14.89MiB / 3.501GiB   0.42%     7.26GB / 548MB    733MB / 0B        5
7499fc2f2515   om_payment_cardcomplete_2    0.40%     12.79MiB / 3.501GiB   0.36%     985MB / 411MB     1.67GB / 0B       16
bcda056a19da   om_service_relay2_1          0.08%     75.02MiB / 3.501GiB   2.09%     1.36GB / 117MB    10.2GB / 4.1kB    7
1a8a784fbb29   om_controller_freeflow2_1    0.49%     71.09MiB / 3.501GiB   1.98%     1.77GB / 1.43GB   14.1GB / 11.8GB   23
19862dc0bc6e   om_service_dsgvo2_1          0.12%     14.88MiB / 3.501GiB   0.42%     925MB / 129MB     9.88GB / 367MB    6
757a191de560   om_service_enforcement_1     0.09%     44.71MiB / 3.501GiB   1.25%     1.11GB / 221MB    6.86GB / 382MB    6
289e63f28699   om_service_misc_1            1.85%     48.96MiB / 3.501GiB   1.37%     9.25GB / 1.35GB   6.82GB / 4.1kB    7
345b039b2da7   om_service_databox_1         0.06%     33.8MiB / 3.501GiB    0.94%     925MB / 118MB     5.58GB / 10.2MB   7
f3436f0e3b45   om_alpr_nn_2                 2.63%     34.03MiB / 3.501GiB   0.95%     1.27TB / 11.7GB   1.57GB / 119MB    17
2a4db9e14299   om_alpr_nn_1                 2.57%     31.51MiB / 3.501GiB   0.88%     1.5TB / 11.3GB    1.03GB / 110MB    17
ecc0277e9d95   om_alpr_nn_4                 3.24%     21.3MiB / 3.501GiB    0.59%     845GB / 28.4GB    1.4GB / 109MB     17
597c3cc43cc4   om_alpr_nn_3                 3.53%     19.96MiB / 3.501GiB   0.56%     944GB / 30.1GB    1.54GB / 101MB    17
4d7c00e670b5   om_service_eventlog2_1       0.19%     43.12MiB / 3.501GiB   1.20%     1.47GB / 208MB    9.95GB / 21.9GB   6
58da41445c55   om_fancy_assistant2_1        0.54%     69.7MiB / 3.501GiB    1.94%     3.74GB / 280MB    19.1GB / 32.8kB   35
800c8d08a7c8   om_service_stats2_1          0.01%     32.86MiB / 3.501GiB   0.92%     945MB / 5.62GB    45.5GB / 4.1kB    6
85967174ec17   om_alpr_worker_1             5.00%     737.5MiB / 3.501GiB   20.57%    880MB / 74.3MB    7.94GB / 0B       40
32ff60cb09d8   om_alpr_tracking_1           0.27%     12.29MiB / 3.501GiB   0.34%     40.5GB / 5.81GB   572MB / 16.4kB    8
313465b59bbe   om_alpr_tracking_2           0.30%     10.89MiB / 3.501GiB   0.30%     40.5GB / 6.13GB   1.11GB / 16.4kB   8
ffac8921de04   om_alpr_tracking_3           0.27%     7.668MiB / 3.501GiB   0.21%     40.4GB / 4.97GB   837MB / 16.4kB    8
2c7c9c7ade0b   om_alpr_tracking_4           0.28%     7.309MiB / 3.501GiB   0.20%     40.5GB / 4.97GB   869MB / 16.4kB    8
fe25745ab4e9   om_io_modbus2_1              1.38%     9.34MiB / 3.501GiB    0.26%     15.1GB / 6.55GB   1.35GB / 4.1kB    8
3d568b264f8f   om_service_doorreader_1      0.36%     7.738MiB / 3.501GiB   0.22%     1.07GB / 115MB    5.63GB / 4.1kB    7
8d5ead4cf25c   om_access_oebb_1             0.03%     48.91MiB / 3.501GiB   1.36%     1.04GB / 77.3MB   6.62GB / 1.04GB   6
b80d9b997dfb   om_oebb_api_1                0.02%     21.77MiB / 3.501GiB   0.61%     1.07GB / 110MB    7.12GB / 0B       5
319969a79a63   om_access_arivo_1            0.03%     35.73MiB / 3.501GiB   1.00%     1.1GB / 82.4MB    6.42GB / 1.44MB   6
f8b63d0400c8   om_validation_arivo2_1       0.12%     15.59MiB / 3.501GiB   0.43%     929MB / 106MB     5.35GB / 856kB    6
841c5c71ee05   om_access_whitelist_1        0.02%     42.29MiB / 3.501GiB   1.18%     1.03GB / 71.3MB   6.15GB / 201kB    6
7e8b839902ad   om_service_setup_1           0.00%     8.32MiB / 3.501GiB    0.23%     0B / 0B           6.21GB / 4.1kB    7
81ac1a1511a7   om_validation_oebb_1         0.32%     113.1MiB / 3.501GiB   3.15%     1.18GB / 101MB    20.5GB / 60.9MB   38
e80c3c014f8c   om_service_trafficlight2_2   0.40%     8.84MiB / 3.501GiB    0.25%     13.3GB / 1.01GB   840MB / 0B        5
23e2909734ad   om_service_trafficlight2_3   0.56%     20.75MiB / 3.501GiB   0.58%     13.3GB / 1.01GB   739MB / 0B        5
09f8db49a8f8   om_medium_http_1             0.01%     12.16MiB / 3.501GiB   0.34%     884MB / 55.8MB    5.08GB / 0B       5
d96ae05c1586   om_io_display2_2             0.26%     25.29MiB / 3.501GiB   0.71%     1.36GB / 2.19GB   4.4GB / 430kB     7
f2120a584111   om_io_display2_1             0.27%     28.68MiB / 3.501GiB   0.80%     1.37GB / 2.43GB   3.07GB / 430kB    7
3dc1cf8f08ee   om_fancy_sentry_1            0.00%     23.27MiB / 3.501GiB   0.65%     307GB / 3.77GB    1.56GB / 0B       9
a8dbfd65d0aa   om_service_logger_1          0.00%     3.867MiB / 3.501GiB   0.11%     1.01GB / 252MB    2.71GB / 4.1kB    7
d37f6580704d   om_service_deeplog_1         0.32%     6.27MiB / 3.501GiB    0.17%     53.4GB / 3.82GB   64.4GB / 0B       8
1592d9f1f39c   om_service_camoperator_1     1.79%     63.67MiB / 3.501GiB   1.78%     8.85GB / 4.58GB   6.4GB / 70.4MB    21
cdb234206b0e   om_service_trafficlight2_1   0.62%     8.469MiB / 3.501GiB   0.24%     13.3GB / 1.01GB   757MB / 0B        5
660116d74a99   om_rpc_client_1              0.14%     48.88MiB / 3.501GiB   1.36%     2.4GB / 1.48GB    9.1GB / 5.52GB    9
60187c214c10   om_service_shootback_1       1.60%     15.56MiB / 3.501GiB   0.43%     1.02GB / 168MB    452MB / 0B        17
183d2d3538ff   om_infra_redis_1             0.12%     138MiB / 3.501GiB     3.85%     3.89GB / 306GB    2.04GB / 72.9GB   4
1c04f6df217c   om_service_broker_1          0.85%     11.91MiB / 3.501GiB   0.33%     83.5GB / 402GB    261MB / 0B        3


procrank after restarting docker

  PID       Vss      Rss      Pss      Uss  cmdline
1420175  3484332K  1030104K  1027713K  1026824K  ./app
1420610   167204K  142432K  142428K  142428K  redis-server
1422351  4550296K   94168K   93486K   93016K  java
1418960  1126116K   58484K   54630K   54376K  python
1419447   496264K   56120K   52065K   51420K  python
1420334   605592K   50280K   47330K   47224K  python
1417524  2914200K   48312K   45733K   45484K  /usr/bin/dockerd
1421760  2135016K   49168K   45727K   45480K  python
1422581   144708K   44168K   43702K   43240K  python
1420508   433248K   46824K   43307K   43148K  python
1422594   555528K   44208K   38814K   36676K  python
1422673   555608K   44160K   38772K   36636K  python
1419515   500468K   43316K   38630K   37040K  python
1422538   481488K   41948K   38544K   38372K  python
1418385   498912K   40960K   37019K   36060K  python
1420358   487700K   38624K   35642K   35532K  python
1417885   559144K   39896K   35216K   34092K  python
1418602   496528K   38216K   34947K   34804K  python
1419502   499932K   39144K   34671K   33140K  python
1420031   408676K   38344K   34577K   34340K  python
1420860   409192K   36312K   32638K   32420K  python
1419932  1517764K   35596K   32378K   32252K  python
1418631   481756K   34892K   31194K   30896K  python
1418217  1209320K   34004K   30096K   28448K  ./app
1419394   408172K   33128K   29607K   29432K  python
1419931   408564K   33008K   29532K   29368K  python
1420908   335804K   32224K   29496K   29400K  python
1419966   149484K   29708K   29242K   28780K  python
1419273   407980K   32448K   29080K   28936K  python
1420841   332468K   31012K   28078K   27964K  python
1418621   409348K   30252K   27429K   27328K  python
1419712   404348K   30288K   27272K   27156K  python
1422589    39700K   27968K   26810K   25956K  python
1418368   872748K   28916K   25755K   25116K  python
1419805   872744K   28676K   25117K   24488K  python
1420401   402108K   28264K   24088K   23560K  python
1418792   402108K   28216K   24054K   23536K  python
1418026   402108K   27436K   23947K   23468K  python
1417801   402108K   27392K   23918K   23432K  python
  503  1568952K   21616K   21592K   21592K  /usr/bin/containerd
1420494   928132K   19156K   16371K   15684K  ./app
1418194   862596K   17352K   14549K   13844K  ./app
1420597    31444K   15296K   14156K   13320K  python
  247   377848K   16092K   13126K   12024K  /lib/systemd/systemd-journald
1419225    31140K   13232K   12612K   12296K  python
1418027   862596K   15328K   12482K   11724K  ./app
  497   399572K   12372K   10304K    9852K  /usr/bin/python3
1405375   134904K    9980K    7962K    7508K  python3
1417119    30068K   10524K    6738K    5988K  /usr/bin/python3
1420522   446300K    9040K    6385K    5420K  ./app
1422210  1818868K    7588K    4738K    3916K  -fish
1418594   446304K    5764K    4188K    3884K  ./app
1418367   446300K    5440K    4134K    3864K  ./app
1418205   446300K    5352K    4105K    3852K  ./app
    1   169212K    7040K    3750K    2692K  /sbin/init
1417833   713008K    5316K    3557K    2204K  /usr/bin/containerd-shim-runc-v2
1417740   712752K    5288K    3529K    2176K  /usr/bin/containerd-shim-runc-v2
  456    32968K    5524K    3414K    2888K  /usr/bin/python3
1422178    19188K    7464K    3097K    2336K  /lib/systemd/systemd
1423059   180572K    5720K    2765K    1924K  -fish
1422501   712688K    2980K    2569K    2560K  /usr/bin/containerd-shim-runc-v2
1422528   712688K    2888K    2466K    2456K  /usr/bin/containerd-shim-runc-v2
1421700   712688K    2820K    2409K    2400K  /usr/bin/containerd-shim-runc-v2
1422453   712688K    2784K    2373K    2364K  /usr/bin/containerd-shim-runc-v2
1422498   712432K    2732K    2321K    2312K  /usr/bin/containerd-shim-runc-v2
1422973    13928K    5888K    2220K    1616K  sshd: root@pts/4
1423298    12836K    4720K    2214K    1904K  sudo
1420838   712432K    2616K    2205K    2196K  /usr/bin/containerd-shim-runc-v2
1422650   711024K    2592K    2181K    2172K  /usr/bin/containerd-shim-runc-v2
1417197    18808K    5264K    2115K    1740K  /lib/systemd/systemd-networkd
 6323    19172K    4104K    1770K    1404K  /lib/systemd/systemd
1418883   712688K    2140K    1729K    1720K  /usr/bin/containerd-shim-runc-v2
1420533   712432K    1948K    1537K    1528K  /usr/bin/containerd-shim-runc-v2
1422634     9864K    1536K    1532K    1532K  ./tunnel
1420268   711024K    1900K    1489K    1480K  /usr/bin/containerd-shim-runc-v2
  446   243056K    3272K    1487K    1128K  /usr/lib/accountsservice/accounts-daemon
1420200   711280K    1884K    1473K    1464K  /usr/bin/containerd-shim-runc-v2
1423205    16264K    5348K    1470K     900K  /lib/systemd/systemd-hostnamed
1420026   711024K    1852K    1441K    1432K  /usr/bin/containerd-shim-runc-v2
1419638   711280K    1848K    1437K    1428K  /usr/bin/containerd-shim-runc-v2
1422094    13908K    5352K    1412K     484K  sshd: alpr [priv]
1419131   712432K    1808K    1397K    1388K  /usr/bin/containerd-shim-runc-v2
1420466   711024K    1784K    1373K    1364K  /usr/bin/containerd-shim-runc-v2
1419650   711280K    1780K    1369K    1360K  /usr/bin/containerd-shim-runc-v2
1419683   711280K    1780K    1369K    1360K  /usr/bin/containerd-shim-runc-v2
1419376   712688K    1776K    1365K    1356K  /usr/bin/containerd-shim-runc-v2
1420197   712688K    1760K    1349K    1340K  /usr/bin/containerd-shim-runc-v2
1418313   712432K    1752K    1341K    1332K  /usr/bin/containerd-shim-runc-v2
1420223   712432K    1744K    1333K    1324K  /usr/bin/containerd-shim-runc-v2
1420056   712432K    1740K    1329K    1320K  /usr/bin/containerd-shim-runc-v2
1419889   712688K    1720K    1309K    1300K  /usr/bin/containerd-shim-runc-v2
1419162   712688K    1700K    1289K    1280K  /usr/bin/containerd-shim-runc-v2
1420282   711024K    1700K    1289K    1280K  /usr/bin/containerd-shim-runc-v2
1422054    15548K    4144K    1265K     904K  ssh
1419327   711280K    1660K    1249K    1240K  /usr/bin/containerd-shim-runc-v2
1419274   712432K    1648K    1237K    1228K  /usr/bin/containerd-shim-runc-v2
1420696   712432K    1632K    1221K    1212K  /usr/bin/containerd-shim-runc-v2
1417991   711280K    1616K    1205K    1196K  /usr/bin/containerd-shim-runc-v2
1418179   712432K    1608K    1197K    1188K  /usr/bin/containerd-shim-runc-v2
1422179   170536K    3104K    1171K     508K  (sd-pam)
1419740   712688K    1564K    1153K    1144K  /usr/bin/containerd-shim-runc-v2
1420595   711280K    1560K    1149K    1140K  /usr/bin/containerd-shim-runc-v2
1417951   710768K    1528K    1117K    1108K  /usr/bin/containerd-shim-runc-v2
1418288   712432K    1516K    1105K    1096K  /usr/bin/containerd-shim-runc-v2
1418589   712432K    1508K    1097K    1088K  /usr/bin/containerd-shim-runc-v2
1418526   712688K    1460K    1049K    1040K  /usr/bin/containerd-shim-runc-v2
1418134   712432K    1456K    1045K    1036K  /usr/bin/containerd-shim-runc-v2
1418290   711280K    1456K    1045K    1036K  /usr/bin/containerd-shim-runc-v2
1418476   712688K    1436K    1025K    1016K  /usr/bin/containerd-shim-runc-v2
1418109   712688K    1432K    1021K    1012K  /usr/bin/containerd-shim-runc-v2
1423308     3300K    2424K     996K     928K  ./procrank
1419413   711280K    1384K     973K     964K  /usr/bin/containerd-shim-runc-v2
  459   224556K    1704K     968K     940K  /usr/sbin/rsyslogd
1418532   711280K    1364K     953K     944K  /usr/bin/containerd-shim-runc-v2
  440    24036K    2864K     928K     688K  /lib/systemd/systemd-resolved
1418449   712688K    1328K     917K     908K  /usr/bin/containerd-shim-runc-v2
  448     7772K    2076K     869K     708K  /usr/bin/dbus-daemon
1419739   711024K    1260K     849K     840K  /usr/bin/containerd-shim-runc-v2
  523   236672K    1996K     840K     528K  /usr/lib/policykit-1/polkitd
1422205    14044K    3484K     806K     124K  sshd: alpr@pts/0
  461    16972K    2656K     660K     456K  /lib/systemd/systemd-logind
  279    21452K    1912K     632K     572K  /lib/systemd/systemd-udevd
  510    75000K    1340K     523K     488K  /usr/sbin/ntpd
  455    81936K    1464K     320K     248K  /usr/sbin/irqbalance
  514    12292K    1668K     304K     156K  sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
1417118    10484K    1536K     257K     176K  SCREEN
1421897    16800K     412K     235K     140K  dnsmasq
  447    10316K     952K     162K     132K  /usr/sbin/cron
1422053     2616K    1504K     160K      96K  /bin/sh
1422629     2388K     296K     157K      88K  /bin/sh
  465    10476K      64K      60K      60K  /bin/bash
 6324   169288K     128K      42K       0K  (sd-pam)
  509     9332K      24K      20K      20K  /sbin/agetty
  273     2496K      84K      18K      16K  bpfilter_umh
                           ------   ------  ------
                          2754244K  2710892K  TOTAL

RAM: 3670912K total, 138044K free, 20380K buffers, 533848K cached, 287776K shmem, 245052K slab


free after restarting docker

              total        used        free      shared  buff/cache   available
Mem:        3670912     2861756      136304      284224      672852      260628
Swap:       1951740      721156     1230584


docker stats after restarting docker

CONTAINER ID   NAME                         CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
26709ca3dfad   om_payment_cardcomplete_1    9.35%     26.33MiB / 3.501GiB   0.73%     122kB / 83.6kB    29MB / 0B         16
3734e4ed6c50   om_service_updater_1         0.07%     33.77MiB / 3.501GiB   0.94%     221kB / 43.9kB    45MB / 4.1kB      7
ebe8ae226e35   om_io_modbus2_2              2.25%     36.88MiB / 3.501GiB   1.03%     157kB / 64.6kB    12.5MB / 4.1kB    8
c1f71c6edeae   om_service_trafficlight2_4   0.83%     23.41MiB / 3.501GiB   0.65%     180kB / 34.6kB    16.2MB / 0B       5
7499fc2f2515   om_payment_cardcomplete_2    0.37%     23.39MiB / 3.501GiB   0.65%     90.6kB / 56kB     5.92MB / 0B       16
bcda056a19da   om_service_relay2_1          0.32%     44.65MiB / 3.501GiB   1.25%     188kB / 24.4kB    81.7MB / 4.1kB    7
1a8a784fbb29   om_controller_freeflow2_1    4.04%     53.89MiB / 3.501GiB   1.50%     78.2kB / 32.1kB   36.8MB / 24.6kB   20
19862dc0bc6e   om_service_dsgvo2_1          0.16%     26.22MiB / 3.501GiB   0.73%     54kB / 30.2kB     36.5MB / 0B       6
757a191de560   om_service_enforcement_1     0.00%     40.19MiB / 3.501GiB   1.12%     51kB / 27.2kB     44.1MB / 4.1kB    6
289e63f28699   om_service_misc_1            0.82%     31.2MiB / 3.501GiB    0.87%     90kB / 58.4kB     32.2MB / 4.1kB    7
345b039b2da7   om_service_databox_1         0.09%     32.82MiB / 3.501GiB   0.92%     46.7kB / 24.2kB   37.2MB / 8.19kB   7
f3436f0e3b45   om_alpr_nn_2                 3.57%     16.55MiB / 3.501GiB   0.46%     51.5MB / 92.9kB   10.8MB / 4.1kB    14
2a4db9e14299   om_alpr_nn_1                 33.71%    23.59MiB / 3.501GiB   0.66%     91.8MB / 470kB    26.6MB / 4.1kB    17
ecc0277e9d95   om_alpr_nn_4                 4.50%     15.27MiB / 3.501GiB   0.43%     7.11MB / 315kB    2.56MB / 4.1kB    14
597c3cc43cc4   om_alpr_nn_3                 0.00%     15.67MiB / 3.501GiB   0.44%     7.69MB / 324kB    28.9MB / 4.1kB    14
4d7c00e670b5   om_service_eventlog2_1       0.00%     33.8MiB / 3.501GiB    0.94%     74.7kB / 29.7kB   32.3MB / 930kB    6
58da41445c55   om_fancy_assistant2_1        0.00%     48.5MiB / 3.501GiB    1.35%     192kB / 47.3kB    68.8MB / 32.8kB   35
800c8d08a7c8   om_service_stats2_1          0.00%     24.99MiB / 3.501GiB   0.70%     44.6kB / 26kB     56MB / 4.1kB      6
85967174ec17   om_alpr_worker_1             0.00%     1.224GiB / 3.501GiB   34.98%    53.9kB / 30.1kB   211MB / 0B        40
32ff60cb09d8   om_alpr_tracking_1           0.00%     6.703MiB / 3.501GiB   0.19%     650kB / 121kB     5.3MB / 16.4kB    8
313465b59bbe   om_alpr_tracking_2           1.34%     4.219MiB / 3.501GiB   0.12%     652kB / 89.1kB    995kB / 16.4kB    8
ffac8921de04   om_alpr_tracking_3           1.46%     4.32MiB / 3.501GiB    0.12%     653kB / 89.1kB    2MB / 16.4kB      8
2c7c9c7ade0b   om_alpr_tracking_4           1.27%     4.867MiB / 3.501GiB   0.14%     653kB / 90.1kB    7.1MB / 16.4kB    8
fe25745ab4e9   om_io_modbus2_1              2.68%     37.7MiB / 3.501GiB    1.05%     158kB / 66.4kB    18.9MB / 4.1kB    8
3d568b264f8f   om_service_doorreader_1      0.84%     32.95MiB / 3.501GiB   0.92%     44.2kB / 17kB     32.8MB / 4.1kB    7
8d5ead4cf25c   om_access_oebb_1             0.00%     28.76MiB / 3.501GiB   0.80%     46.9kB / 21.8kB   29.7MB / 0B       6
b80d9b997dfb   om_oebb_api_1                0.00%     26.4MiB / 3.501GiB    0.74%     29.4kB / 13.3kB   30.2MB / 0B       5
319969a79a63   om_access_arivo_1            0.10%     32.11MiB / 3.501GiB   0.90%     46.7kB / 22.1kB   32.4MB / 4.1kB    6
f8b63d0400c8   om_validation_arivo2_1       0.86%     28.52MiB / 3.501GiB   0.80%     46.1kB / 23.8kB   30.4MB / 4.1kB    6
841c5c71ee05   om_access_whitelist_1        0.00%     31.97MiB / 3.501GiB   0.89%     50.7kB / 24.3kB   33.6MB / 4.1kB    6
7e8b839902ad   om_service_setup_1           0.42%     35.54MiB / 3.501GiB   0.99%     0B / 0B           30.7MB / 4.1kB    7
81ac1a1511a7   om_validation_oebb_1         27.80%    143.5MiB / 3.501GiB   4.00%     47.4kB / 20.4kB   112MB / 0B        39
e80c3c014f8c   om_service_trafficlight2_2   0.00%     22.82MiB / 3.501GiB   0.64%     180kB / 34kB      1.74MB / 0B       5
23e2909734ad   om_service_trafficlight2_3   1.33%     23.27MiB / 3.501GiB   0.65%     182kB / 31.9kB    9.98MB / 0B       6
09f8db49a8f8   om_medium_http_1             2.26%     31MiB / 3.501GiB      0.86%     26.5kB / 10.7kB   31.7MB / 0B       5
d96ae05c1586   om_io_display2_2             0.79%     32.57MiB / 3.501GiB   0.91%     65.3kB / 58kB     18.5MB / 430kB    7
f2120a584111   om_io_display2_1             0.98%     36.37MiB / 3.501GiB   1.01%     72.8kB / 181kB    21.9MB / 430kB    7
3dc1cf8f08ee   om_fancy_sentry_1            0.00%     25.86MiB / 3.501GiB   0.72%     2.78MB / 31.7kB   23MB / 0B         9
a8dbfd65d0aa   om_service_logger_1          0.00%     40.68MiB / 3.501GiB   1.13%     16.7kB / 11.7kB   18.3MB / 4.1kB    7
d37f6580704d   om_service_deeplog_1         0.00%     13.63MiB / 3.501GiB   0.38%     815kB / 81.2kB    5.15MB / 0B       8
1592d9f1f39c   om_service_camoperator_1     1.90%     30.6MiB / 3.501GiB    0.85%     158kB / 84.5kB    27.5MB / 4.1kB    21
cdb234206b0e   om_service_trafficlight2_1   0.00%     22.61MiB / 3.501GiB   0.63%     175kB / 31kB      4.24MB / 0B       5
660116d74a99   om_rpc_client_1              0.00%     35.9MiB / 3.501GiB    1.00%     61.6kB / 51.7kB   30.8MB / 53.2kB   8
60187c214c10   om_service_shootback_1       0.00%     11.78MiB / 3.501GiB   0.33%     88.1MB / 47.4kB   6.35MB / 0B       8
183d2d3538ff   om_infra_redis_1             0.00%     130.4MiB / 3.501GiB   3.64%     38.9kB / 2.78MB   167MB / 0B        4
1c04f6df217c   om_service_broker_1          6.27%     27.12MiB / 3.501GiB   0.76%     2.2MB / 6.64MB    5.75MB / 0B       3

Edit 2:
Restarting all docker container fixes the problem.
In one test we had to restart all containers and only after restarting the last container (om_alpr_worker_1) the problem was fixed.
In a next test we restarted om_alpr_worker_1 first, but this did not fix the problem. In the second test after restarting all but 4 containers, the problem was fixed.
So our conclusion is that stuff running inside the containers cannot be the problem

Make sure to be comparing the same kind of memory. Linux does some caching and some commends show it as used, some as free (as it can be freed any time).

As far as we know we are comparing the correct memory values (see Edit). We are also getting services killed by the out of memory killer which is our real problem here. So the memory is definitely used according to linux and we don’t know what is leaking this memory as no process seems to use it.

I have a same problem on ubuntu server 20.04.6, this production server have 32GB of memory running 33 containers (all .net core 5 and 8) and limit memory 512MB per container total is 16GB. after running 1 week memory grow up to 30GB( some time memory grow up and server stop working, need to hard restart), i try to stop all containers memory drop down a little bit, then i try to restart docker.service now memory drop down below 5GB, and I have developer server is config and run the same production but is not have a same problem. now i try to figure out what a problem.

Have you updated Ubuntu and Docker to latest? Which versions are you running?

We have one image mis-behaving and it is correctly killed when reaching memory limit, on Debian with Docker CE.

I have update docker to last version, But is not solve problem. now i change to fedora 40, i hope it can help.

In our usual setup the communication between our services (every service has its own docker container) is done with zmq (https://zeromq.org/) using TCP. Here we experience the problem.

Now we ran different tests:

  • use --network=host instead of a docker network: Same problem
  • use IPC instead of TCP in zmq: Solves problem

So there seems to be a network related problem .

We found the problem:
We communicate between our services with zmq using TCP.
zmq has a weird quirk:

  • pub sockets receives data for new subscriptions
  • pub sockets only process this data when it publishes a new message
    Because of this, if you have pub sockets which do not publish at all for some time and there are a lot of subscribe and unsubscribe messages, these messages accumulate in the tcp buffer, creating this memory leak
1 Like