Postgres Docker: Weird hidden temp folders in overlay2

jarnovs · July 23, 2025, 9:22am

I found a folder in overlay2 that takes too much space in disk (47GB).

df -h

Filesystem      Size  Used Avail Use% Mounted on
overlay          59G   57G   2G  97% /var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/diff

Then i found the container whom it belongs to by

docker inspect $(docker ps -q) | grep "2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d"

It was postgres:

"GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d-init/diff:/var/lib/docker/overlay2/7d50ba641ce957cc3c0321d27255c43735758cc6728f40130ccf7647b65ad233/diff:/var/lib/docker/overlay2/40a587cc6f3be0b18110c90509183b487505d9dc189f304652f7f7be7876a10e/diff:/var/lib/docker/overlay2/fb9d0c414e9bcb9d928280094740624a89186a4c5dcf00d498859c00c267f9fb/diff:/var/lib/docker/overlay2/aa047b8451d960dd27456bd0e458530aaa063d0758f1d92ea13ced051443fa6e/diff:/var/lib/docker/overlay2/890f3834764b95663138b19f9bd6a561667394bdc33e1eaecb404fad394f1ff7/diff:/var/lib/docker/overlay2/1309aecf54255c28ac3107dd51013b6d469efcd25dce9b66c7006acdc6e9c6ee/diff:/var/lib/docker/overlay2/0b84a10532eacd9849def9e5f8364c46e0f87ced1631edb4c985bad32c46cab0/diff:/var/lib/docker/overlay2/55a815ca599679ba7a63b2c09c6e4bbd1095ac3f4eba7598ce55eff457091188/diff:/var/lib/docker/overlay2/f58d216a605ce8b79bc3b6c6fc63e84303d8161339529b508d4d302595e3b1a0/diff:/var/lib/docker/overlay2/a7d89e0b09d6947f7b8f273a6be5196abc2494da61e2519d465a89fd4e291fc9/diff:/var/lib/docker/overlay2/87d7a4d8f6ef8dfffb7aa50f470f605f2512d06beb49b6f2c4d0795f49fd337b/diff:/var/lib/docker/overlay2/0262f038f6e246db951a6ae826672d9ba35018de04fcbfdd52df041a4cfb2125/diff:/var/lib/docker/overlay2/6d2113ec32d4be4247b5c01875f698f3c14c4d0d04466926a9bb97954e0d8c71/diff:/var/lib/docker/overlay2/28174875a68a44b3e17be86c7c075e3b7b397a4a1c0ba7cec3539e1a187e383c/diff",
                "MergedDir": "/var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/merged",
                "UpperDir": "/var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/diff",
                "WorkDir": "/var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "builder_postgres_data",
                "Source": "/var/lib/docker/volumes/builder_postgres_data/_data",
                "Destination": "/var/lib/postgresql/data",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            }
        ],

Confirmation:

docker ps -as

CONTAINER ID   IMAGE                COMMAND                  CREATED       STATUS                     PORTS                                                                      NAMES      SIZE
77dcd2012a76   postgres             "docker-entrypoint.s…"   3 weeks ago   Exited 2 hours ago                 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                  postgres   42GB (virtual 47GB)

docker system df

TYPE            TOTAL     ACTIVE    SIZE   
Images          13        5         3.49GB  
Containers      6         4         42GB  
Local Volumes   62        5         1.724GB  
Build Cache     0         0         0B

I entered these directory and find strange folders with files.

root@it:~# cd /var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/diff
root@it:/var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/diff# ls
run  tmp
root@it:/var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/diff# cd tmp
root@it:/var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/diff/tmp# ls -a
.  ..  .lckpj1fj  .pirkjdnkg  .pj1fj1  .zesycgwfd ...(18000 folder and files like these)
root@it:/var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/diff/tmp# cd .pirkjdnkg/
root@it:/var/lib/docker/overlay2/2efd9346078d4c841d1ed0b9d75e3f0ea0dd6ae7b1b0a608f3b02071bdb5115d/diff/tmp/.pirkjdnkg# ls
cnsrknage

Each folder has 1 file that weights 2.6 MB.
Download example: cnsrknage

Other filenames from that folder:
docker diff postgres

A /tmp/.pkvywpbzv/raddhnuwi
A /tmp/.vlzbogolh
A /tmp/.vlzbogolh/aguxbnnyi
A /tmp/.xtvnndzqd
A /tmp/.xtvnndzqd/zmgzmlrju
A /tmp/.ezkrrvuxn
A /tmp/.ezkrrvuxn/vryccghrd
A /tmp/.fvagbtkwj
A /tmp/.fvagbtkwj/hiexxwqxp
A /tmp/.lxiyxwnez
A /tmp/.lxiyxwnez/bextnxcgp
A /tmp/.nfnhptyvz
A /tmp/.nfnhptyvz/qogbrnxed
A /tmp/.ooobbmryu
A /tmp/.ooobbmryu/wwhfqsceb
A /tmp/.tiqzcrhdn
A /tmp/.tiqzcrhdn/nvuuqwzhl
A /tmp/.cxiixjvnt
A /tmp/.cxiixjvnt/ppsmdoadj
A /tmp/.ermqjvxtc
A /tmp/.ermqjvxtc/vxvvqnrhe
A /tmp/.fvtqwwoww
A /tmp/.fvtqwwoww/okryuvqov
A /tmp/.ilkmckwmu
A /tmp/.ilkmckwmu/evtamwiiv
A /tmp/.liewgdhfm
A /tmp/.liewgdhfm/spuclhwzx
A /tmp/.imzyyrpiu
A /tmp/.imzyyrpiu/htjpszkrp
A /tmp/.inytkxmzl
A /tmp/.inytkxmzl/koapokzzu
A /tmp/.ujbiwkrju
A /tmp/.ujbiwkrju/jzvzsuzhr
A /tmp/.enuifmazs
A /tmp/.enuifmazs/nbwpiavkc
A /tmp/.kfjyarsfg
A /tmp/.kfjyarsfg/puuxdcxfj
A /tmp/.kuylqnhwn
A /tmp/.kuylqnhwn/tvrmqfadu
A /tmp/.nmvxgepyn
A /tmp/.nmvxgepyn/eygpspvka
A /tmp/.wuqzcyggj
A /tmp/.wuqzcyggj/dthsrrnrw
A /tmp/.yolbgaqrr
A /tmp/.yolbgaqrr/sddsfqunw
A /tmp/.zrzkjcysy
A /tmp/.zrzkjcysy/ekvwrjqyy
A /tmp/.jltjgocpp
A /tmp/.jltjgocpp/gtgwbrpma
A /tmp/.qnqvrshto
A /tmp/.qnqvrshto/djgvnuxxg
A /tmp/.gefklqyqx
A /tmp/.gefklqyqx/zqjexorml
A /tmp/.gstkfjfrj
A /tmp/.gstkfjfrj/uwlifisuk
A /tmp/.iyvxvjety
A /tmp/.iyvxvjety/xxdcxolox
A /tmp/.njqxnbdvw
A /tmp/.njqxnbdvw/fkffdiepy
A /tmp/.nkpnkmrvi
A /tmp/.nkpnkmrvi/kbqohhdfb
A /tmp/.ialyxooyv
A /tmp/.ialyxooyv/qeqcwnmmb
A /tmp/.jwxrmldbj
A /tmp/.jwxrmldbj/dumvggvvm
A /tmp/.mpotmxoaj
A /tmp/.mpotmxoaj/dvptbdugq
A /tmp/.mshjucbzy
A /tmp/.mshjucbzy/vjbgfmcnt
A /tmp/.nqdrpzxyw
A /tmp/.nqdrpzxyw/octzrdlvy
A /tmp/.itjjyxxug
A /tmp/.itjjyxxug/lpxtepvpl
A /tmp/.qcowypcyr
A /tmp/.qcowypcyr/zqillzdka
A /tmp/.wudjegncz
A /tmp/.wudjegncz/bbdflgzby
A /tmp/.yiseqokky
A /tmp/.yiseqokky/gmpejvruo
A /tmp/.agknjcpld
A /tmp/.agknjcpld/mxpqhsteo
A /tmp/.fhyatwfqz
A /tmp/.fhyatwfqz/pxehfdtcu
A /tmp/.pdubjpxgg
A /tmp/.pdubjpxgg/tdjcqoslq
A /tmp/.sujiyblnr
A /tmp/.sujiyblnr/egnspftgr
A /tmp/.jlplnqdcq
A /tmp/.jlplnqdcq/wyjondemk
A /tmp/.qmluurwfo
A /tmp/.qmluurwfo/qvyoayjrh
A /tmp/.vksodkkpl
A /tmp/.vksodkkpl/ogqtauiso
A /tmp/.wrmznqcet
A /tmp/.wrmznqcet/vtmcmpaqc
A /tmp/.ypeyvtumd
A /tmp/.ypeyvtumd/vcstwindo
A /tmp/.yxtaitqpz
A /tmp/.yxtaitqpz/xllfryugd
A /tmp/.ahczxfqgt
A /tmp/.ahczxfqgt/kriwirded
A /tmp/.baqlrbmmh
A /tmp/.baqlrbmmh/remfzwblp
A /tmp/.dhlctpkes
A /tmp/.dhlctpkes/cvkuwnmtw
A /tmp/.dqfnvpmyl
A /tmp/.dqfnvpmyl/nvyeeawbu
A /tmp/.ukanaqzed
A /tmp/.ukanaqzed/qazskkzhl
A /tmp/.vrmkbbscg
A /tmp/.vrmkbbscg/lkeevvmhs
A /tmp/.afegkrpqf
A /tmp/.afegkrpqf/tvcjdrzga
A /tmp/.itxotgmwt
A /tmp/.itxotgmwt/mrjylupmu
A /tmp/.mjkqdrgqk
A /tmp/.mjkqdrgqk/kjyxwuyxt
A /tmp/.wagohawzb
A /tmp/.wagohawzb/xtszfjali
A /tmp/.kvzcihofu
A /tmp/.kvzcihofu/calocqxhy
A /tmp/.nfemlrezc
A /tmp/.nfemlrezc/dgfxshyhj
A /tmp/.oukhhftot
A /tmp/.oukhhftot/aihlqfjue
A /tmp/.wdnglkeqw
A /tmp/.wdnglkeqw/rljirbwqi
A /tmp/.wqbvfcnkw
A /tmp/.wqbvfcnkw/cdqmvrdbk
A /tmp/.cqftyixgc
A /tmp/.cqftyixgc/dqovfhhvo
A /tmp/.okttzaqvx
A /tmp/.okttzaqvx/zrlbbyvpz
A /tmp/.qfpdbdqmx
A /tmp/.qfpdbdqmx/yzovbbjqt
A /tmp/.wffirwcpy
A /tmp/.wffirwcpy/qggiwayjd
A /tmp/.dvtskelba
A /tmp/.dvtskelba/xviyrxodl
A /tmp/.flgkqegfy
A /tmp/.flgkqegfy/ftehcgnky
A /tmp/.kfnpfwhtn
A /tmp/.kfnpfwhtn/ccfnfescv
A /tmp/.yeecaspmz

docker-compose.yml

 db:
    image: postgres
    container_name: postgres
    environment:
      POSTGRES_DB: kan_db
      POSTGRES_USER: root
      POSTGRES_PASSWORD: secret
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

I read articles about OverlayFS driver, and problems with big overlay2 folders from others. Most people suggest to prune, delete images and containers and recreate them. It might work but doesn’t solve problem entirely.

Eventually, I deleted /var/lib/docker/overlay2/2edf../diff/tmp folder. And now my disk is free by 40 GB. But after few minutes those files started appear again. I am afraid that this might happen after some time and my disk will be full soon.

Can you please explain why those files and folders appear and how to prevent this?

bluepuma77 · July 23, 2025, 1:59pm

Are you running your Postgres container on a public server/vm with a simple password? A lot of bots are always scanning the internet and messing with services they can connect to.

I would check the container logs. The file begins with “ELF”, could be a Linux binary, maybe be crash files.

rimelek · July 23, 2025, 9:07pm

It is more likely to be a Postgres issue and the database server just happens to run in a container. The overlay filesystem is not relevant here, it is just the container’s temporary writable filesystem. So probably need someone who knows Postgres better and has some idea when it could generate temp files.

As @bluepuma77 wrote, checking the logs should give you some clues.

Topic		Replies	Views
Docker files fills up disk space Docker Hub docker	7	12579	September 28, 2020
Some way to clean up / identify contents of /var/lib/docker/overlay General	78	918316	March 7, 2025
Overlay2 folder takes up entire drive General	3	431	January 23, 2025
Files ending up in overlay2 instead of being written to volume General docker	4	30	May 27, 2025
How can I cleanup OVERLAY2 folder General docker	7	17463	April 24, 2022

Postgres Docker: Weird hidden temp folders in overlay2

Related topics