Recovering deleted symlinks from docker/overlay2/l

General
1

Hi All, I’ve mistakenly deleted the docker/overlay2/l folder but all my docker images and containers are still in tact. My understanding is this contains symlinks which allows the docker engine to match up the ids of containers/images/etc. to their actual file locations. As a result, I cannot attach or inspect any container or image, but they are still running in the background.

Is there a way to recover the symlinks? I can simply do ls in docker/overlay2 to get the list of things I need to link to, but I cant figure out how to match them to their respective image/layer id.

I can look in a specific folder and there are suspiciously relevant files e.g. docker/overlay2/abcdef1234/lower contains a bunch of information l/abc123:l/123abc:l/def456:l/162dfe.

Any guidance would be appreciated,

Thanks

2

Interesting question. Before the answer, let me note that this is a kind of mistake I think when I don’t recommend manually doing anything in the docker data root. I’m not saying it to you, but every future reader :slight_smile: But mistake can happen if someone really needs to do some advanced debugging.

I think this command would generate a list of commands to restore the links and you can decide whether you execute them or not:

find . -maxdepth 2 -name diff \
  | awk -F/ '{print $2}' \
  | xargs -I '{}' -- sh -c 'echo -n "ln -s {}/diff l/"; cat {}/link; echo'

Make sure you run it in the “overlay2” folder under the docker data root. /var/lib/docker in your case.

Why I think it could work: After your question I tried to find out where my filenames were stored and it turned out that the link names are in each layer in the file called “link”. I didn’t think it would be as easy…

Note that I haven’t tested if this would be enough or it just restores part of the folder.

Just an example how the generated commands look like on my machine:

ln -s db34ae20946287ade4dcfaf17eb9cee67085d16134bcfb7b655c955b19fa5547/diff l/B6ZSOABJLGZ6JLQPSUKY573FWW
ln -s 012915d0c5b0a8b1efa78a95814beec7eac326566e0cb97f3a5d12ab21b07bdc/diff l/24SQTVZLOPFZLZQ3CP3ABXXSZ7
ln -s 1001899746a8411a6adbc1ac5284c1c4b9cb1abeb30322009739792bc997f9c2/diff l/BPILH3ZXMMOETKH26YORXEHLWQ
ln -s ffe4c1801766249ff3085f26cd38fc511e253831cfe4e76ed516eac1b61f9ead/diff l/FQ4NX2R4PGPHPEZO34T4H2YSQE
ln -s 5734704d8558723a5448cb15b2794d56fa070094af067a69d6b78de970a75282/diff l/FOHAUECG34N3NKZCO6ZP24O4YB
ln -s a6534da901efb8101d791e88da3b3f5d4bea4d342d595b52b70b3a4d5fdf1bf6/diff l/Z6XLC3XTMZHSYCXYTFFXBUMRLU
ln -s 8d46118b8706c2cbe72ae12bcabe5f3666a41dc793d9f2d5bd4219ee921799dd/diff l/6BLINWBFANC57KZKJCWXV3YYPG
3

Thanks so much for this. I’ve learnt my lesson so I’m making a backup of the folder before I try this :rofl: but will get back to you with how it goes. I’ve modified the echo command to link to {}/diff but the output is promising :crossed_fingers:

4

I fixed my post too, thanks

5

Solution worked, thank you! Seems everything has rebuilt correctly. All images and containers fully functioning as before.