I am having quite strange issue. I have docker image, its setup to be sftp server. I setup jails for each user and then use syslog-ng to use unix-stream in syslog-ng to that file. When I did restart syslog-ng somehow old file did show up in my folder with file attributes where I as root I was unable to delete that file, had to issue chattr -i on that file so that I could remove it and then syslog-ng could create unix-stream on it again.
Here are all the details for my problem.
I am running docker:
Server Version: 17.12.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
this is what I did notice when syslog-ng failed to restart that there was issue with /d/ftp/user01/dev/log
my syslog-ng config
/etc/syslog-ng/conf.d/sftp_chroot.conf
source s_src2 {
unix-stream("/d/ftp/user01/dev/log");
};
log { source(s_src2); filter(f_auth); filter(f_sshd_zabbix); destination(d_auth); };
here is what I did do to correct that issue
root@server1a:~# ls -la /d/ftp/user01/dev/log
-rw-r–r-- 1 root root 0 Jun 23 05:00 /d/ftp/user01/dev/log
root@server1a:~# rm /d/ftp/user01/dev/log
rm: cannot remove ‘/d/ftp/user01/dev/log’: Operation not permitted
root@server1a:~#
root@server1a:~# chattr -i /d/ftp/user01/dev/log
root@server1a:~# rm /d/ftp/user01/dev/log
root@server1a:~# ls -la /d/ftp/user01/dev/
total 4
drwxr-xr-x 2 root root 4096 Nov 28 13:13 .
drwxr-xr-x 4 root root 0 Jun 21 04:40 …
root@server1a:~# systemctl restart syslog-ng
[ ok ] Restarting syslog-ng (via systemctl): syslog-ng.service.
root@server1a:~# ls -la /d/ftp/user01/dev/log
srw-rw-rw- 1 root root 0 Nov 28 13:13 /d/ftp/user01/dev/log
root@server1a:~#
but then problem started again when new user was added to the sftp. But it only did happen for the user01, the other user or users were fine, no issue there.
Then I was looking and I did notice something weird, that file was created in June 23 at 5:00, so I did go to my server1 where it hosts my docker images, and I did run find on all my overlay2 folders, and I did notice that there was some file created on that day as a backup file.
root@server1:~# find /var/lib/docker/overlay2/ -ls | grep user01 | grep log
11280643 0 srw-r–r-- 1 root root 0 Jun 21 00:29 /var/lib/docker/overlay2/4e3be84b3561945b314403bc176fa30f615e3dbf9ec11ba6b474475274fce1db/merged/d/ftp.bak/user01/dev/log
12059545 0 -rw-r–r-- 1 root root 0 Jun 23 05:00 /var/lib/docker/overlay2/4e3be84b3561945b314403bc176fa30f615e3dbf9ec11ba6b474475274fce1db/merged/d/backup20180711/user01/dev/log
11283698 0 srw-rw-rw- 1 root root 0 Nov 28 13:17 /var/lib/docker/overlay2/4e3be84b3561945b314403bc176fa30f615e3dbf9ec11ba6b474475274fce1db/merged/local/ftp/user01/dev/log
11280643 0 srw-r–r-- 1 root root 0 Jun 21 00:29 /var/lib/docker/overlay2/4e3be84b3561945b314403bc176fa30f615e3dbf9ec11ba6b474475274fce1db/diff/d/ftp.bak/user01/dev/log
12059545 0 -rw-r–r-- 1 root root 0 Jun 23 05:00 /var/lib/docker/overlay2/4e3be84b3561945b314403bc176fa30f615e3dbf9ec11ba6b474475274fce1db/diff/d/backup20180711/user01/dev/log
11283698 0 srw-rw-rw- 1 root root 0 Nov 28 13:17 /var/lib/docker/overlay2/4e3be84b3561945b314403bc176fa30f615e3dbf9ec11ba6b474475274fce1db/diff/local/ftp/user01/dev/log
root@server1:~#
I did not find old file in the proper location which which would indicate that file is exactly what is causing my issue, but the date and time is exactly the same, I am suspecting that original docker image was created at that time, then later it was renamed or copied to ftp.bak, and somehow that file does resurface since overlay2 file system uses differential folders.
How do I prevent that old file to re-surface when my syslog-ng restarts? How can I figure out what is wrong with my image? Maybe my system is not configured correctly, please help.