Userns-remap makes deleted files in one RUN command reappear

Hello,

I have a very confusing issue. In a Dockerfile, on a RUN I do an rm -r of files present in the parent image.
When userns-remap is not enabled, this works fine.
When userns-remap is enabled, the removed files reappear after the RUN is finished.

Here is a minimal dockerfile to reproduce the issue:

FROM tomcat:8.0.53-jre8

RUN set -eux; \
    ls -lrt ${CATALINA_HOME}/webapps/*; \ 
    rm -rf ${CATALINA_HOME}/webapps/*; \
    ls -lrt ${CATALINA_HOME}/webapps/; 

RUN ls -lrt ${CATALINA_HOME}/webapps/; 

Without the userns-remap, first ls show the files, second and third do not.
With userns-remap, first ls show the files second does not, but third does show the files as well. And if starting a container with the resulting image, files are there as well.

I saw in the documentation that userns-remap is not compatible with all storage options but with no further information. I’m using overlay2 on a RHEL with docker 1.13.1

Anyone had that issue or any idea how to fix it?

Thanks.

Update,
We did a test with devicemapper and the behavior is normal.
So the issues seems to be from a combination of userns-remap and overlay2