Docker container can't startup with a 166G(nfs/efs) volume

General
1

I have a folder with 166G files (470k+ files nfs/efs) in the host machine (AWS ec2 ubuntu 18), when I volume to the docker, the docker container is stuck and can’t startup, if we try another 2G folder, it can startup.
Does someone know why? Is there a limitation for the volume?
my docker version is 20.10.7.

2

I do have bind-mounts on block devices in the TB range with way more files + I have remote machines that use nfs exports from the machine that expose these block devices as named volume backed as nfs. Works without issues.

By volume you mean:

  • bind-mount?
  • named volume backed by bind-mount?
  • named volume backed by nfs/efs?
  • named volume backed by ebs (using a plugin)?

Are you sure it’s not a permission problem? What is the exact error message?

4

I formatted the compose file content in your lat post by wrapping it in a </> block.
After I saved the change, the post disappeared… at least I can’t see it anymore.
Sorry if it realy become deleted.

5

Yes, that’s weird, my reply disappeared, so let me post it again, thank you @meyay . I use the EFS mounted to the EC2 with amazon-efs-utils tool, and then volume to the docker, below is my docker-compose file

services:      
    app:
        image: xxxx
        restart: always
        environment:
            - ASPNETCORE_ENVIRONMENT=Production
        ports:
            - "8002:8002"
        volumes:
            - "./BinaryObjects/Static:/app/App_Data/BinaryObjects/Static"
            - "./BinaryObjects/Temp:/app/App_Data/BinaryObjects/Temp"

the /BinaryObjects is the root which EFS mounted to, and /Static folder has the 166G files, and /Temp folder has 10M files, if I remove the /Static in volumes, and the docker container can startup. so I think it is not the permission issue. I did the same thing in our staging env, and it works well. I have tried that created the nfs volumes first,

docker volume create \
    --driver local \
    --opt type=nfs \
    --opt o=addr=xxxx.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport \
    --opt device=:/Static 
	efs-static

docker volume create \
    --driver local \
    --opt type=nfs \
    --opt o=addr=xxxx.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport \
    --opt device=:/Temp 
	efs-temp

and no luck, the docker container can’t startup neither, and works if I remove the efs-static volume, it is just stuck, and no error message, please refer to below screenshots.
image

Can you please share how you mount your TB files to docker? Maybe I can take a try. thank you so much.

6

Update: after I upgraded the OS from ubuntu 18.04 to ubuntu 20.04, this issue has gone. the docker can startup in 6 minutes. but it is still longer than before. in the past, the docker can startup in 1 minute.