Docker Community Forums

Share and learn in the Docker community.

EFS "There is not enough space on the disk." - but there is


(Georgejenkins) #1

Expected behavior

I have deployed a docker swarm on AWS (using the provided CloudFormation template) on which I want to run a bunch of microservices. Each microservice runs from the same image microsoft/dotnet:1.1.2-runtime and has the code to run mounted via a cloudstor EFS volume.
The command is “dotnet microservice.dll” with the appropriate assembly name.
All microservices run OK as long as there are few of them per swarm node.

Actual behavior

As soon as I try to bring more than 3-4 microservices up on the same node, they start to fail to launch. It doesn’t matter whether they are the same ones or different ones.
Dotnet gives back various errors that always amount to something like “There is not enough space on the disk.” as the root cause.

Additional Information

df in container reveals less than 1% of the mounted EFS volume is in use
df on host reveals no (local) disks are used for more than 10%
docker stats indicate there is still 90% of the available RAM memory left on the nodes.

Steps to reproduce the behavior

  1. Install Docker for AWS from https://docs.docker.com/docker-for-aws/
  2. I configured it with 3 managers / 2 workers (m3.medium machines) and Cloudstor EFS enabled, but one machine would probably also do for testing.
  3. get and build a sample dotnet core project (e.g. https://github.com/dotnet/docs/tree/master/samples/csharp/getting-started/WeatherMicroservice) onto the EFS volume called CodeLib below
  4. create a stack.yml in which a service for the above mounts the EFS volume

services:
mymicroservice:
image: microsoft/dotnet:1.1.2-runtime
volumes:
- CodeLib:/home/latest_builds
command: dotnet microservice.dll
deploy:
replicas: 10
volumes:
CodeLib:
external: true