Not sure if this is the right forum for this, but we’ve been running docker in production for a while and realized that our default loop-back storage engine was still in use on our Centos machines. After trying a few options we got the best speed of spinning up new instances using the overlay storage engine so we switched to that.
We’re running probably ~30-45 containers at any given time on each machine processing though various queues. Around 3-4 machines get cycled per minute, and another ad hoc process probably create about 5-10 more, so we probably get around 10-15 machines being created a minute. We periodically get this error: “Error response from daemon: mkdir /var/lib/docker/overlay/3851fa644830e4f2aefa2d789cc43b75be43f8869b41cb003472ddbae948459a-init/merged/dev/pts: cannot allocate memory”
We have a swap set up, and in general are well under both our physical memory limit and ram limit, and the hard drive also has considerable space open.
I realize this was reported a few months ago but was there any resolution to this Jason? Did you figure out what was causing the problem and how to fix it?
Sorry for the late reply. We weren’t able to get a really satisfactory answer to this. We ended up switching our architecture slightly by moving to Amazon ECS, and those are running on boxes using amazon linux ami and we haven’t run into the same issue there with the overlay file system.
I am also facing the same issue sporadically, we are using EC2 instances only.
Error response from daemon: Error response from daemon: open /var/lib/docker/overlay/4c2d73e20c3de1a69ec11d06cbb115e42bf5ca926f1f21a4f6593b75f06ed139-init/merged/dev/console: cannot allocate memory