Docker Community Forums

Share and learn in the Docker community.

Free Space (or lack there of...)


(Evan Montgomery-Recht) #1

We had a issue in our initial deployment where we didn’t allocate enough space to /var. So when the machine ran out of space about 6 hours after installing, we were unable to login to UCP first thing in the morning (kind of amusing actually). We’re not entirely sure what happened, but the web interface would not start on the ‘master’ server. Everything else seemed to start however when quering from other machine’s it didn’t appear the swarm or ucp were fully converged/synched.

In the end, due to deadline, we increased space, removed all images (including manually removing some container folders) and restarted, including rebuilding all nodes, and immediately installed Zabbix to monitor servers.

Recommend the following:

  1. Update documentation to reflect what directories will be written to/installed into.
  2. Put sometype of notification to users that space is low (assuming it’s not there already, which I assume not since it’s reporting more space is available then is in reality.)

thanks,

evan


(Clearleap) #2

Better docs about what gets written to would be great but I fee like alerting on low space is outside the scope of UCP. Lots of other tools exist for that kind of thing already.

Just my two bits.


(Vivek Saraswat) #3

Hi folks, thanks for the tip. We’ll plan to get the docs updated, and will take a look at potential for notifications on this.


(Vivek Saraswat) #4

Actually @montge it would help us a lot to a go a bit deeper into the issue. Can you provide the following info:

  • How did you size /var in the beginning (what amount of space was allocated)
  • Did you pull several images in between installation and when the issues started occurring?

(Evan Montgomery-Recht) #5

We had a bare bone’s CentOS 7 box that only had 2GB assigned to /var.
We had only pulled 1 image (happened to be jenkins which is 675MB.)

We’re using a drive partitions similar to what is used for PCI-DSS, which has separate partitions for various things.


(Vivek Saraswat) #6

Interesting. I believe we require a minimum 3 GB space, but that should have been caught by our bootstrapper. The fact that it didn’t means something may be wrong in our check. Perhaps we can talk about this further in the call.


(Evan Montgomery-Recht) #7

You know you say that and I’m realizing that when I do a docker info it thinks my machine has 100GB, but I think it’s double counting tmpfs and a few things.

Suggest you try the following kickstart file from the OpenSCAP people if you’re looking to reproduce.

https://github.com/OpenSCAP/scap-security-guide/tree/master/RHEL/7/kickstart


(Evan Montgomery-Recht) #8

Vivek,

If you’re available still I’m on 8778851087,3018257250#

Thanks,

evan


(Evan Montgomery-Recht) #10

Yes call me at either 2407086309 or 6182071894 directly.