Our applications use docker-compose for our internal environments for testing, and we’re investigating the use of docker engine swarm (d-e-s)for our production environments.
I’ve started looking at health checks this week, and implemented simple scripts for each of our container types to check the health status. With docker-compose this all worked well, but failed miserably with docker engine swarm. The difference I’ve found is when I add health checks to our containers within d-e-s they aren’t discoverable until they pass a health checks but with docker-compose they are always discoverable, health passed, unhealthy or “health check: starting” status.
We use mongoDB as a database/document store. Within the d-e-s environment we use a 3 way replica-set for resilience and load balancing, but I can also replicate this environment internally using docker-compose. When our mongoDB container starts a fresh it works out if it’s a master in a replica-set, creates the new replica-set, creates users, and loads a load of seeding data. If the container works out it’s a slave, it waits until the replica-set set is running, discovers the master and then add itself to the replica-set through the master. The master works fine in a d-e-s environment with health checks enabled; the trouble we’re having is with the slaves. They can discover the master, once the master has started, created the replica-set and passed a health check but they fail to add themselves to the replica-set because the master can’t discover them, because the slave has yet to pass a health check.
Our health check script for our mongoDB instances just pokes the database with a simple server status command to see if the database is responding:
/opt/mongodb/bin/mongo admin --port $MONGO_PORT -u admin -p $MONGO_ADMIN_PASS --ssl \
--sslAllowInvalidCertificates --eval "db.serverStatus()" > /dev/null || exit 1We could make this a bit more clever in the future (i.e. check the replica-set status if it’s running in a cluster, to make sure the replica-set is OK), but this is a start. The problem is that this command fails on the slave until it’s been added to the replica-set and the slave has synced with the master. So it’s catch-22, the slave can’t add itself to the replica-set because it hasn’t passed a health check, and it can’t pass it’s health check because it’s not part of a replica-set.
One way is for the slave to write a flag somewhere once it’s added itself to a replica-set, and the health check script passes (for a limited time) until that flag is set, and then checks against mongoDB. But I can see all sorts of issues with files/flags left hanging about, as well as the point that the health check scripts passes even though the slave isn’t really healthy.
[We also use openLDAP in a active master-master setup, and I suspect it will have similar issues]
Any suggestions on a better solution would be appreciated.