Best way to deploy monitoring containers, missing deploy strategy "EVERY_NODE"

Hi,

monitoring and logging containers might need to run on each node. I made some shell scripts using docker-machine to deploy Sematext Agent to each node - because there is no feature like you have in Tutum Cloud with the flag “deploy_strategy: EVERY_NODE”. In addition I wondered that docker-machine did not activate the Unix Socket on each machine, so I had to pass DOCKER_CERT_PATH and mount the path, to enable API access to the Docker Daemon on each host. Do you see security issues, mounting the certificates into a container?

I would appreciate if you could advise about the best way to deploy monitoring/logging agents, which need API access to each node.

Here are my scripts:
Create a Swarm cluster and enable Unix-Socket.

export SWARM_TOKEN=$(docker run swarm create)
docker-machine create \
    	-d virtualbox \
    	--engine-env 'DOCKER_OPTS="-H unix:///var/run/docker.sock"' \
    	--swarm \
    	--swarm-master \
    	--swarm-discovery token://$SWARM_TOKEN \
    	swarm-master

docker-machine create \
	--engine-env 'DOCKER_OPTS="-H unix:///var/run/docker.sock"' \
	-d virtualbox \
	--swarm \
	--swarm-discovery token://$SWARM_TOKEN  \
	swarm-agent-0

And two version of deployment scripts, one with TLS, one with Unix Socket:
Unix-Socket (simpler, does not need to deal with TLS):

export SPM_TOKEN=fe31fc3a-4660-47c6-XXXX-XXXXXXXX
export LOGSENE_TOKEN=3b549a2c-653a-4832-XXXX-XXXXXXXX
export NODES=$(docker-machine ls | grep Running | grep -v error | awk '{printf $1 "\t"}')
echo Swarm nodes: "$NODES"
for node in $NODES; do
  echo "get docker-machine env $node"
  eval "$(docker-machine env $node)"
  echo "Using docker host: $DOCKER_HOST"
  echo "deploy sematext-agent-docker to $node"
  docker rm -f sematext-agent 
  docker pull sematext/sematext-agent-docker > /dev/null
  docker run -d --name sematext-agent --restart=always \
  -e SPM_TOKEN \
  -e LOGSENE_TOKEN \
  -e HOSTNAME=$node \
  -v /var/run/docker.sock:/var/run/docker.sock \
  sematext/sematext-agent-docker
  sleep 2
  docker logs sematext-agent
done

TLS Socket:

export SPM_TOKEN=fe31fc3a-4660-47c6-XXXX-XXXXXXXX
export LOGSENE_TOKEN=3b549a2c-653a-4832-XXXX-XXXXXXXX
export NODES=$(docker-machine ls | grep Running | grep -v error | awk '{printf $1 "\t"}')
echo Swarm nodes: "$NODES"
for node in $NODES; do
  echo "get docker-machine env $node"
  eval "$(docker-machine env $node)"
  echo "Using docker host: $DOCKER_HOST"
  echo "deploy sematext-agent-docker to $node"
  docker rm -f sematext-agent 2>1 /dev/null
  docker pull sematext/sematext-agent-docker > /dev/null
  docker run -d --name sematext-agent --restart=always \
  -e SPM_TOKEN \
  -e LOGSENE_TOKEN \
  -e HOSTNAME=$node \
  -e DOCKER_TLS_VERIFY -e DOCKER_CERT_PATH -e DOCKER_HOST \
  -v $DOCKER_CERT_PATH:$DOCKER_CERT_PATH sematext/sematext-agent-docker
  sleep 2
  docker logs sematext-agent | grep -e "error|fail|exception"
done

Stefan Thies, @seti321

1 Like

I wrote a blog post about monitoring for Docker Swarm: https://sematext.com/blog/2016/01/12/docker-swarm-collecting-metrics-events-logs/

Final update global swarm mode solves the original issue https://sematext.com/blog/2016/09/19/docker-swarm-mode-full-cluster-monitoring-logging-with-1-command/