Running ELK stack on Docker - Question about correct setup

General
1

Hello everyone!,

This is my first post here and I have a question about setting up an ELK stack for server/application logging on Docker.

What is the correct way to setup ELK with Docker? I’ve read there’s two ways to do it.

  1. Pull an image such sebp/elk and run all three within one container.
  2. Create three containers and run each service in its own container.

Which is the recommended way?

I’m planning to pull in logs from around 80 instances and a few rails applications. A total of around 20GB of log data a day. I’ve read that option two is better for a production level setup which is what I’m working with.

Thanks in advance for the help!

2

Run one service per container, three containers total.

So there are some interesting scale issues there. In particular, the scaling properties of the three parts of this stack are different: you might want multiple Elasticsearch containers for storage capacity, redundancy, and performance; Logstash containers only depend on the data ingest rate; and Kibana containers strictly for interactive queries. I could easily see wanting three ES containers, on separate systems, with 1 TB attached storage each, but not needing more than one Logstash or Kibana container.

Operationally it’s also just easier if you have only one thing going at a time: if you need to restart, say, Logstash, it’s a little unfortunate if your docker stop; docker rm sequence takes an Elasticsearch node with it.

(As a custom image author, it’s also far easier if your image does exactly one thing than to have to build a complicated system around multiple build steps and adding some sort of process supervisor to it, though in this case you’re not wrangling with that.)

1 Like
3

Thank you for the reply! It does make sense to run the services in separate containers and from what I’ve learned that’s the standard when using Docker.

Scalability is definitely a concern so thank you for the suggestions. I’ll definitely place three ES containers on separate instances with 1TB attached storage and see if that can keep up. I would agree that Logstash and Kibana should be okay on the same instance in their own container.

Is there an easy way to get these containers to talk to each other on separate systems?

4

Elastic Co. have official images for all the different parts of the stack, at https://www.docker.elastic.co/. Each service is on its own image. You should use those.

Ideally, you should set up a docker swarm cluster. This will take care of the fiddly parts of deploying containers on separate systems and having them talk to each other.

I use the following docker-compose file to set up a three-container stack (two elasticsearch, one kibana). This can be tweaked to ensure that the elasticsearch containers run on distinct nodes, and a logstash container can be added.

version: '3.0'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
	soft: -1
	hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet

  elasticsearch2:
    depends_on: 
      - elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.3
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
	soft: -1
	hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

  kibana:
    depends_on: 
      - elasticsearch
    image: docker.elastic.co/kibana/kibana-oss:6.2.3
    container_name: kibana
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
    ports:
      - 5601:5601
    networks:
      - esnet

volumes:
  esdata1:
  esdata2:

networks:
  esnet:
5

I would suggest NOT going 100% container approach. Keep Elasticsearch in its own VM it makes it easier to port to AWS which provides a hosted Elastic Search service as it takes up so much IO and resources it does not make sense to run it in a containerized environment.,

However, Kibana can be placed in a container since it is relatively small and can be deployed easily to a cluster. Logstash can also be in a container as well.