Good day to you all, Let me preface this by saying I am very new to docker and am working on building out elasticsearch for logging data. I have been able to successfully build a cluster, and start kibana with all of my nodes listed. the problem I run into Is getting the container settings to persist after a container restart.
What ends up happening is the securoty auto configuration skips because the container has already been started once with the enrollment token. Once it is enrolled in the cluster it works as expected. I know what settigns change in the elasticsearch.yml file, but I’m not sure how to make that data persist beyond restarting the container. I have created the containers with volumes to keep the data persistent. Is this another stuation here where I have to persist the yml file for settings for all my containers using a volume?
Is there a different way of doing this?
Been pouring over the documentation for both elasticsearch and docker but seem to be missing something between the two.
I added the guide i have been following, when i get to the add more nodes section and use this command: docker run -e ENROLLMENT_TOKEN="<token>" --name es02 --net elastic -it -m 1GB docker.elastic.co/elasticsearch/elasticsearch:8.11.0
the node starts and security configuration is successful. if i however ctrl +c out and try to restart the container, node start up fails. This part I understand as I have run into it in the past on a bare metal install.
ERROR: Skipping security auto configuration because it appears that the node is not starting up for the first time. The node might already be part of a cluster and this auto setup utility is designed to configure Security for new clusters only.
Exit code is 80 here. My assumption is between starting the up the second container, and restarting the settings are getting blown out of it. I’m not sure how to ensure this does not happen.
I’m guessing, because I did not spend much time on interpreting it, but it would be logical to assume that the enrollment token is necessary only when you want to add a new node. When you just restart it, you should just use the existing data without setting the variable.
In futher troubleshooting, I discovered that the cluster needs volumes to persist data beyond the first restart. Again I’m new to docker, This works great for my first node I can customize the .yml files and reboot the container and everything is persistent, but the second node still is having an issue after enrolling. Keeps telling me that it has already been enrolled in a cluster (true), but after enrollment the settings shoudl be persistent after enrollment, but they do not apper to be. This is a second container. I’ve been using this guide Install Elasticsearch with Docker | Elasticsearch Guide [8.11] | Elastic to work through building out the stack.
You need a bind mount (folder on host) or mount (Docker volume) on each node.
When using multiple nodes in a swarm, be aware that swarm will not move volumes to different nodes, so you need to manage that by yourself. If an instance is spun up on a different node, it might not have any configuration.
Also, make sure that you don’t accidentally delete the volume when re-creating a container.