Shared file from host to container

Hi everyone,

I have an elastic project running on a swarm cluster and I’m trying to share a file with a container so that I can update the file on the host and the change is automatically reflected on the container. The problem is every time I update the file on the host I need to remove the stack and the config file and deploy everything again.

I’m using compose version 3 to deploy everything more easily.

This is an example of my configuration for that file.

version: ‘3.3’

  • type: bind
    source: monitors_config
    target: /usr/share/heartbeat/monitors.d/monitor.icmp.yml

configs:
monitors_config:
file: ./heartbeat/monitors.d/monitor.icmp.yml

Can anyone help me find a way to do this?

Tank you,

This is normal behavior. A config name is distributed amongst the nodes during stack deployment. To archive a distribution of a changed config, you need to add a suffix to the config name (like the .v1 I added). Whenever you make a change to the file, you have to modify the suffix as well. As a result the “new” config will be distribution and applied to the container. On a stack delete, you will notice that all versions changed during uptime will be listed as delted as well…

version: '3.3'
    ...
    configs: 
      - source: monitors_config.v1
        target:  /usr/share/heartbeat/monitors.d/monitor.icmp.yml
    ...
configs:
  monitors_config.v1:
    file: ./heartbeat/monitors.d/monitor.icmp.yml

HI Meyay,

Thank you for responding to my question so quickly.

The way you show me in the previous post, i still need to remove and deploy the stack or I just need to update the elastic_heartbeat service ?

Best regards,
Luis Pereira

redeploy without removing it first.

I deployed the stack without removing it first, but now I have an error saying that "“No such image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0” in all containers.

Seems like you changed more than just the handle on the config file and its content.
Changing a config file handle is not related to not beeing able to fetch an image.

Since there is no before/after post of your docker-compose.yml, it is hard to guess what’s wrong.

I assume you use a recent version of the docker engine…

This is my docker compose file at this moment…
In order to be able to post the image field is preformatted.
version: ‘3.3’

services:

elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
environment:
ELASTIC_PASSWORD: ******
discovery.type: single-node
ES_JAVA_OPTS: “-Xmx256m -Xms256m”
configs:
- type: bind
source: elastic_config
target: /usr/share/elasticsearch/config/elasticsearch.yml
ports:
- “9200:9200”
- “9300:9300”
networks:
- elknet

kibana:
image: docker.elastic.co/kibana/kibana:7.6.0
ports:
- “5601:5601”
configs:
- type: bind
source: kibana_config
target: /usr/share/kibana/config/kibana.yml
networks:
- elknet

logstash:
image: docker.elastic.co/logstash/logstash:7.6.0
ports:
- “5000:5000”
- “9600:9600”
configs:
- type: bind
source: logstash_config
target: /usr/share/logstash/config/logstash.yml
- type: bind
source: logstash_pipeline
target: /usr/share/logstash/pipeline/logstash.conf
environment:
LS_JAVA_OPTS: -Xmx256m -Xms256m
networks:
- elknet

heartbeat:
image: docker.elastic.co/beats/heartbeat:7.6.0
configs:
- type: volume
source: heartbeat_config
target: /usr/share/heartbeat/heartbeat.yml
read_only: true
- type: bind
source: monitors_config.v2
target: /usr/share/heartbeat/monitors.d/monitor.icmp.yml
environment:
- output.elasticsearch.hosts=[“elasticsearch:9200”]
networks:
- elknet

filebeat:
image: docker.elastic.co/beats/filebeat:7.6.0
user: root
ports:
- “9002:9002”
configs:
- type: bind
source: filebeat_config
target: /usr/share/filebeat/filebeat.yml
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /var/lib/docker/containers/:/var/lib/docker/containers/:ro
command: ["–strict.perms=false"]
environment:
- ELASTICSEARCH_USERNAME={ELASTICSEARCH_USERNAME:******} - ELASTICSEARCH_PASSWORD={ELASTICSEARCH_PASSWORD:******}
networks:
- elknet

metricbeat:
image: docker.elastic.co/beats/metricbeat:7.6.0
user: root
configs:
- type: bind
source: metricbeat_config
target: /usr/share/metricbeat/metricbeat.yml
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
- /proc:/hostfs/proc:ro
- /:/hostfs:ro
networks:
- elknet

configs:
elastic_config:
file: ./elasticsearch/config/elasticsearch.yml
logstash_config:
file: ./logstash/config/logstash.yml
logstash_pipeline:
file: ./logstash/pipeline/logstash.conf
kibana_config:
file: ./kibana/config/kibana.yml
heartbeat_config:
file: ./heartbeat/config/heartbeat.yml
filebeat_config:
file: ./filebeat/config/filebeat.yml
metricbeat_config:
file: ./metricbeat/config/metricbeat.yml
monitors_config.v2:
file: ./heartbeat/monitors.d/monitor.icmp.yml

networks:
elknet:
driver: overlay

The only thing that I’ve change was the suffix to version 2
Im going to remove the stack redeploy everything again .
Then I’m going to add a new host to the monitor.icmp.yml file and changed the version to V3.

I’am using this command to update the stack docker stack deploy - c docker-compose.yml elastic

And hove everything works .

The docker-compose.yml is hard to read like this. You need to add a blank line before the preformated text block to actualy apply the formating.

I used this pattern a lot in the past and never had problems. What happens if you execute docker pull docker.elastic.co/elasticsearch/elasticsearch:7.6.0?

I assume your added a space between the dash and the c in “-c” by accident.

version: ‘3.3’

services:

elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
environment:
ELASTIC_PASSWORD: ******
discovery.type: single-node
ES_JAVA_OPTS: “-Xmx256m -Xms256m”
configs:

  • type: bind
    source: elastic_config
    target: /usr/share/elasticsearch/config/elasticsearch.yml
    ports:
  • “9200:9200”
  • “9300:9300”
    networks:
  • elknet

kibana:
image: docker.elastic.co/kibana/kibana:7.6.0
ports:

  • “5601:5601”
    configs:
  • type: bind
    source: kibana_config
    target: /usr/share/kibana/config/kibana.yml
    networks:
  • elknet

logstash:
image: docker.elastic.co/logstash/logstash:7.6.0
ports:

  • “5000:5000”
  • “9600:9600”
    configs:
  • type: bind
    source: logstash_config
    target: /usr/share/logstash/config/logstash.yml
  • type: bind
    source: logstash_pipeline
    target: /usr/share/logstash/pipeline/logstash.conf
    environment:
    LS_JAVA_OPTS: -Xmx256m -Xms256m
    networks:
  • elknet

heartbeat:
image: docker.elastic.co/beats/heartbeat:7.6.0
configs:

  • type: volume
    source: heartbeat_config
    target: /usr/share/heartbeat/heartbeat.yml
    read_only: true
  • type: bind
    source: monitors_config.v2
    target: /usr/share/heartbeat/monitors.d/monitor.icmp.yml
    environment:
  • output.elasticsearch.hosts=[“elasticsearch:9200”]
    networks:
  • elknet

filebeat:
image: docker.elastic.co/beats/filebeat:7.6.0
user: root
ports:

  • “9002:9002”
    configs:
  • type: bind
    source: filebeat_config
    target: /usr/share/filebeat/filebeat.yml
    volumes:
  • /var/run/docker.sock:/var/run/docker.sock:ro
  • /var/lib/docker/containers/:/var/lib/docker/containers/:ro
    command: ["–strict.perms=false"]
    environment:
  • ELASTICSEARCH_USERNAME={ELASTICSEARCH_USERNAME:******}
  • ELASTICSEARCH_PASSWORD={ELASTICSEARCH_PASSWORD:******}
    networks:
  • elknet

metricbeat:
image: docker.elastic.co/beats/metricbeat:7.6.0
user: root
configs:

  • type: bind
    source: metricbeat_config
    target: /usr/share/metricbeat/metricbeat.yml
    volumes:
  • /var/run/docker.sock:/var/run/docker.sock:ro
  • /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
  • /proc:/hostfs/proc:ro
  • /:/hostfs:ro
    networks:
  • elknet

configs:
elastic_config:
file: ./elasticsearch/config/elasticsearch.yml
logstash_config:
file: ./logstash/config/logstash.yml
logstash_pipeline:
file: ./logstash/pipeline/logstash.conf
kibana_config:
file: ./kibana/config/kibana.yml
heartbeat_config:
file: ./heartbeat/config/heartbeat.yml
filebeat_config:
file: ./filebeat/config/filebeat.yml
metricbeat_config:
file: ./metricbeat/config/metricbeat.yml
monitors_config.v2:
file: ./heartbeat/monitors.d/monitor.icmp.yml

networks:
elknet:
driver: overlay

Yes the - c was a typo.

The indention is still broken on your pasted compose.yml.
What is the result of the docker pull? If it works, the stack redployment/update should work too

juts for the sake of demonstration… your preformated text should look like this:

version: '3.3'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
    environment:
      ELASTIC_PASSWORD: ******
      discovery.type: single-node
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
    configs:
      type: bind
      source: elastic_config
      target: /usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      elknet:
...

Paste your compose.yml, then mark the text, then click on “preformated text”.

It’s given me an error about symlink broken.

The worker node only has 4Gb free of disk space I’m going to do some cleaning.

version: '3.3'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
    environment:
       ELASTIC_PASSWORD: changeme
       discovery.type: single-node
       ES_JAVA_OPTS: "-Xmx256m -Xms256m"
    configs:
      - type: bind
        source: elastic_config
        target: /usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - elknet


  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.0
    ports:
      - "5601:5601"
    configs:
      - type: bind
        source: kibana_config
        target: /usr/share/kibana/config/kibana.yml
    networks:
      - elknet

  logstash:
    image: docker.elastic.co/logstash/logstash:7.6.0
    ports:
      - "5000:5000"
      - "9600:9600"
    configs:
      - type: bind
        source: logstash_config
        target: /usr/share/logstash/config/logstash.yml
      - type: bind
        source: logstash_pipeline
        target: /usr/share/logstash/pipeline/logstash.conf
    environment:
      LS_JAVA_OPTS: -Xmx256m -Xms256m
    networks:
      - elknet


  heartbeat:
    image: docker.elastic.co/beats/heartbeat:7.6.0
    configs:
      - type: volume
        source: heartbeat_config
        target: /usr/share/heartbeat/heartbeat.yml
        read_only: true
      - type: bind
        source: monitors_config.v2
        target: /usr/share/heartbeat/monitors.d/monitor.icmp.yml
    environment:
      - output.elasticsearch.hosts=["elasticsearch:9200"]
    networks:
      - elknet

  filebeat:
    image: docker.elastic.co/beats/filebeat:7.6.0
    user: root
    ports:
      - "9002:9002"
    configs:
      - type: bind
        source: filebeat_config
        target: /usr/share/filebeat/filebeat.yml
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/containers/:/var/lib/docker/containers/:ro
    command: ["--strict.perms=false"]
    environment:
      - ELASTICSEARCH_USERNAME=${ELASTICSEARCH_USERNAME:$elastic}
      - ELASTICSEARCH_PASSWORD=${ELASTICSEARCH_PASSWORD:$changeme}
    networks:
      - elknet

  metricbeat:
    image: docker.elastic.co/beats/metricbeat:7.6.0
    user: root
    configs:
      - type: bind
        source: metricbeat_config
        target: /usr/share/metricbeat/metricbeat.yml
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /proc:/hostfs/proc:ro
      - /:/hostfs:ro
    networks:
      - elknet

configs:
   elastic_config:
     file: ./elasticsearch/config/elasticsearch.yml
   logstash_config:
     file: ./logstash/config/logstash.yml
   logstash_pipeline:
     file: ./logstash/pipeline/logstash.conf
   kibana_config:
     file: ./kibana/config/kibana.yml
   heartbeat_config:
     file: ./heartbeat/config/heartbeat.yml
   filebeat_config:
     file: ./filebeat/config/filebeat.yml
   metricbeat_config:
     file: ./metricbeat/config/metricbeat.yml
   monitors_config.v2:
     file: ./heartbeat/monitors.d/monitor.icmp.yml


networks:
  elknet:
    driver: overlay

You compose.yml looks fine. From my pov it seems impossible anything in the pasted compose.yml should cause the image pull problem. Renaming the config handles and experience pull problems afterwards are accidentally causal at best.

Please loose the “type: bind” when you use a config with a service, There is a reason why my first example didn’t include this setting: it does not exist (see: docs)

Thank you for the input, I’ve done the correction as suggested I’m gonna try bringing the stack up again and I will update this post with the result

Oh, and neither “type: volume”, nor “read_only: true” exist for configs. You seem to mixed up the attributes with those for volumes.

I will leave you up to yourself now. The original question is answered. Though, it seems you have a different problem. Good luck!

Thanks meyay! I just got a solution from you!

Regard: CEO of Account Generator

Shared hosting file system works!
10freeaccounts