Containers not working on all nodes

Docker Engine Swarm
1

I deployed a container on two nodes in a swarm (via docker-compose.yml). On one node, the container functions as normal. On another node, the container doesn’t.

I checked the Docker and Docker Compose versions on both systems and they are the same.

Is there something else that could cause this situation? Is there something I’m missing?

2

It’s difficult to suggest the fix without looking at the compose file. Can you share what’s in the compose file.
Also, provide details about the node(is it bare metal, VM or a Cloud instance).

3

Sure thing.

The nodes are bare metal.

compose file:


services:

  kafka03:
    image: confluentinc/cp-kafka:7.3.0
    hostname: kafka03
    ports:
      - "39092:39092"
      - "39093:39093"
    environment:
      KAFKA_BROKER_ID: 5
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka03:9092,PLAINTEXT_HOST://<host_ip_is_here>:39093'
      KAFKA_JMX_PORT: 19101
      KAFKA_AUTO_CREATE_TOPICS_ENABLED: 'False'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_NODE_ID: 5
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_CONTROLLER_QUORUM_VOTERS: '5@kafka03:39093'
      KAFKA_LISTENERS: 'PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:39093,PLAINTEXT_HOST://0.0.0.0:39092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      #KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS: 720000
      KAFKA_SOCKET_CONNECTION_SETUP_TIMEOUT_MS: 360000
      KAFKA_INITIAL_BROKER_REGISTRATION_TIMEOUT_MS: 360000
      KAFKA_CONTROLLER_QUORUM_REQUEST_TIMEOUT_MS: 120000
      KAFKA_LOG_DIRS: '/tmp/logs'
    networks:
      - kafka_test_network
    deploy:
      replicas: 1
      placement:
        constraints: [node.labels.kafka == 2]
    volumes:
      - /home/kafka/logs:/tmp/logs
      - /home/kafka/data/update_run.sh:/tmp/update_run.sh
    command: "bash -c 'if [ ! -f /tmp/update_run.sh ]; then echo \"ERROR: Did you forget the update_run.sh file that came with this docker-compose.yml file?\" && exit 1 ; else /tmp/update_run.sh && /etc/confluent/docker/run ; fi'"


networks:
  kafka_test_network:
    driver: overlay
    attachable: true```
4

Can you share the output of the following commands:

  1. docker service ls
  2. docker service ps
  3. docker service logs -f

Also, I assume that you are running docker stack deploy on the manager node to deploy the Kafka service.

5

The systems are VMs. I verified with the admins.

7

It was an environment configuration issue. The admins fixed it. It was something dealing with getting Swarm and VMWare’s networking stacks to work together.