Deploying a container (using Kafka) on VM (AWS / GCP)

General
1

I am trying to deploy a docker-compose.yaml on a public cloud VM. There is no other file I would like to deploy

My file that contains 3 images to build the container (see code below): Kafka Broker, Zookeeper and Python-Producer (which basically just produces random numbers periodically and writes to a Kafka Topic; and this image is deployed publicly).

I have followed 2 comprehensive tutorials on how to deploy a container on VM using GCP & AWS EC2 meticulously. But I am getting the following errors

  • On AWS EC2 on Linux → I am unable to install docker-compose engine
  • On GCP VM on Ubuntu → I am able to run docker-compose up command but it throws an error stating the broker cannot be started

Link to AWS Deploy tutorial: https://www.youtube.com/watch?v=gRgdnHHuvoI&t=142s

Link to GCP Deploy Tutorial: https://www.youtube.com/watch?v=nt7fpz4JXzY

version: '2'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:latest
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

  producer:
    image: daftpunkapi/flink_v1:latest
    depends_on:
      - broker
    restart: "on-failure"

Any help and guidance will be much appreciated - TIA!

2

I assume with “AWS EC2 on Linux” you mean Amazon Linux 2?

Can you explain what you mean by docker-compose engine?
There is either the Docker Engine or Docker Compose.

You can install Docker Compose on every Linux host like this:
https://docs.docker.com/compose/install/linux/#install-the-plugin-manually

You could also have created the EC2 instance using an Ubuntu AMI. You don’t have to stick Amazon Linux.

Regarding gcp: please start your compose project in the foreground (without -d) and share the error messages you receive.