How to run kafka container using docker compose file

Photo by frank mckenna on Unsplash

This Docker Compose file sets up a Kafka broker and a Kafka UI service using the Confluent Kafka image and a Kafka UI image. Below is a breakdown of each section and its components:

---
version: '2'
services:

  broker:
    image: confluentinc/confluent-local:7.4.1
    hostname: broker
    container_name: broker
    ports:
      - "8082:8082"
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports:
      - 9999:8080
    environment:
      DYNAMIC_CONFIG_ENABLED: true
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:29092

Services

This section defines the services that will be deployed in this Docker Compose application.

1. Kafka Broker

broker:
    image: confluentinc/confluent-local:7.4.1
    hostname: broker
    container_name: broker
    ports:
      - "8082:8082"
      - "9092:9092"
      - "9101:9101"

• broker: This is the name of the service for the Kafka broker.
• image: This specifies the Docker image to use. In this case, it uses Confluent's Kafka local image version 7.4.1.
• hostname: Sets the hostname of the container to broker.
• container_name: Names the container broker.
• ports: Maps ports from the container to the host:
• 8082: This is typically used for REST proxy or other management APIs.
• 9092: This is the default port for the Kafka broker that clients will connect to.
• 9101: This is an additional port that might be used for specific functionalities like schema registry or other services.

Environment Variables

environment:
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'

• KAFKA_ADVERTISED_LISTENERS: Defines how the broker advertises itself to clients. It uses two listeners:
• PLAINTEXT://broker:29092: For internal communication within Docker (e.g., between services).
• PLAINTEXT_HOST://localhost:9092: For external communication with clients outside the Docker network (e.g., applications on your local machine).
• KAFKA_CONTROLLER_QUORUM_VOTERS: Specifies the Kafka controller quorum voters. In a typical setup, the broker would be part of a larger cluster, but this is simplified here.
• KAFKA_LISTENERS: Defines the listeners the broker will use:
• PLAINTEXT://broker:29092: The internal listener.
• CONTROLLER://broker:29093: Used for the Kafka controller, which manages broker leadership.
• PLAINTEXT_HOST://0.0.0.0:9092: Binds the listener to all interfaces for external access.

2. Kafka UI

kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports:
      - 9999:8080
    environment:
      DYNAMIC_CONFIG_ENABLED: true
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:29092

• kafka-ui: This is the name of the service for the Kafka UI.
• image: Specifies the image for Kafka UI from Provectus Labs.
• ports: Maps port 9999 on the host to port 8080 on the Kafka UI container, allowing access to the UI through http://localhost:9999.

Environment Variables

environment:
      DYNAMIC_CONFIG_ENABLED: true
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:29092

• DYNAMIC_CONFIG_ENABLED: Enables dynamic configuration for the Kafka UI.
• KAFKA_CLUSTERS_0_NAME: Sets the name of the Kafka cluster as local.
• KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: Points the UI to the Kafka broker at broker:29092 to connect and manage the Kafka cluster.

Summary

This Docker Compose file sets up a local Kafka broker with the Confluent image, along with a web UI for managing Kafka topics, producers, and consumers. The configurations are designed to allow both internal and external access to the Kafka broker and provide a user-friendly interface to interact with Kafka.

How to push messages to kafka broker

How to consume message form kafka topic

Things to consider if kafka broker crashes

Backup strategy for your Kafka messages