How to push messages to kafka broker

Photo by Joshua Mayo on Unsplash

To push messages to Kafka using Python, you can use the kafka-python library. Here's a simple example to demonstrate how to produce messages to a Kafka topic

Step 1: Install kafka-python

You need to install the kafka-python library if you haven't already. You can do this using pip:

pip install kafka-python

Step 2: Producer Code

Here’s a basic Python script to send messages to a Kafka topic:py

from kafka import KafkaProducer
import json
import time

# Define the Kafka topic and broker address
KAFKA_BROKER = 'localhost:9092'  # Adjust this if your broker is on a different host
TOPIC_NAME = 'my-topic'
# Create a Kafka producer
producer = KafkaProducer(
    bootstrap_servers=[KAFKA_BROKER],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')  # Serialize messages to JSON format
)
# Send messages to the Kafka topic
try:
    for i in range(10):  # Send 10 messages
        message = {'number': i}
        producer.send(TOPIC_NAME, value=message)
        print(f"Sent: {message}")
        time.sleep(1)  # Sleep for a second before sending the next message
finally:
    # Close the producer
    producer.close()

Explanation:

• KafkaProducer: This class is used to create a Kafka producer instance. You provide it with the Kafka broker’s address and the serialization method for the messages.
• bootstrap_servers: This parameter specifies the Kafka broker address. Change it if your broker is running on a different host or port.
• value_serializer: This function is used to convert the message value into bytes before sending it to Kafka. Here, it serializes Python dictionaries to JSON format.
• send(): This method sends the message to the specified topic.
• close(): This method closes the producer to free up resources.

Step 3: Running the Script

Make sure your Kafka broker is running and the specified topic exists. You can create the topic manually or let Kafka create it automatically when you send the first message.

Run the script, and you should see messages being sent to the specified Kafka topic. If you have a Kafka consumer running, it will be able to receive these messages.