Skip to main content

Getting started with Kafka

This is a quick guide to set up Kafka environment for local development and learn Kafka. I use Kafka Udemy course and documentation for this.
Setup and configuring Kafka echo system may be a boring task. However with Docker and Landoop (now they are Lenses) it is as easy as running a docker command.

Note: You need Docker installed to follow this post.


Get landoop docker image

When you have setup docker in your environment you can pull landoop docker image.

$ docker pull landoop/fast-data-dev

Start the Kafka broker

I'm going to run the docker container in interactive mode.

$ docker container run --rm --name my-kafka-broker -it \
           -p 2181:2181 -p 3030:3030 -p 8081:8081 \
           -p 8082:8082 -p 8083:8083 -p 9092:9092 \
           -e ADV_HOST=127.0.0.1 \
           landoop/fast-data-dev
This will bring up all necessary tools to work with Kafka. After about 1 minute you can access landoop's UI console from http://127.0.0.1:3030/.


If you scroll down, you can see running services with their ports.


Using the command line

We can connect to our running docker container to access command line tools. It is a handy way to get started with basic Kafka APIs. What i'm going to do is create a topic, produce few messages and consume messages. All from command line.

Access Kafka docker container

We can use docker exec to access our Kafka docker container.

docker exec -it my-kafka-broker bash
Now you are inside the container. Type kafka- and hit tab. You can see all command line interfaces to work with Kafka.

Create a topic

First, we need a topic to publish our messages. Type kafka-topics and hit enter. It will list all available options specially mentioning required field. However it is little confusing. Use below command to create our first topic name first_topic.

$ kafka-topics --bootstrap-server 127.0.0.1:9092 --create --topic first_topic --partitions 1 --replication-factor 1

Use --list to list down all available topics. You should be able see our newly created topic with it.

$ kafka-topics --bootstrap-server 127.0.0.1:9092 --list

Publish messages

Publishing a message with CLI is very easy. Copy below command into your terminal and hit enter. Then it will give you the option to publish your message. Type a message and hit enter. Type as many messages as you want.

$ kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic

Consume messages

Just as publishing messages, consuming messages also very simple. Consumer will immediately start to listen to your messages. However it will not listen to previous messages. If you want to listen to previous messages you need to specify it using --from-beginning option.

$ kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning

I hope now you have a running Kafka setup in your local environment. Enjoy learning.

Further learning


Comments

Post a Comment