This is a quick guide to set up Kafka environment for local development and learn Kafka. I use Kafka Udemy course and documentation for this.
Setup and configuring Kafka echo system may be a boring task. However with Docker and Landoop (now they are Lenses) it is as easy as running a docker command.
Note: You need Docker installed to follow this post.
Get landoop docker image
When you have setup docker in your environment you can pull landoop docker image.
$ docker pull landoop/fast-data-dev
Start the Kafka broker
I'm going to run the docker container in interactive mode.
$ docker container run --rm --name my-kafka-broker -it \ -p 2181:2181 -p 3030:3030 -p 8081:8081 \ -p 8082:8082 -p 8083:8083 -p 9092:9092 \ -e ADV_HOST=127.0.0.1 \ landoop/fast-data-dev
This will bring up all necessary tools to work with Kafka. After about 1 minute you can access landoop's UI console from http://127.0.0.1:3030/.
If you scroll down, you can see running services with their ports.
Using the command line
We can connect to our running docker container to access command line tools. It is a handy way to get started with basic Kafka APIs. What i'm going to do is create a topic, produce few messages and consume messages. All from command line.
Access Kafka docker container
We can use docker exec to access our Kafka docker container.
docker exec -it my-kafka-broker bash
Now you are inside the container. Type kafka- and hit tab. You can see all command line interfaces to work with Kafka.
Create a topic
First, we need a topic to publish our messages. Type kafka-topics and hit enter. It will list all available options specially mentioning required field. However it is little confusing. Use below command to create our first topic name first_topic.
$ kafka-topics --bootstrap-server 127.0.0.1:9092 --create --topic first_topic --partitions 1 --replication-factor 1
Use --list to list down all available topics. You should be able see our newly created topic with it.
$ kafka-topics --bootstrap-server 127.0.0.1:9092 --list
Publishing a message with CLI is very easy. Copy below command into your terminal and hit enter. Then it will give you the option to publish your message. Type a message and hit enter. Type as many messages as you want.
$ kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic
Just as publishing messages, consuming messages also very simple. Consumer will immediately start to listen to your messages. However it will not listen to previous messages. If you want to listen to previous messages you need to specify it using --from-beginning option.
$ kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning
I hope now you have a running Kafka setup in your local environment. Enjoy learning.