Kafka is distributed event streaming software based on publish and subscribe to model. Its very powerful streaming software used by many organizations.
More details are find in the below link
Prerequisite
Kafka is required minimum JAVA version 1.8 and above (JAVA_HOME environment variable need setup in the system)
Need 7zip software to extract .tgz files in windows.
Download and Extract
Download latest Kafka software from below location.
http://kafka.apache.org/downloads.html
Direct link is below
https://www.apache.org/dyn/closer.cgi?path=/kafka/2.8.0/kafka_2.13-2.8.0.tgz
Extract “kafka_2.13-2.8.0.tgz” in local drive.
Start Zookeeper
Kafka is required zookeeper service to manage Kafka cluster. Latest version of Kafka has embedded zookeeper. Older version of Kafka was required to install zookeeper as separate service outside of Kafka.
Open “zookeeper.properties” file in the config directory and update location for zookeeper data directory.
dataDir=C:\\kafka-workspace\\kafka_2.13-2.8.0\\zookeeper\\data |
Zookeeper uses the default port 2181. There are many zookeeper properties can update based on requirement.
Open command prompt and locates to Kafka root directory and use following command to start zookeeper service.
bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties |
Finally, zookeeper is running on localhost:2181
We can see startup logs in console.
Once zookeeper started successfully, we can see data directory with snapshot files.
Start Kafka
Locate to Kafka config directory and update “server.properties”
Update Kafka logs directory to below location or any other valid directory.
log.dirs=C:\\kafka-workspace\\kafka_2.13-2.8.0\\kafka-logs |
It also have important property “zookeeper.connect” and it uses the default value. If any changes in port or host name need to update same property.
zookeeper.connect=localhost:2181 |
Open command prompt and locate to Kafka root directory. Run following command to start Kafka broker service.
bin\windows\kafka-server-start.bat .\config\server.properties |
Once Kafka started successfully we can see “started (kafka.server.KafkaServer)” message in the logs.
Verify Installation
We create Kafka topic and produce some messages on topic. Other end we will run consumer to receive messages.
Create Kafka Topic
Open command prompt and locate to Kafka bin windows directory. Use following create topic command.
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic first-kafka-topic |
List topics
There are many other commands we can see by typing --help
Start Producer
Open command prompt and locate to Kafka bin windows directory. Use following producer command to start producer and post messages on specific topic.
kafka-console-producer.bat --broker-list localhost:9092 --topic first-kafka-topic |
Broker-list is Kafka server and which is running on 9092default port.
Start Consumer
Open command prompt and locate to Kafka bin windows directory. Use following consumer command to start consumer.
kafka-console-consumer.bat --bootstrap-serverlocalhost:9092 --topic first-kafka-topic |
Now type some messages on producer command prompt and same messages are receiving at consumer command prompt. This confirms the installation of Kafka is successful.
References
https://kafka.apache.org/quickstart
