How to deploy Kafka and Zookeeper cluster on Kubernetes

How to deploy Kafka and Zookeeper cluster on Kubernetes  Ashish Pandey
Posted on April 10, 2020, 8:14 a.m.

Kafka zookeeper Architecture on Kubernetes

This tutorial explains how to Deploy Kafka and zookeeper on Kubernetes.

Prerequisite:

Let's start

Log in to the master node. Clone this repository ..

git clone https://github.com/ashishrpandey/kafka-demo
cd kafka-demo/

Deploy zookeeper statefulset, zookeeper service and a headless service

kubectl apply -f zookeeper.yaml 

Deploy kafka statefulset and a headless service

kubectl apply -f kafka.yaml 

Create a topic

Run a pod, move into the pod and run kafka-topics.sh script to create the topic

kubectl run -ti --image=gcr.io/google_containers/kubernetes-kafka:1.0-10.2.1 createtopic --restart=Never --rm -- kafka-topics.sh --create --topic test --zookeeper zk-cs.default.svc.cluster.local:2181 --partitions 1 --replication-factor 3

Run the Consumer process

kubectl run -ti --image=gcr.io/google_containers/kubernetes-kafka:1.0-10.2.1 consume --restart=Never --rm -- kafka-console-consumer.sh --topic test --bootstrap-server kafka-0.kafka-hs.default.svc.cluster.local:9093

This will block the terminal. Keep the terminal open as it is.

In another terminal Run the Producer process

kubectl run -ti --image=gcr.io/google_containers/kubernetes-kafka:1.0-10.2.1 produce --restart=Never --rm -- kafka-console-producer.sh --topic test --broker-list kafka-0.kafka-hs.default.svc.cluster.local:9093,kafka-1.kafka-hs.default.svc.cluster.local:9093,kafka-2.kafka-hs.default.svc.cluster.local:9093 done;

Write some text in the Producer terminal. After you press "enter" after the text, you shall see the same message on the Consumer terminal.

Note:

In this tutorial, we are using non-persistent volume. This volume gets deleted with the deletion of the pod. So you should always run Kafka/Zookeeper statefulSets with the persistentVolumeClaimTemplate (See the commented code in yaml files) with appropriate persistent volumes.


Ashish Graduated from MNNIT Allahabad in Computer Science stream. With ~10 years of experience Corporate experience in companies like Aricent in telecom domain & Guavus in big data domain, Ashish brings a lot of insight in Tech Industry Space. Here goes his expertise: Cloud Technologies - AWS, Azure, Docker, Kubernetes, Terraform, DevOps Tools - Chef, Puppet, Ansible, Jenkins, Programming - Python, C, Java. He worked with Nokia Siemens Network regarding Infrastructure Management. Apart from Tech. industry, he is passionate about the social sector.




Keywords : cloud containers deployment DevOps docker kubernetes-training kubernetes training


Recommended Reading


How to install Kubernetes Cluster on AWS EC2 instances

When someone begins learning Kubernetes, the first challenge is to setup the kubernetes cluster. Most of the online tutorials take help of virtual boxes and minikubes, which are good to begin with but have a lot of limitations. This article will guide you t...


Container is the new process and Kubernetes is the new Unix.

Once a microservice is deployed in a container it shall be scheduled, scaled and managed independently. But when you are talking about hundreds of microservices doing that manually would be inefficient. Welcome Kubernetes, for doing container orchestration ...


What are Big Data, Hadoop & Spark ? What is the relationship among them ?

Big Data is a problem statement & what it means is the size of data under process has grown to 100's of petabytes ( 1 PB = 1000TB ). Yahoo mail generates some 40-50 PB of data every day. Yahoo has to read that 40-50 PB of data & filter out spans. E-commerce...