Скачать или смотреть P-1 Setting Up a Kafka Cluster with Zookeeper for High Availability

Installation Guide :   / installation-guide-for-kafka-cluster-with-...  

Introduction:
Apache Kafka is a powerful distributed streaming platform known for its real-time data streaming capabilities. To harness its full potential, setting up a Kafka cluster with high availability is essential. This article will guide you through the process of creating a Kafka cluster with three Zookeeper nodes and three broker nodes, ensuring a robust and fault-tolerant environment for your data streaming needs.

Requirements:
Before diving into the setup, make sure you have the following prerequisites in place:

Install Java:
Kafka relies on Java to run, so ensure that all machines in your cluster have Java installed.

Kafka Binary:
Download the Kafka binary and copy it to all three machines where you plan to set up Kafka.

Setting Up Zookeeper Cluster with High Availability:
Zookeeper plays a crucial role in maintaining Kafka cluster metadata. Here's how to set up Zookeeper with high availability:

Configure DNS Entries:
Ensure DNS entries are properly configured on all servers. You can edit the /etc/hosts file to map server IP addresses to hostnames.

bash
Copy code
vim /etc/hosts
Download and Extract Zookeeper:
Download the Zookeeper distribution tarball on all nodes and extract it.

bash
Copy code
wget https://downloads.apache.org/kafka/3....
tar -zxvf kafka_2.12–3.4.0.tgz
Create a Common Directory:
Create a directory named /app on all nodes and move the Kafka binary to that directory.

bash
Copy code
mkdir /app
mv kafka_2.12–3.4.0 /app
Configure Zookeeper Properties:
Create and edit the zookeeper.properties file in the Kafka configuration directory (/app/kafka_2.12–3.4.0/config/zookeeper.properties) on all nodes. Add the following properties:

properties
Copy code
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/app/data/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
server.1=192.168.213.133:2888:3888
server.2=192.168.213.134:2888:3888
server.3=192.168.213.135:2888:3888
Set Zookeeper IDs:
On each Zookeeper node, create a unique ID file in the /app/data/zookeeper directory:

On Zookeeper Node 1:

bash
Copy code
vi /app/data/zookeeper/myid
Add the value 1 and save the file.

Repeat this process on Zookeeper Node 2 and Node 3 with their respective IDs.

Start Zookeeper:
Start Zookeeper on all nodes:

bash
Copy code
./bin/zookeeper-server-start.sh -daemon ./config/zookeeper.properties
After a short while, Zookeeper services should be up and running.

Setting Up Kafka Cluster with Multiple Brokers:
Now that your Zookeeper cluster is running smoothly, it's time to set up the Kafka cluster with multiple brokers:

Create Kafka Data Directory:
Create a directory for Kafka data on all nodes:

bash
Copy code
mkdir /app/data/kafka-logs
Configure Server Properties:
Edit the server.properties file in the Kafka configuration directory on each node. Customize the following properties, ensuring that each broker has a unique broker.id:

On Centos-1:

bash
Copy code
vim /app/kafka_2.12–3.4.0/config/server.properties
properties
Copy code
broker.id=0 # unique value
listeners=PLAINTEXT://localhost:9092
delete.topic.enable=true
auto.create.topics.enable=false
advertised.listeners=PLAINTEXT://localhost:9092
log.dirs=/app/data/kafka-logs
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=2
zookeeper.connect=192.168.213.133:2181,192.168.213.134:2181,192.168.213.135:2181
Repeat this process on Centos-2 and Centos-3, incrementing the broker.id accordingly.

Start Kafka Brokers:
Start the Kafka brokers on all nodes:

bash
Copy code
./bin/kafka-server-start.sh -daemon ./config/server.properties
Check Broker Status:
Verify that the Kafka brokers are running and connected to Zookeeper:

bash
Copy code
ss -tulpn | grep java
Conclusion:
You've successfully set up a Kafka cluster with three Zookeeper nodes and three broker nodes, ensuring high availability for your data streaming needs. This robust and fault-tolerant architecture will help you process real-time data efficiently and reliably. Enjoy harnessing the power of Kafka for your applications!