Скачать или смотреть Real Time Kafka Installation And Enable Metricbeat Kafka Module To Ingest Logs Elasticsearch 8.x V

Prev video links
=============
Install And Configure Elasticsearch,Logstash,Kibana,Filebeat 8.15.2 In Ubuntu 22 .04 Using AWS EC2.    • Install And Configure Elasticsearch,Logsta...  
Install And Configure Elasticsearch/Metricbeat/Kibana 8.15.2 Version in Ubuntu 22.04 Using AWS EC2    • Install And Configure Elasticsearch/Metric...  
Enable and Configure XPACK Security In Elasticseach 8.x version Realtime.
   • Enable and Configure XPACK  Security In El...  

Real Time Kafka Installation And Enable Metricbeat Kafka Module To Ingest Logs Elasticsearch 8.x V

Please find the below commands for kafka Installation
Add a new user called kafka:
sudo adduser kafka
Next, you need to add the kafka user to the sudo group to have the necessary privileges for Kafka installation.
sudo adduser kafka sudo
Then, log in to the kafka account:
su -l kafka
The kafka user now is ready to be used.
Step2: Installing Java Development Kit (JDK) 
Apache Kafka is written in Java and Scala, which means Java Runtime Environment (JRE) is required to run it. However, for a complete development setup that may involve custom Kafka clients or plugins, the full Java Development Kit (JDK) is recommended.
Installing Java Development Kit
Open the terminal and update the package index:
sudo apt update
Install the OpenJDK 11 package:
sudo apt install openjdk-11-jdk
Step3: Downloading Kafka 
mkdir ~/downloads
cd ~/downloads
wget https://archive.apache.org/dist/kafka/3.4....
Then, move to ~ and extract the archive you downloaded:
cd ~
tar -xvzf ~/downloads/kafka_2.12-3.4.0.tgz
Let’s rename the directory kafka_2.12-3.4.0 to kafka.
mv kafka_2.12-3.4.0/ kafka/
Now that you’ve downloaded Kafka, you can start configuring your Kafka server.
Step4: Configuring the Kafka server
First, start by setting the log.dirs property to change the directory where the Kafka logs are.
To do so, you need to edit the server.properties file:
vi ~/kafka/config/server.properties
Look for log.dirs and set the value to /home/kafka/kafka-logs.

You can also change the value of num.partition to 3 so that when you create the topic you don’t specify the number of partitions, it will be 3 by default.
Now that you’ve finished configuring your Kafka server, you can run the server.
Step5: Starting the Kafka server
To start the Kafka server, you need to first start Zookeeper and then start Kafka.
Unit File for Zookeeper:

sudo vi /etc/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper Service
Requires=network.target
After=network.target

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Unit File for Kafka:
sudo nano /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka Service that requires zookeeper service
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart= /home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target
Then, you can start the Kafka server:
sudo systemctl start kafka
Check the status:
sudo systemctl status kafka
Step 6: Testing the Kafka server
You can check if the Kafka server is up with netcat. By default, Kafka server runs on 9092:
nc -vz localhost 9092
You can also check logs:
cat ~/kafka/logs/server.log
It looks like it’s all good.

If your server is running successfully, try to create a topic:
~/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic firstTopic
Let’s check the topics’ list:
~/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
You can produce messages to the topic:
~/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic firstTopic
You can then read the messages:
~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic firstTopic --from-beginning