stock-market-Kafka-real-time-data-engineering-project/command_kafka.txt at main · haseenanissar/stock-market-Kafka-real-time-data-engineering-project · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
wget https://dist.apache.org/repos/dist/dev/kafka/4.1.0-rc3/kafka_2.13-4.1.0.tgz
tar -xvf kafka_2.13-4.1.0.tgz


-----------------------
java -version
sudo yum install java-17-amazon-corretto -y
java -version
cd kafka_2.12-3.3.1

mkdir -p config/kraft
nano config/kraft/server.properties

# Kafka Cluster ID: already generated(paste uuid)
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://localhost:9093
advertised.listeners=PLAINTEXT://<YOUR-EC2-PUBLIC-IP>:9092
log.dirs=/tmp/kraft-combined-logs


bin/kafka-storage.sh format --config config/kraft/server.properties --cluster-id $(bin/kafka-storage.sh random-uuid)

if invalid cluster id
kill kafka
sudo rm -rf /tmp/kraft-combined-logs


# to kill kafka
ps -ef | grep kafka
kill -9 <pid>
 #cleanup


to edit
sudo nano /home/ec2-user/kafka_2.13-4.1.0/config/server.properties


if success
Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 4.1-IV1.


to run kafka in the background
# Go to Kafka directory
cd ~/kafka_2.13-4.1.0

Start Kafka-server:
----------------------------------------
Duplicate the session & enter in a new console --
export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"
cd kafka_2.13-4.1.0
bin/kafka-server-start.sh config/kraft/server.properties


It is pointing to private server , change server.properties so that it can run in public IP

To do this , you can follow any of the 2 approaches shared belwo --
Do a "sudo nano config/server.properties" - change ADVERTISED_LISTENERS to public ip of the EC2 instance


Create the topic:
-----------------------------
Duplicate the session & enter in a new console --
cd kafka_2.12-3.3.1
bin/kafka-topics.sh --create --topic demotopic --bootstrap-server 3.136.155.161:9092 --replication-factor 1 --partitions 1


to list existing topic
-----------------------
bin/kafka-topics.sh --list --bootstrap-server <your-ec2-public-ip>:9092

Start Producer:
--------------------------
bin/kafka-console-producer.sh --topic demotopic --bootstrap-server 3.136.155.161:9092

Start Consumer:
-------------------------
Duplicate the session & enter in a new console --
cd kafka_2.12-3.3.1
bin/kafka-console-consumer.sh --topic demotopic --bootstrap-server 3.136.155.161:9092

to check if borker responds
----------------------------
cd kafka_2.13-4.1.0
bin/kafka-topics.sh --list --bootstrap-server <your-ec2-public-ip>:9092

if address already in use when kafka start error

ss -tulnp | grep -E "9092|9093"
//it will show ports
 kill -9 number


 bin/kafka-topics.sh --describe --topic demotopic --bootstrap-server 52.15.165.198:9092
Topic: demotopic        TopicId: oZpZXcmIQvmQaUszpTYGPw PartitionCount: 1       ReplicationFactor: 1    Configs: min.insync.replicas=1
        Topic: demotopic        Partition: 0    Leader: 1       Replicas: 1     Isr: 1  Elr:    LastKnownElr: