Saturday, February 11, 2017

Create 5 brokers in 1 kafka server

Why are we doing this:
Testing the high performance of Kafka with multiple partitions and multiple consumer for single topic.


What are we going to do:

1. Create 5 brokers 
2. Create 3 partitions for a topic
3. Create 2 producer to send messages to one topic which will distribute messages to 3 partitions, evenly
4. Create 3 consumers, one for each partition, to consume messages
in parallel, by order.

What do we need:
1. Ubuntu 14.04
2. kafka_2.11-0.10.1.1


#1. create configuration file

cd /PATH/kafka_2.11-0.10.1.1/config

for i in $(seq 1 5);

do
cp server.properties server.properties${i};

#change id

sed -i "s/broker.id=0/broker.id=${i}/g" server.properties${i};

#change port

port=$(expr ${i} + 9092)
sed -i "s/#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/:${port}/g" server.properties${i};

#change log file
sed -i "s/log.dirs=\/tmp\/kafka-logs/log.dirs=\/tmp\/kafka-logs${i}/g" server.properties${i};

done





#2. create starting script "kafka-start.sh"

####################################
#start zookeeper
zookeeper-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/zookeeper.properties &

# start 5 kafka brokers 
kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties1 &
kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties2 &
kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties3 &
kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties4 &

kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties5 &
####################################

#3.  Run above script to launch brokers and type "jps", you should see 5 kafka processes

$jps
8700 Jps
4928 Kafka
4924 Kafka
4925 Kafka
4926 Kafka
4927 Kafka
4923 QuorumPeerMain

#4. Create a topic "Hello "with 3 partitions

$kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic Hello

#5. Check out the partitions
$kafka-topics.sh --describe --zookeeper localhost:2181 --topic Hello
Topic: Hello    Partition: 0 ...
Topic: Hello    Partition: 1 ... 
Topic: Hello    Partition: 2 ...


#5. Open 3 terminals then start 3 consumers which only consume message from partition 0, 1 and 2 respectively
kafka-console-consumer.sh --bootstrap-server localhost:9093 --new-consumer --partition 0 -topic Hello

kafka-console-consumer.sh --bootstrap-server localhost:9093 --new-consumer --partition 1 -topic Hello

kafka-console-consumer.sh --bootstrap-server localhost:9093 --new-consumer --partition 2 -topic Hello


#6. Open 2 terminals to produce some messages
$kafka-console-producer.sh --broker-list localhost:9093,localhost:9094,localhost:9095,localhost:9096,localhost:9097, --topic Hello
m1
m2

m3


$kafka-console-producer.sh --broker-list localhost:9093,localhost:9094,localhost:9095,localhost:9096,localhost:9097, --topic Hello
m4
m5
m6


#7. Switch back to the terminals from step #5. you will see each consumer will consume exactly 2 messages
m1
m4

#8. Let Kafka do the load balance. To do so we must assign all consumers to one group
  8.1. create a file named "kafka.consumer.group" with one line:
       group.id=group1 
  8.2  Launch kafka consumer with that file
  $kafka-console-consumer.sh --bootstrap-server localhost:9093 --     consumer.config kafka.consumer.group --new-consumer -topic Hello

#9. Now messages will be evenly distributed across all consumers without specifying partition