Testing the high performance of Kafka with multiple partitions and multiple consumer for single topic.
What are we going to do:
1. Create 5 brokers
2. Create 3 partitions for a topic
3. Create 2 producer to send messages to one topic which will distribute messages to 3 partitions, evenly
4. Create 3 consumers, one for each partition, to consume messages
in parallel, by order.
What do we need:
1. Ubuntu 14.04
2. kafka_2.11-0.10.1.1
#1. create configuration file
cd /PATH/kafka_2.11-0.10.1.1/config
for i in $(seq 1 5);
do
cp server.properties server.properties${i};
#change id
sed -i "s/broker.id=0/broker.id=${i}/g" server.properties${i};
#change port
port=$(expr ${i} + 9092)
sed -i "s/#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/:${port}/g" server.properties${i};
#change log file
sed -i "s/log.dirs=\/tmp\/kafka-logs/log.dirs=\/tmp\/kafka-logs${i}/g" server.properties${i};
done
done
#2. create starting script "kafka-start.sh"
#start zookeeper
zookeeper-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/zookeeper.properties &
# start 5 kafka brokers
kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties1 &
kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties2 &
kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties3 &
kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties4 &
kafka-server-start.sh /PATH/kafka_2.11-0.10.1.1/config/server.properties5 &
####################################
#3. Run above script to launch brokers and type "jps", you should see 5 kafka processes
$jps
4928 Kafka
4924 Kafka
4925 Kafka
4926 Kafka
4927 Kafka
4923 QuorumPeerMain
#4. Create a topic "Hello "with 3 partitions
$kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic Hello
#5. Check out the partitions
$kafka-topics.sh --describe --zookeeper localhost:2181 --topic Hello
Topic: Hello Partition: 0 ...
Topic: Hello Partition: 1 ...
Topic: Hello Partition: 2 ...
#5. Open 3 terminals then start 3 consumers which only consume message from partition 0, 1 and 2 respectively
kafka-console-consumer.sh --bootstrap-server localhost:9093 --new-consumer --partition 0 -topic Hello
kafka-console-consumer.sh --bootstrap-server localhost:9093 --new-consumer --partition 1 -topic Hello
kafka-console-consumer.sh --bootstrap-server localhost:9093 --new-consumer --partition 2 -topic Hello
#6. Open 2 terminals to produce some messages
$kafka-console-producer.sh --broker-list localhost:9093,localhost:9094,localhost:9095,localhost:9096,localhost:9097, --topic Hello
m1
m2
m3
$kafka-console-producer.sh --broker-list localhost:9093,localhost:9094,localhost:9095,localhost:9096,localhost:9097, --topic Hello
m4
m5
m6
m1
m4
#8. Let Kafka do the load balance. To do so we must assign all consumers to one group
8.1. create a file named "kafka.consumer.group" with one line:
group.id=group1
8.2 Launch kafka consumer with that file
$kafka-console-consumer.sh --bootstrap-server localhost:9093 -- consumer.config kafka.consumer.group --new-consumer -topic Hello
#9. Now messages will be evenly distributed across all consumers without specifying partition