Kafka first step – running a simple server

Kafka is a very fast queue server.

The main functions are as follows.

  • Support for peer to peer, publish – subscribe messaging
  • Built in cluster support
  • Consistent performance as data grows

This post shows the first step to run a simple server. (Refer to the full documention)

The following steps are tested on Kafka 2.12

1. Install Kafka

  • Download from kafka.apache.org
  • Unzip kafka_version.tgz
  • For test, I copied kafka_2.12-2.1.0 directory into /home/tkstone/kafka_2.12-2.1.0 (I point it as KAKFA_HOME)

2. Start Zookeeper

Kafka uses Zookeeper as a metadata repository. (also as a Kafka cluster manager)

You can get Zookeeper from zookeeper.apache.org or use the one which comes with Kafka.

Run Zookeeper which is contained within Kafka as follows.

#{KAKFA_HOME}/bin/zookeeper-server-start.sh

3. Build a Kafka config

There are several config samples inside ${KAFKA_HOME}/conf. Copy server.properties into test.properties. And there are some properties to customize.

  • broker.id : This value must be unique among cluster
  • listeners : listen ip and port.
  • log.dirs : The directories which saves log. Kafka logs are not an informative files, but persistent data files
  • zookeeper.connect : zookeeper connect string

4. Start a standalone server

export JAVA_HOME=/usr/local/jdk1.8.0_151
KAFKA_HOME=/home/tkstone/kafka
${KAFKA_HOME}/bin/kafka-server-start.sh ${KAFKA_HOME}/config/test.properties > ${KAFKA_HOME}/testserver/server.log 2>&1 &

5. Make a topic

${KAFKA_HOME}/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic topic_name

Now you are ready to test Kafka.

To check the topics which are generated, run the following command.

${KAFKA_HOME}/bin/kafka-topics.sh –list –zookeeper localhost:2181

After starting Kafka, you can verify that some zookeeper znodes are made.

[zk: localhost:2181(CONNECTED) 59] ls /
[log_dir_event_notification, isr_change_notification, zookeeper, admin, consumers, cluster, config, latest_producer_id_block, brokers, controller_epoch]

Also, you can verify generated topics.

[zk: localhost:2181(CONNECTED) 61] ls /config/topics
[test-topic2]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.