Deploying a Kafka Cluster using Containers

There are many different Kafka distributions available, for example Strimzi or Apache

The instructions below use Strimzi and are provided as a simplified guide, not as configuration for a production system.

Important

Kafka and LogScale should be run on different instances and hardware. This will help ensure performance for both systems when ingesting data.

For production, a valid Kafka cluster should have at least six instances (pods/nodes), consisting of three brokers and three controllers. There also needs to be a minimum of three availability zones for an HA cluster. Additionally, there must be low latency (<300ms) between pods.

When configuring Kafka, each Kafka host must have its own unique node number. For example, in a three node cluster:

Host Kafka ID
kafka1 1
kafka2 2
kafka3 3

When using Kafka in KRaft mode, you need to configure:

  • A node.id parameter, or KAFKA_NODE_ID variable in Docker to set the node ID.
  • A unique a cluster ID.
  • In ZooKeeper mode (only for versions of LogScale before 1.108), Kafka uses ZooKeeper and the broker.id parameter, or KAFKA_BROKER_ID variable in Docker is used to set the unique broker ID.

When configuring each node, ensure that the listener host and port number is accessible to other hosts and that the LogScale instances can reach them over the network. The default Kafka ports should be open and accessible between docker images. If in doubt, please refer to the Kafka documentation.

Verify Kafka is running

To verify that Kafka is running, use the following command:

  • Use nc to get the status of each ZooKeeper instance. The following must respond with either Leader or Follower for all instances

    shell
    $ echo stat | nc 192.168.1.1 2181 | grep '^Mode: '

  • Optionally, use your favorite Kafka tools to validate the state of your Kafka cluster. You could list the topics using the following command, to get an empty list since this is a fresh install of Kafka:

    shell
    $ kafka-topics.sh --zookeeper localhost:2181 --list