Skip to content
LogScale Self-Hosted Deployment
This is an archived version of the documentation downloaded from LogScale Documentation Archives. Up to date documentation, and documentation for later versions, is available on LogScale Documentation

### Deploying a Kafka Cluster using Containers

There are many different Kafka distributions available, for example Strimzi or Apache

The instructions below use Strimzi and are provided as a simplified guide, not as configuration for a production system.

### Important

For production purposes, Kafka and LogScale should be run on different instances and hardware. This helps ensure high performance for both systems when ingesting data.

For production, a valid Kafka cluster should have at least six instances (pods/nodes), consisting of three brokers and three controllers. There also needs to be a minimum of three availability zones for an HA cluster. Additionally, there must be low latency (<300ms) between pods.

When configuring Kafka, each Kafka host must have its own unique node number. For example, in a three node cluster:

Host | Kafka ID
--- | ---
kafka1 | 1
kafka2 | 2
kafka3 | 3

When using Kafka in KRaft mode, you need to configure:

- A `node.id` parameter, or `KAFKA_NODE_ID` variable in Docker to set the node ID.

- A unique cluster ID.

- In ZooKeeper mode (only for versions of LogScale before 1.108), Kafka uses ZooKeeper and the `broker.id` parameter, or `KAFKA_BROKER_ID` variable in Docker is used to set the unique broker ID.

When configuring each node, ensure that the listener host and port number is accessible to other hosts and that the LogScale instances can reach them over the network. The default Kafka ports should be open and accessible between docker images. If in doubt, please refer to the Kafka documentation.

### Note

When deploying Kafka using KRaft mode, nodes can be specified as a broker, a controller, or both. For production deployments it is recommended you have a minimum of three Kafka controllers, and three Kafka brokers, in addition to your three LogScale nodes. Further information on configuring Kafka in KRaft mode can be found in the KRaft documentation.

#### Verify Kafka is running

To verify that Kafka is running, use the following command:

- Use nc to get the status of each ZooKeeper instance. The following must respond with either Leader or Follower for all instances

  shell

      $ echo stat | nc 192.168.1.1 2181 | grep '^Mode: '

- Optionally, use your favorite Kafka tools to validate the state of your Kafka cluster. You could list the topics using the following command, to get an empty list since this is a fresh install of Kafka:

  shell

      $ kafka-topics.sh --zookeeper localhost:2181 --list

#### Enter search term