LogScale on Bare Metal - Installing Apache Kafka Cluster

LogScale uses Apache Kafka to manage ingest processing and inter-node communication within the LogScale cluster. LogScale recommends a minimum 3-node installation.

When deploying a Kafka, Kafka can be deployed using ZooKeeper or without ZooKeeper using Kraft mode. Each Kafka node must have a unique ID:

  • Using Kafka with ZooKeeper or Kraft Mode

    For Kafka using ZooKeeper a simple integre can be used with the broker.id configuration value in the server.properties file:

    ini
    # The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

  • Using Kafka using Kraft

    If using Kafka using Kraft mode, an alternative is to create a UUID:

    shell
    $ bin/kafka-storage.sh random-uuid

  • If using ZooKeeper, a file called myid will need to be created in the ZooKeeper data directory that contains the node ID number. To create, echo the number to the file:

    shell
    $ echo 1 >/kafka/zookeeper/myid

LogScale on Bare Metal - Apache Kafka Server Preparation

We recommend installing on Ubuntu, at least version 18.04. Before installing Kafka, make sure the server is up-to-date:

shell
$ apt-get update
$ apt-get upgrade

Create a non-administrative user named kafka to run Kafka:

shell
$ adduser kafka --shell=/bin/false --no-create-home --system --group

Add this user to the DenyUsers section of each nodes /etc/ssh/sshd_config file to prevent it from being able to ssh or sftp into the node.

Restart the sshd daemon after making the change. Once the system has finished updating and the user has been created, Kafka can be installed.