LogScale on Bare Metal - Installing Apache Kafka Cluster
LogScale uses Apache Kafka to manage ingest processing and inter-node communication within the LogScale cluster. LogScale recommends a minimum 3-node installation.
When deploying a Kafka, Kafka can be deployed using ZooKeeper or without ZooKeeper using Kraft mode. Each Kafka node must have a unique ID:
Using Kafka with ZooKeeper or Kraft Mode
For Kafka using ZooKeeper a simple integre can be used with the
broker.id
configuration value in theserver.properties
file:ini# The id of the broker. This must be set to a unique integer for each broker. broker.id=1
Using Kafka using Kraft
If using Kafka using Kraft mode, an alternative is to create a UUID:
shell$
bin/kafka-storage.sh random-uuid
If using ZooKeeper, a file called
myid
will need to be created in the ZooKeeper data directory that contains the node ID number. To create, echo the number to the file:shell$
echo 1 >/kafka/zookeeper/myid
LogScale on Bare Metal - Apache Kafka Server Preparation
We recommend installing on Ubuntu, at least version 18.04. Before installing Kafka, make sure the server is up-to-date:
$ apt-get update
$ apt-get upgrade
Create a non-administrative user named
kafka
to run Kafka:
$ adduser kafka --shell=/bin/false --no-create-home --system --group
Add this user to the DenyUsers
section of each
nodes /etc/ssh/sshd_config
file to prevent it from
being able to ssh or sftp into the
node.
Restart the sshd daemon after making the change. Once the system has finished updating and the user has been created, Kafka can be installed.