Skip to content
LogScale Self-Hosted Deployment
Content was updated: May 21, 2024
This is an archived version of the documentation downloaded from LogScale Documentation Archives. Up to date documentation, and documentation for later versions, is available on LogScale Documentation

## LogScale on Bare Metal - Installing Apache Kafka Cluster

Instructions for installing and configuring Apache Kafka clusters in KRaft mode as part of a LogScale bare metal deployment, with a focus on setting up a minimum three-node installation on Ubuntu 18.04 or later. This page covers essential steps including system preparation, user creation, Kafka installation, configuration of node-specific properties, and service setup, ensuring proper inter-node communication and cluster management.

LogScale uses Apache Kafka to manage ingest processing and inter-node communication within the LogScale cluster.

When deploying a Kafka cluster, each node must have a unique ID. When using Kafka using KRaft mode, this is achieved by using a UUID, which can be created as follows:

shell

    $ bin/kafka-storage.sh random-uuid

### Note

When deploying Kafka using KRaft mode, nodes can be specified as a broker, a controller, or both. For production deployments it is recommended you have a minimum of three Kafka controllers, and three Kafka brokers, in addition to your three LogScale nodes. Further information on configuring Kafka in KRaft mode can be found in the KRaft documentation.

### LogScale on Bare Metal - Apache Kafka Server Preparation

We recommend installing on Ubuntu, at least version 18.04. Before installing Kafka, make sure the server is up-to-date:

shell

    $ apt-get update
    $ apt-get upgrade

Create a non-administrative user named `kafka` to run Kafka:

shell

    $ adduser kafka --shell=/bin/false --no-create-home --system --group

Add this user to the `DenyUsers` section of each nodes `/etc/ssh/sshd_config` file to prevent it from being able to ssh or sftp into the node.

Restart the sshd daemon after making the change. Once the system has finished updating and the user has been created, Kafka can be installed.

### LogScale on Bare Metal - Apache Kafka Installation using KRaft

To install Kafka using KRaft:

1. Go to the `/opt` directory and download the latest release. The package can be downloaded using wget:

   shell

       $ cd /opt
       $ wget https://downloads.apache.org/kafka/${kafka_version}/kafka_${scala_version}-${kafka_version}.tgz

   ### Note

   In this example, the `${kafka_version}` variable would be replaced by the latest Kafka version and `${scala_version}` replaced by the Scala version used to build that version of Kafka. The list of currently available Kafka downloads can be found on the download page.

2. Extract the archive and create directories it needs like this:

   shell

       $ tar zxf kafka_${scala_version}-${kafka_version}.tgz

3. Now create the directories where the information will be stored. We will use the top level directory `/kafka` since that could be a mount point for a separate filesystem. We will also create a directory for application log files in `/var/log/kafka`:

   shell

       $ mkdir /var/log/kafka
       $ mkdir /kafka/kafka
       $ chown kafka:kafka /kafka/kafka

   Now link the application directory to `/opt/kafka` which will allow us to use `/opt/kafka` for the application and scripts, but update the version by downloading and relinking to the updated application directory:

   shell

       $ ln -s /opt/kafka_2.13-3.9.0 /opt/kafka

4. Using a text editor, open the Kafka properties file, `server.properties`, located in the `kafka/config` sub-directory. The following options are configured for using Kafka in Kraft mode. The hostnames and port numbers will be shared across each node. Configuration files for three different nodes are shown below:

   Node 1
   ini

       node.id=1
       controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
       listeners=PLAINTEXT://kafka1:9092,CONTROLLER://kafka1:9093
       advertised.listeners=PLAINTEXT://kafka1:9092
       num.partitions=6
       logs.dir=/opt/kafka/kraft-combined-logs

   Node 2
   ini

       node.id=2
       controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
       listeners=PLAINTEXT://kafka2:9092,CONTROLLER://kafka2:9093
       advertised.listeners=PLAINTEXT://kafka2:9092
       num.partitions=6
       logs.dir=/opt/kafak/kraft-combined-logs

   Node 3
   ini

       node.id=3
       controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
       listeners=PLAINTEXT://kafka3:9092,CONTROLLER://kafka3:9093
       advertised.listeners=PLAINTEXT://kafka3:9092
       num.partitions=6
       logs.dir=/opt/kafka/kraft-combined-logs

   The first line sets the `node.id` value, a unique node must be set for each node. The `controller.quarum.voters` sets the nodes that will choose how work is distributed. The next two lines set the ports and protocol types. The `logs.dir` sets the location of data directories.

   Update the ownership of the directory where the logs are stored:

   shell

       $ chown -R kafka:kafka /opt/kafka/logs

   Modify the directory according to the version of Kafka that has been installed.

5. If deploying a multi-node Kafka cluster, make sure that each node can resolve the hostname of each other node in the cluster. One way to achieve this is to edit the `/etc/hosts` file on each node with the host information:

   ini

       192.168.1.15 kafka1
       192.168.1.16 kafka2
       192.168.1.17 kafka3

   ### Important

   Be aware that in some Linux distributions, the hosts file may contain a line that by default resolves the hostname to the localhost address, `127.0.0.1`. This will cause servers to only listen on the localhost address and therefore not accessible to other hosts on the network. In this case, change the line to:

   ini

       127.0.1.1 kafka1 localhost

   Updating the IP address to the public address of the host.

6. Now create a service file for Kafka.

   Create the file `/etc/systemd/system/kafka.service` sub-directory, edit the file add the following lines:

   ini

       [Unit]

       [Service]
       Type=simple
       User=kafka
       LimitNOFILE=800000
       Environment="LOG_DIR=/var/log/kafka"
       Environment="GC_LOG_ENABLED=true"
       Environment="KAFKA_HEAP_OPTS=-Xms512M -Xmx4G"
       ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
       Restart=on-failure
       TimeoutSec=900

       [Install]
       WantedBy=multi-user.target

7. Now start the Kafka service:

   shell

       $ systemctl start kafka
       $ systemctl status kafka
       $ systemctl enable kafka

   These steps must be repeated on each host in a multi-node deployment.

#### Enter search term