Go to the /opt directory and download the latest release. You can do thatub sing wget:

$ cd /opt
$ wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz

Extract the archive and create directories directories it needs like this:

$ tar zxf kafka_2.13-3.7.0.tgz

Now create the directories where the information will be stored. We will use the top level directory /kafka since that could be a mount point for a separate filesystem. We will also create a directory for application log files in /var/log/kafka:

$ mkdir /var/log/kafka
$ mkdir /var/log/zookeeper
$ mkdir /kafka/kafka
$ chown kafka:kafka /var/log/kafka /var/log/zookeeper
$ chown kafka:kafka /kafka/kafka
$ chown kafka:kafka /kafka/zookeeper

Now link the application directory to /opt/kafka which will allow us to use /opt/kafka for the applcation and scripts, but update the version by downloading and relinking to the updated application directory:

$ ln -s /opt/kafka_2.13-3.7.0 /opt/kafka

Using a text editor, open the Kafka properties file, server.properties, located in the kafka/config sub-directory. You'll need to set a few options — the lines below are not necessarily the order in which they'll be found in the configuration file:

broker.id=1
log.dirs=/kafka/kafka
delete.topic.enable = true

The first line sets the broker.id value to match the server number (in the myid file) you set when configuring ZooKeeper. The second sets the data directory. The third line should be added to the end of the configuration file. When you're finished, save the file and change the owner to the kafka user:

$ chown -R kafka:kafka /opt/kafka_2.13-3.7.0

You'll have to adjust this to the version you installed. Note, changing the ownership of the link /opt/kafka doesn't change the ownership of the files in the directory.

If you are a deploying a multi-node Kafka cluster, make sure that each node can resolve the hostname of each other node in the cluster. One way to achieve this is to edit your /etc/hosts file:

192.168.1.15 kafka1
192.168.1.16 kafka2
192.168.1.17 kafka3

Be aware that in some Linux distributions, the hosts file may contain a line that by default resolves the hostname to the localhost address, 127.0.0.1. This will cause servers to only listen on the localhost address and therefore not accessible to other hosts on the network. In this case, change the line:

127.0.1.1 kafka1 kafka1

Updating the IP address to the public addres of the host.

To configure the properties for ZooKeeper, edit the config/zookeeper.properties file with the following options:

dataDir=/kafka/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
server.1=kafka1:2888:3888
server.2=kafka2:2888:3888
server.3=kafka3:2888:3888
4lw.commands.whitelist=*
tickTime=2000
initLimit=5
syncLimit=2

The server.1, server.2 and server.3 configure the hostname and host-to-host ports used to communicate. These must match the broker.id Kafka configuration and myid file value.

The last three lines are required by ZooKeeper in multi-node configurations to set the timing interval for communicating with the other hosts and the time limit before reporting an error.

Set the node id for ZooKeeper:

$ mkdir /kafka/zookeeper
$ echo 1 > /kafka/zookeeper/myid
$ chown -R kafka:kafka /kafka/zookeeper

Create a service file for ZooKeeper so that it will run as a system service and be automatically managed to keep running.

Create the file /etc/systemd/system/zookeeper.service sub-directory, edit the file add the following lines:

[Unit]

[Service]
Type=simple
User=kafka
LimitNOFILE=800000
Environment="LOG_DIR=/var/log/zookeeper"
Environment="GC_LOG_ENABLED=true"
Environment="KAFKA_HEAP_OPTS=-Xms512M -Xmx4G"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/zookeeper.properties
Restart=on-failure
TimeoutSec=900

[Install]
WantedBy=multi-user.target

Now start and enable the service:

$ systemctl start zookeeper

You can check if the service is running by using the status option:

$ systemctl status zookeeper

You should get output similar to the following showing active (running) if the service is OK:

zookeeper.service
     Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-07 05:31:36 GMT; 1s ago
   Main PID: 4968 (java)
      Tasks: 16 (limit: 1083)
     Memory: 24.6M
        CPU: 1.756s
     CGroup: /system.slice/zookeeper.service
             ??4968 java -Xms512M -Xmx4G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true "-Xlog:gc*:file=/var/log/zookeeper/zookeeper-gc.log:time,tags:filecount=10,filesize=100M" -Dcom.sun.management.>

Mar 07 05:31:36 kafka1 systemd[1]: Started zookeeper.service.

This should report any issues which you should address before starting the service again. If everthing is OK, enable the service so that it will always start on boot:

$ systemctl enable zookeeper

Now create a service for Kafka. This file is slightly different because there is a dependency added so that the system will start ZooKeeper first if it is not running before trying to Kafka.

Create the file /etc/systemd/system/zookeeper.service sub-directory, edit the file add the following lines:

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
LimitNOFILE=800000
Environment="LOG_DIR=/var/log/kafka"
Environment="GC_LOG_ENABLED=true"
Environment="KAFKA_HEAP_OPTS=-Xms512M -Xmx4G"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
Restart=on-failure
TimeoutSec=900

[Install]
WantedBy=multi-user.target

Now start the Kafka service:

$ systemctl start kafka
$ systemctl status kafka
$ systemctl enable kafka

You will need to repeat this on each host in a multi-node deployment.