Switch Kafka using Kraft Mode
There are three options for switching Kafka and ZooKeeper:
Set up a new Kafka/ZooKeeper cluster and re-configure to use the new cluster.
Delete the Kafka data.
Create new Kafka queues and topics with new names in the Kafka cluster.
The first method is the simplest.
Delete Kafka Data and Re-Use Cluster
You could reset the current Kafka cluster by deleting the data directories on the filesystem on each node. This is the same as starting up a new and empty Kafka cluster. To do this, delete everything inside Kafka data directory.
Create New Kafka Queues/Topics with New Names in Kafka Cluster
Instead of resetting Kafka as described above, you can let
use a new set of queues in the existing Kafka
cluster. When reusing the same Kafka cluster, must be
configured with a new HUMIO_KAFKA_TOPIC_PREFIX
to
detect the changes.
It's important to note that it will not work to delete and recreate
topics with the same names. In that case cannot
detect the Kafka switch. If Kafka is managed by
(KAFKA_MANAGED_BY_HUMIO
), the new topics will be
created automatically when starts up. Otherwise you
must create topics externally before you start .
Restarting Kafka using Kraft and
Now you're ready to get the Kafka cluster started. Start all Kafka nodes, wait for them to be running and verify the Kafka cluster.
Once Kafka has started, start the nodes. It's important to start one node first. This node will detect the Kafka switch and create a new epoch in .
To verify that the Kafka switch was detected and handled, look for this line in the debug log:
Switching epoch to=${epochKey} from=${latestEpoch.kafkaClusterId}
- I'm the first cluster member to get here
for this kafka. newEpoch=${newEpoch}
When the first node is up and running and the above logline confirms a new epoch has been created, the rest of the nodes can be started.
At that point, the cluster should be running again.
Check the cluster nodes in the administrative section of the
user interface:
http://$HUMIOHOST/system/administration/partitions/ingest