How-To: Migrating Kafka to humio-core Deployment

A LogScale deployment can be migrated from one Kafka deployment method to another. This can be useful if:

  • Migrating from the humio combined container, deprecated in LogScale 1.122.0.

  • Migrating from the humio-kafka and optional humio-zookeeper container deployments, deprecated in 1.123.0, that are already using the humio-core

When migrating, the following conditions exist:

  • This process does not move or change the data associated in the LogScale cluster, as this data is not stored within Kafka.

  • Do not attempt to run multiple Kafka clusters with LogScale. Use the method shown here to stop and reconfigure the LogScale cluster to use a different Kafka cluster.

Migration Steps

Warning

Following these instructions may cause a small period of data loss while the individual nodes are moved from one Kafka cluster to another.

To migrate to a new Kafka cluster, follow these steps to create a new cluster and point your LogScale configuration:

  1. Deploy a new Kafka cluster. A configuration using containers is available: Deploying a Kafka Cluster using Containers.

  2. If possible, stop or minimize ingest to the cluster. The instructions for blocking ingest are one option, detailed in Disabling Ingestion.

  3. If available, consider using the Ingest in the humio/insights package to monitor for ingest latency and delay.

  4. Shutdown the entire LogScale cluster.

  5. On each node in the LogScale cluster, update the Kafka configuration variables:

    • Update the KAFKA_SERVERS variable to point to the new list of Kafka servers in the new cluster.

      For example:

      shell
      KAFKA_SERVERS=newkafka1:9092,newkafka2:9092,newkafka3:9092

  6. Restart each container.

    When each node comes backed up, the new Kafka cluster should be used.