LogScale requires low latency access to a Kafka cluster to operate optimally, you must have an available Kafka cluster before deploying LogScale.

Non-kafka systems that are similar are not supported, for example Google PubSub or Azure EventHub are not supported.

Sub 50 millisecond ping times from the LogScale pods to the Kafka cluster will ensure data is ingested quickly and be available for search in less than a second.

In its default configuration LogScale will automatically create the Kafka topics and partitions required to operate. This functionality can be disabled by setting the KAFKA_MANAGED_BY_HUMIO value to false. Note that when enabled, LogScale will set up the ingest queue, global, and chatter topics in Kafka using reasonable default settings, but does not edit existing topics to conform to these defaults.

Running Kafka on Kubernetes can be accomplished a variety of ways. The Strimzi Kafka Operator is one such way and uses the same operator patterns that the Humio Operator uses to manage the life-cycles of both Kafka and KRaft nodes. In production setups, LogScale, Kafka, and KRaft should be run on separate worker nodes in the cluster. Both Kafka and KRaft must use persistent volumes provided by the Kubernetes environment, and they should not use ephemeral disks in a production deployment. Kafka brokers and KRaft instances should be deployed on worker nodes in different locations such as racks, data centers, or availability zone to ensure reliability. Kubernetes operators such, as the Strimzi operator, do not create worker nodes and label them, that task is left to the administrators.

For the latest configurations see the LogScale GitHub repository.

For more general information on configuring Strimzi please see their documentation.