HumioCluster Node Pools (Beta)

You can run multiple groups of LogScale nodes as part of a HumioCluster. For example, you can configure a node pool for ingest-only nodes and a node pool for digest and storage nodes.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  license:
    secretKeyRef:
      name: example-humiocluster-license
      key: data
  nodePools:
    - name: digest-storage
      spec:
        image: "humio/humio-core:1.36.0"
        nodeCount: 3
        dataVolumeSource:
          hostPath:
            path: "/mnt/disks/vol1"
            type: "Directory"
        resources:
          limits:
            cpu: "8"
            memory: 56Gi
          requests:
            cpu: "6"
            memory: 52Gi
        environmentVariables:
          - name: "KAFKA_SERVERS"
            value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
    - name: ingest-only
      spec:
        image: "humio/humio-core:1.36.0"
        nodeCount: 3
        dataVolumePersistentVolumeClaimSpecTemplate:
          storageClassName: standard
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 10Gi
        resources:
          limits:
            cpu: "8"
            memory: 56Gi
          requests:
            cpu: "6"
            memory: 52Gi
        environmentVariables:
          - name: NODE_ROLES
            value: "httponly"
          - name: "KAFKA_SERVERS"
            value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"

Lightweight Ingest-Only Node Pool

Like standard ingest-only nodes, lightweight ingest-only nodes have no local segment storage, do not run digest or query workloads, and cannot serve as a table coordinator. The key differences are:

  • Lower memory footprint: lightweight ingest-only nodes skip loading segment metadata from the global entity store.

  • No lookup file management: lightweight ingest-only nodes do not participate in lookup file synchronization, storage, or cleanup.

  • Reduced HTTP surface: only ingest-related HTTP endpoints are served on nodes with this role.

Enabling lightweight ingest-only nodes requires two steps performed in sequence:

  1. Set ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITY to true on every node in the cluster and restart. This enables the global data layer that supports lightweight ingest-only nodes, and is compatible with existing configurations even if no lightweight ingest-only nodes are present yet.

  2. After verifying the cluster is healthy, add new nodes with NODE_ROLES set to lightweightingestonly.

The following example shows a node pool configuration that includes a lightweight ingest-only node pool alongside a digest-storage pool:

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  license:
    secretKeyRef:
      name: example-humiocluster-license
      key: data
  nodePools:
    - name: digest-storage
      spec:
        image: "humio/humio-core:1.248.0"
        nodeCount: 3
        dataVolumeSource:
          hostPath:
            path: "/mnt/disks/vol1"
            type: "Directory"
        resources:
          limits:
            cpu: "8"
            memory: 56Gi
          requests:
            cpu: "6"
            memory: 52Gi
        environmentVariables:
          - name: ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITY
            value: "true"
          - name: "ZOOKEEPER_URL"
            value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
          - name: "KAFKA_SERVERS"
            value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
    - name: lightweight-ingest-only
      spec:
        image: "humio/humio-core:1.248.0"
        nodeCount: 3
        dataVolumePersistentVolumeClaimSpecTemplate:
          storageClassName: standard
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 10Gi
        resources:
          limits:
            cpu: "4"
            memory: 16Gi
          requests:
            cpu: "2"
            memory: 12Gi
        environmentVariables:
          - name: NODE_ROLES
            value: "lightweightingestonly"
          - name: ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITY
            value: "true"
          - name: "ZOOKEEPER_URL"
            value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
          - name: "KAFKA_SERVERS"
            value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"

Important

To roll back lightweight ingest-only nodes, remove the lightweightingestonly nodes from the cluster. The ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITY configuration does not need to be reverted to false. If rolling back the compatibility configuration itself, set ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITY to false and restart the cluster.