
HumioCluster Node Pools (Beta)
You can run multiple groups of LogScale nodes as part of a HumioCluster. For example, you can configure a node pool for ingest-only nodes and a node pool for digest and storage nodes.
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
name: example-humiocluster
spec:
targetReplicationFactor: 2
storagePartitionsCount: 24
digestPartitionsCount: 24
license:
secretKeyRef:
name: example-humiocluster-license
key: data
nodePools:
- name: digest-storage
spec:
image: "humio/humio-core:1.36.0"
nodeCount: 3
dataVolumeSource:
hostPath:
path: "/mnt/disks/vol1"
type: "Directory"
resources:
limits:
cpu: "8"
memory: 56Gi
requests:
cpu: "6"
memory: 52Gi
environmentVariables:
- name: "KAFKA_SERVERS"
value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
- name: ingest-only
spec:
image: "humio/humio-core:1.36.0"
nodeCount: 3
dataVolumePersistentVolumeClaimSpecTemplate:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 10Gi
resources:
limits:
cpu: "8"
memory: 56Gi
requests:
cpu: "6"
memory: 52Gi
environmentVariables:
- name: NODE_ROLES
value: "httponly"
- name: "KAFKA_SERVERS"
value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"Lightweight Ingest-Only Node Pool
Like standard ingest-only nodes, lightweight ingest-only nodes have no local segment storage, do not run digest or query workloads, and cannot serve as a table coordinator. The key differences are:
Lower memory footprint: lightweight ingest-only nodes skip loading segment metadata from the global entity store.
No lookup file management: lightweight ingest-only nodes do not participate in lookup file synchronization, storage, or cleanup.
Reduced HTTP surface: only ingest-related HTTP endpoints are served on nodes with this role.
Enabling lightweight ingest-only nodes requires two steps performed in sequence:
Set
ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITYtotrueon every node in the cluster and restart. This enables the global data layer that supports lightweight ingest-only nodes, and is compatible with existing configurations even if no lightweight ingest-only nodes are present yet.After verifying the cluster is healthy, add new nodes with
NODE_ROLESset tolightweightingestonly.
The following example shows a node pool configuration that includes a lightweight ingest-only node pool alongside a digest-storage pool:
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
name: example-humiocluster
spec:
targetReplicationFactor: 2
storagePartitionsCount: 24
digestPartitionsCount: 24
license:
secretKeyRef:
name: example-humiocluster-license
key: data
nodePools:
- name: digest-storage
spec:
image: "humio/humio-core:1.248.0"
nodeCount: 3
dataVolumeSource:
hostPath:
path: "/mnt/disks/vol1"
type: "Directory"
resources:
limits:
cpu: "8"
memory: 56Gi
requests:
cpu: "6"
memory: 52Gi
environmentVariables:
- name: ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITY
value: "true"
- name: "ZOOKEEPER_URL"
value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
- name: "KAFKA_SERVERS"
value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
- name: lightweight-ingest-only
spec:
image: "humio/humio-core:1.248.0"
nodeCount: 3
dataVolumePersistentVolumeClaimSpecTemplate:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 10Gi
resources:
limits:
cpu: "4"
memory: 16Gi
requests:
cpu: "2"
memory: 12Gi
environmentVariables:
- name: NODE_ROLES
value: "lightweightingestonly"
- name: ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITY
value: "true"
- name: "ZOOKEEPER_URL"
value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
- name: "KAFKA_SERVERS"
value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"Important
To roll back lightweight ingest-only nodes, remove the
lightweightingestonly nodes
from the cluster. The
ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITY configuration
does not need to be reverted to
false. If rolling back the
compatibility configuration itself, set
ENABLE_LIGHTWEIGHT_INGEST_COMPATIBILITY to
false and restart the cluster.