HumioCluster Storage Configuration

LogScale supports several storage backends when running with the Humio Operator. Choose the configuration that matches your environment and durability requirements.

Ephemeral with S3 storage

A highly recommended LogScale Cluster configuration is to run in ephemeral mode, using S3 for persistent storage. The LogScale pods will be configured with hostPath, which mounts a directory from the host machine to the pod as local storage, and ideally made up of NVME SSDs. This configuration also has fairly high resource limits and affinity policies that ensure no two LogScale pods may be scheduled on the same host. This is an ideal storage configuration for production workloads running in AWS.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: "humio/humio-core:1.36.0"
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  resources:
    limits:
      cpu: "8"
      memory: 56Gi
    requests:
      cpu: "6"
      memory: 52Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: humio_node_type
                operator: In
                values:
                  - core
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                  - amd64
          - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                  - linux
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - humio
          topologyKey: kubernetes.io/hostname
  dataVolumeSource:
    hostPath:
      path: "/mnt/disks/vol1"
      type: "Directory"
  environmentVariables:
    - name: S3_STORAGE_BUCKET
      value: "my-cluster-storage"
    - name: S3_STORAGE_REGION
      value: "us-west-2"
    - name: S3_STORAGE_ENCRYPTION_KEY
      value: "my-encryption-key"
    - name: USING_EPHEMERAL_DISKS
      value: "true"
    - name: S3_STORAGE_PREFERRED_COPY_SOURCE
      value: "true"
    - name: "KAFKA_SERVERS"
      value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"

Ephemeral with GCS storage

A highly recommended LogScale Cluster configuration is to run in ephemeral mode, using GCS for persistent storage. The LogScale pods will be configured with hostPath, which mounts a directory from the host machine to the pod as local storage, and ideally made up of NVME SSDs. This configuration also has fairly high resource limits and affinity policies that ensure no two LogScale pods may be scheduled on the same host. This is an ideal storage configuration for production workloads running in GCP.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: humio-core:dynxml_current_release
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  resources:
    limits:
      cpu: "8"
      memory: 56Gi
    requests:
      cpu: "6"
      memory: 52Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: humio_node_type
                operator: In
                values:
                  - core
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                  - amd64
          - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                  - linux
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values:
                  - humio-core
          topologyKey: kubernetes.io/hostname
  dataVolumeSource:
    hostPath:
      path: "/mnt/disks/vol1"
      type: "Directory"
  extraHumioVolumeMounts:
    - name: gcp-storage-account-json-file
      mountPath: /var/lib/humio/gcp-storage-account-json-file
      subPath: gcp-storage-account-json-file
      readOnly: true
  extraVolumes:
    - name: gcp-storage-account-json-file
      secret:
        secretName: gcp-storage-account-json-file
  environmentVariables:
    - name: GCP_STORAGE_ACCOUNT_JSON_FILE
      value: "/var/lib/humio/gcp-storage-account-json-file"
    - name: GCP_STORAGE_BUCKET
      value: "my-cluster-storage"
    - name: GCP_STORAGE_ENCRYPTION_KEY
      value: "my-encryption-key"
    - name: USING_EPHEMERAL_DISKS
      value: "true"
    - name: "KAFKA_SERVERS"
      value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"

Persistent volumes

You can use Persistent Volumes as the backing store for LogScale data. This can be used as an alternative to Bucket Storage, however, Persistent Volumes using network block storage are significantly slower than local disks and LogScale will not perform well on this medium. For using Persistent Volumes using local storage, see Local Persistent Volumes (Beta).

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: "humio/humio-core:1.36.0"
  license:
     secretKeyRef:
       # Secret must be created with the following command: kubectl create secret generic example-humiocluster-license --from-literal=data=<license>
       name: example-humiocluster-license
       key: data
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  resources:
    limits:
      cpu: "8"
      memory: 56Gi
      requests:
      cpu: "6"
      memory: 52Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: humio_node_type
            operator: In
            values:
            - core
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - humio-core
        topologyKey: kubernetes.io/hostname
  dataVolumePersistentVolumeClaimSpecTemplate:
    storageClassName: standard
    accessModes: [ReadWriteOnce]
    resources:
      requests:
        storage: 500Gi
  environmentVariables:
    - name: "KAFKA_SERVERS"
      value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"

Local Persistent Volumes (Beta)

You can use local Persistent Volumes as the backing store for LogScale data, using something like Local Persistent Volumes. This can be used in combination with Bucket Storage, as the operator can be configured to clean up local Persistent Volumes that are attached to a node when that node is removed from the Kubernetes cluster. For this reason, it is extremely important to use USING_EPHEMERAL_DISKS=true along with Bucket Storage when using this option. This cleanup setting is enabled in the following example by setting dataVolumePersistentVolumeClaimPolicy.reclaimType=OnNodeDelete.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: "humio/humio-core:1.36.0"
  license:
     secretKeyRef:
       # Secret must be created with the following command: kubectl create secret generic example-humiocluster-license --from-literal=data=<license>
       name: example-humiocluster-license
       key: data
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  resources:
    limits:
      cpu: "8"
      memory: 56Gi
      requests:
      cpu: "6"
      memory: 52Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: humio_node_type
            operator: In
            values:
            - core
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - humio-core
        topologyKey: kubernetes.io/hostname
  dataVolumePersistentVolumeClaimSpecTemplate:
    storageClassName: local-storage
    accessModes: [ReadWriteOnce]
    resources:
      requests:
        storage: 500Gi
  dataVolumePersistentVolumeClaimPolicy:
    reclaimType: OnNodeDelete
  environmentVariables:
    - name: S3_STORAGE_BUCKET
      value: "my-cluster-storage"
    - name: S3_STORAGE_REGION
      value: "us-west-2"
    - name: S3_STORAGE_ENCRYPTION_KEY
      value: "my-encryption-key"
    - name: USING_EPHEMERAL_DISKS
      value: "true"
    - name: S3_STORAGE_PREFERRED_COPY_SOURCE
      value: "true"
    - name: "KAFKA_SERVERS"
      value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"