LogScale Operator Resource Management

The LogScale Operator manages a number of LogScale components such as Repositories, Parsers and Ingest Tokens.

After installing the Operator by following the Operator Installation Guide, a LogScale Cluster resource (referred to as HumioCluster) can be created along with a number of other resource types.

Creating the Resource

Any of the resources can be created by applying the yaml via kubectl. First, create a resource.yaml file with the desired content, and then run:

shell
$ kubectl create -f ./resource.yaml

The content of the yaml file may contain any number of resources. The full list of resources and examples are below.

HumioCluster

A HumioCluster resource tells the LogScale Operator to create a LogScale Cluster. Multiple creations of HumioCluster may be done and managed by the Operator.

The content of the yaml file will depend on how the LogScale Cluster should be configured to run. The next parts of this document explain some common cluster configurations.

Ephemeral with S3 Storage

A highly recommended LogScale Cluster configuration is to run in ephemeral mode, using S3 for persistent storage. The LogScale pods will be configured with hostPath, which mounts a directory from the host machine to the pod which is local storage, and ideally made up of NVME SSDs. This configuration also has fairly high resource limits and affinity policies that ensure no two LogScale pods may be scheduled on the same host. This is an ideal storage configuration for production workloads running in AWS.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: "humio/humio-core:1.36.0"
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  resources:
    limits:
      cpu: "8"
      memory: 56Gi
    requests:
      cpu: "6"
      memory: 52Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: humio_node_type
                operator: In
                values:
                  - core
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                  - amd64
          - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                  - linux
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - humio
          topologyKey: kubernetes.io/hostname
  dataVolumeSource:
    hostPath:
      path: "/mnt/disks/vol1"
      type: "Directory"
  environmentVariables:
    - name: S3_STORAGE_BUCKET
      value: "my-cluster-storage"
    - name: S3_STORAGE_REGION
      value: "us-west-2"
    - name: S3_STORAGE_ENCRYPTION_KEY
      value: "my-encryption-key"
    - name: USING_EPHEMERAL_DISKS
      value: "true"
    - name: S3_STORAGE_PREFERRED_COPY_SOURCE
      value: "true"
    - name: "ZOOKEEPER_URL"
      value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
    - name: "KAFKA_SERVERS"
      value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
Ephemeral with GCS Storage

A highly recommended LogScale Cluster configuration is to run in ephemeral mode, using GCS for persistent storage. The LogScale pods will be configured with hostPath, which mounts a directory from the host machine to the pod which is local storage, and ideally made up of NVME SSDs. This configuration also has fairly high resource limits and affinity policies that ensure no two LogScale pods may be scheduled on the same host. This is an ideal storage configuration production workloads running in GCP.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: humio-core:dynxml_current_release
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  resources:
    limits:
      cpu: "8"
      memory: 56Gi
    requests:
      cpu: "6"
      memory: 52Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: humio_node_type
                operator: In
                values:
                  - core
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                  - amd64
          - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                  - linux
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values:
                  - humio-core
          topologyKey: kubernetes.io/hostname
  dataVolumeSource:
    hostPath:
      path: "/mnt/disks/vol1"
      type: "Directory"
  extraHumioVolumeMounts:
    - name: gcp-storage-account-json-file
      mountPath: /var/lib/humio/gcp-storage-account-json-file
      subPath: gcp-storage-account-json-file
      readOnly: true
  extraVolumes:
    - name: gcp-storage-account-json-file
      secret:
        secretName: gcp-storage-account-json-file
  environmentVariables:
    - name: GCP_STORAGE_ACCOUNT_JSON_FILE
      value: "/var/lib/humio/gcp-storage-account-json-file"
    - name: GCP_STORAGE_BUCKET
      value: "my-cluster-storage"
    - name: GCP_STORAGE_ENCRYPTION_KEY
      value: "my-encryption-key"
    - name: USING_EPHEMERAL_DISKS
      value: "true"
    - name: "ZOOKEEPER_URL"
      value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
    - name: "KAFKA_SERVERS"
      value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
Nginx Ingress with Cert Manager

Configuring Ingress with Cert Manager will ensure you have an Ingress resource that can be used to access the cluster, along with a valid cert provided by Cert Manager.

Note

Ingress is currently not supported for Node Pools.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: "humio/humio-core:1.36.0"
  environmentVariables:
    - name: "ZOOKEEPER_URL"
      value: "humio-cp-zookeeper-0.humio-cp-zookeeper-headless:2181"
    - name: "KAFKA_SERVERS"
      value: "humio-cp-kafka-0.humio-cp-kafka-headless:9092"
  hostname: "humio.example.com"
  esHostname: "humio-es.example.com"
  ingress:
    enabled: true
    controller: nginx
    annotations:
      use-http01-solver: "true"
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
Nginx Ingress with Custom Path

If the case where you want to run LogScale under a custom path.

Note

Ingress is currently not supported for Node Pools.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: "humio/humio-core:1.36.0"
  environmentVariables:
    - name: "ZOOKEEPER_URL"
      value: "humio-cp-zookeeper-0.humio-cp-zookeeper-headless:2181"
    - name: "KAFKA_SERVERS"
      value: "humio-cp-kafka-0.humio-cp-kafka-headless:9092"
  hostname: "humio.example.com"
  esHostname: "humio-es.example.com"
  path: /logs
  ingress:
    enabled: true
    controller: nginx
HumioCluster with Hostname References

In the case where spec.hostname and/or spec.esHostname cannot be managed in the HumioCluster resource, it's possible to use a reference to an external source for either. Currently secretKeyRef is supported.

To use secretKeyRef containing the hostname, create the secret as shown below, where <hostname> is the hostname for the HumioCluster:

shell
$ kubectl create secret generic <cluster-name>-hostname --from-literal=data=<hostname> -n <namespace>

You would then update the HumioCluster resource to use the hostname reference:

yaml
spec:
  hostnameSource:
    secretKeyRef:
      name: <cluster-name>-hostname
      key: data

To use ``secretKeyRef`` containing the ``esHostname``, create the secret as shown below, where ``<es-hostname>`` is the es-hostname for the HumioCluster:
shell
$ kubectl create secret generic <cluster-name>-es-hostname --from-literal=data=<es-hostname> -n <namespace>

You would then update the HumioCluster resource to use the esHostname reference:

yaml
spec:
  esHostnameSource:
    secretKeyRef:
      name: <cluster-name>-es-hostname
      key: data
Persistent Volumes

It's possible to use Persistent Volumes as the backing store for LogScale data. This can be used as an alternative to Bucket Storage, however, Persistent Volumes using network block storage are significantly slower than local disks and LogScale will not perform well on this medium. For using Persistent Volumes using local storage, see Local Persistent Volumes (Beta).

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: "humio/humio-core:1.36.0"
  license:
     secretKeyRef:
       # Secret must be created with the following command: kubectl create secret generic example-humiocluster-license --from-literal=data=<license>
       name: example-humiocluster-license
       key: data
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  resources:
    limits:
      cpu: "8"
      memory: 56Gi
      requests:
      cpu: "6"
      memory: 52Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: humio_node_type
            operator: In
            values:
            - core
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - humio-core
        topologyKey: kubernetes.io/hostname
  dataVolumePersistentVolumeClaimSpecTemplate:
    storageClassName: standard
    accessModes: [ReadWriteOnce]
    resources:
      requests:
        storage: 500Gi
  environmentVariables:
    - name: "ZOOKEEPER_URL"
      value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
    - name: "KAFKA_SERVERS"
      value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
Local Persistent Volumes (Beta)

It's possible to use local Persistent Volumes as the backing store for LogScale data, using something like Local Persistent Volumes. This can be used in combination with Bucket Storage, as the operator can be configured to clean up local Persistent Volumes that are attached to a node when that node is removed from the Kubernetes cluster. For this reason, it is extremely important to use USING_EPHEMERAL_DISKS=true along with Bucket Storage when using this option. This cleanup setting is enabled in the following example by setting dataVolumePersistentVolumeClaimPolicy.reclaimType=OnNodeDelete.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  image: "humio/humio-core:1.36.0"
  license:
     secretKeyRef:
       # Secret must be created with the following command: kubectl create secret generic example-humiocluster-license --from-literal=data=<license>
       name: example-humiocluster-license
       key: data
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  resources:
    limits:
      cpu: "8"
      memory: 56Gi
      requests:
      cpu: "6"
      memory: 52Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: humio_node_type
            operator: In
            values:
            - core
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - humio-core
        topologyKey: kubernetes.io/hostname
  dataVolumePersistentVolumeClaimSpecTemplate:
    storageClassName: local-storage
    accessModes: [ReadWriteOnce]
    resources:
      requests:
        storage: 500Gi
  dataVolumePersistentVolumeClaimPolicy:
    reclaimType: OnNodeDelete
  environmentVariables:
    - name: S3_STORAGE_BUCKET
      value: "my-cluster-storage"
    - name: S3_STORAGE_REGION
      value: "us-west-2"
    - name: S3_STORAGE_ENCRYPTION_KEY
      value: "my-encryption-key"
    - name: USING_EPHEMERAL_DISKS
      value: "true"
    - name: S3_STORAGE_PREFERRED_COPY_SOURCE
      value: "true"
    - name: "ZOOKEEPER_URL"
      value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
    - name: "KAFKA_SERVERS"
      value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
Node Pools (Beta)

Multiple groups of LogScale nodes may be run as part of the HumioCluster. An example of this may include a node pool for ingest-only nodes and a node pool for digest and storage nodes.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  targetReplicationFactor: 2
  storagePartitionsCount: 24
  digestPartitionsCount: 24
  license:
    secretKeyRef:
      name: example-humiocluster-license
      key: data
  nodePools:
    - name: digest-storage
      spec:
        image: "humio/humio-core:1.36.0"
        nodeCount: 3
        dataVolumeSource:
          hostPath:
            path: "/mnt/disks/vol1"
            type: "Directory"
        resources:
          limits:
            cpu: "8"
            memory: 56Gi
          requests:
            cpu: "6"
            memory: 52Gi
        environmentVariables:
          - name: "ZOOKEEPER_URL"
            value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
          - name: "KAFKA_SERVERS"
            value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
    - name: ingest-only
      spec:
        image: "humio/humio-core:1.36.0"
        nodeCount: 3
        dataVolumePersistentVolumeClaimSpecTemplate:
          storageClassName: standard
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 10Gi
        resources:
          limits:
            cpu: "8"
            memory: 56Gi
          requests:
            cpu: "6"
            memory: 52Gi
        environmentVariables:
          - name: NODE_ROLES
            value: "httponly"
          - name: "ZOOKEEPER_URL"
            value: "z-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181,z-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:2181"
          - name: "KAFKA_SERVERS"
            value: "b-2-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-1-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092,b-3-my-zookeeper.c4.kafka.us-west-2.amazonaws.com:9092"
Configuring TLS with Operator

By default, TLS is enabled on each LogScale pod. This is recommended, however, in some cases you may want TLS to be disabled. To do this, use the below configuration.

If TLS is enabled here, it is assumed that TLS is also used for the connection to Kafka. If TLS on the LogScale pods is disabled but the connection to Kafka should use SSL, then Kafka will need to be configured explicitly to use SSL.

yaml
spec:
  tls:
    enabled: false
Additional Kafka Configuration with Operator

Extra Kafka configs can be set and used by the LogScale pods. This is mainly used to toggle TLS when communicating with Kafka. To enable TLS for example, set the configuration below.

SSL is enabled by default when using TLS for the LogScale pods. See Configuring TLS with Operator.

yaml
spec:
  extraKafkaConfigs: "security.protocol=SSL"
ZooKeeper Deployment in Kubernetes and Operator

When TLS is enabled for LogScale, TLS is by default also enabled for connections to ZooKeeper. In some cases, such as with MSK, TLS will be enabled for the Kafka brokers but not for ZooKeeper. To disable TLS for ZooKeeper, set the following in values for the HUMIO_OPTS environment variable: -Dzookeeper.client.secure=false.

Authentication - SAML with Operator

When using SAML, it's necessary to follow the SAML Authentication and once the IDP certificate is obtained, you must create a secret containing that certificate using kubectl.

shell
$ kubectl create secret generic <cluster-name>-idp-certificate --from-file=idp-certificate.pem=./my-idp-certificate.pem -n <namespace>

Once the secret has been created, a configuration similar to below can be added to enable SAML, adjusting for your cluster URL and IDP token.

yaml
spec:
  environmentVariables:
    - name: AUTHENTICATION_METHOD
      value: saml
    - name: AUTO_CREATE_USER_ON_SUCCESSFUL_LOGIN
      value: "true"
    - name: PUBLIC_URL
      value: https://my-cluster.example.com
    - name: SAML_IDP_SIGN_ON_URL
      value: https://accounts.google.com/o/saml2/idp?idpid=idptoken
    - name: SAML_IDP_ENTITY_ID
      value: https://accounts.google.com/o/saml2/idp?idpid=idptoken
Authentication - Single User

If running LogScale in single user mode, you will need to set a password for the user user. This can be done via a plain text environment variable or using a Kubernetes secret that is referenced by an environment variable. If supplying a secret, you must populate this secret prior to creating the HumioCluster resource otherwise the pods will fail to start.

By setting a password using an environment variable plain text value:

yaml
spec:
  environmentVariables:
    - name: "SINGLE_USER_PASSWORD"
      value: "MyVeryS3cretPassword"

By setting a password using an environment variable secret reference:

yaml
spec:
  environmentVariables:
    - name: "SINGLE_USER_PASSWORD"
      valueFrom:
        secretKeyRef:
          name: developer-user-password
          key: password
License Management with Operator

LogScale licenses can be managed with the operator. In order to do so, a Kubernetes secret must be created which contains the value of the license. First create the secret as shown below, where <license> is the license content that is obtained from LogScale:

shell
$ kubectl create secret generic <cluster-name>-license --from-literal=data=<license> -n <namespace>

And then update the HumioCluster resource to use the secret reference:

yaml
spec:
  license:
    secretKeyRef:
      name: <cluster-name>-license
      key: data
Update Strategy

HumioCluster resources may be configured with an Update Strategy. The updateStrategy.type controls how the operator restarts LogScale pods in response to a image change to the HumioCluster spec or nodePools spec. The available values for type are: OnDelete, RollingUpdate, ReplaceAllOnUpdate, and RollingUpdateBestEffort.

  • ReplaceAllOnUpdate

    All LogScale pods will be replaced at the same time during an update. Pods will still be replaced one at a time when there are other configuration changes such as updates to pod environment variables. This is the default behavior.

  • OnDelete

    No LogScale pods will be terminated but new pods will be created with the new spec. Replacing existing pods will require each pod to be deleted by the user.

  • RollingUpdate

    Pods will always be replaced one pod at a time. There may be some LogScale updates where rolling updates are not supported, so it is not recommended to have this set all the time.

  • RollingUpdateBestEffort

    The operator will evaluate the LogScale version change and determine if the LogScale pods can be updated in a rolling fashion or if they must be replaced at the same time.

yaml
spec:
  updateStrategy:
    type: ReplaceAllOnUpdate
Custom Service Accounts

ServiceAccount resources may be created prior to creating the HumioCluster resource and then the HumioCluster may be configured to use them rather than relying on the LogScale Operator to create and manage the ServiceAccounts and bindings.

These can be configured for the initServiceAccountName, authServiceAccountName and serviceAccountName fields in the HumioCluster resource. They may be configured to use a shared ServiceAccount or separate ServiceAccounts. It is recommended to keep these separate unless otherwise required.

Shared Service Account

In the following example, we configure all three to share the same ServiceAccount.

To do this, first create the ServiceAccount, ClusterRole, ClusterRoleBinding, Role and RoleBinding, adjusting the namespace to be where the HumioCluster resource is created.

yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: humio
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: humio
  namespace: default
rules:
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: humio
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: humio
subjects:
  - kind: ServiceAccount
    name: humio
    namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: humio
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: humio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: humio
subjects:
  - kind: ServiceAccount
    name: humio
    namespace: default

Now include the following in the HumioCluster resource so it will use the shared ServiceAccount:

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  humioServiceAccountName: humio
  initServiceAccountName: humio
  authServiceAccountName: humio
Separate Service Accounts

In the following example, we configure all three to use different ServiceAccount resources.

To do this, create the ServiceAccount, ClusterRole, ClusterRoleBinding for the initServiceAccount:

yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: humio-init
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: humio-init
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: humio-init
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: humio-init
subjects:
  - kind: ServiceAccount
    name: humio-init
    namespace: default

Followed by the ServiceAccount, Role, RoleBinding for the authServiceAccount:

yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: humio-auth
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: humio-auth
  namespace: default
rules:
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: humio-auth
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: humio-auth
subjects:
  - kind: ServiceAccount
    name: humio-auth
    namespace: default

And finally the ServiceAccount for the main LogScale container.

Note

Ensure to configure the appropriate annotations if using IRSA.

yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: humio
  namespace: default

Now include the following in the HumioCluster resource so it will use the ServiceAccounts:

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
  name: example-humiocluster
spec:
  humioServiceAccountName: humio
  initServiceAccountName: humio-init
  authServiceAccountName: humio-auth
HumioRepository

A HumioRepository resource tells the LogScale Operator to create a LogScale Repository. Any number of HumioRepository resources may be created and managed by the Operator.

The content of the yaml file will depend on how the LogScale Repository should be configured. The following shows an example HumioRepository resource.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioRepository
metadata:
  name: example-humiorepository
  namespace: logging
spec:
  managedClusterName: example-humiocluster
  name: example-humiorepository
  description: "Example LogScale Repository"
  retention:
    timeInDays: 30
    ingestSizeInGB: 50
    storageSizeInGB: 10
HumioParser

A HumioParser resource tells the LogScale Operator to create a LogScale Parser. Any number of HumioParser resources may be created and managed by the Operator.

The content of the yaml file will depend on how the LogScale Parser should be configured. The following shows an example of HumioParser resource.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioParser
metadata:
  name: example-humioparser
  namespace: logging
spec:
  managedClusterName: example-humiocluster
  name: example-humioparser
  repositoryName: example-humiorepository
  parserScript: |
    case {
      kubernetes.pod_name=/fluentbit/
        | /\[(?<@timestamp>[^\]]+)\]/
        | /^(?<@timestamp>.*)\[warn\].*/
        | parseTimestamp(format="yyyy/MM/dd' 'HH:mm:ss", field=@timestamp);
      parseJson();
      * | kvParse()
    }
EOF
HumioIngestToken

A HumioIngestToken resource tells the LogScale Operator to create a LogScale Ingest Token. Any number of HumioIngestToken resources may be created and managed by the Operator.

The content of the yaml file will depend on how the LogScale Ingest Token should be configured. The following shows an example HumioIngestToken resource.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioIngestToken
metadata:
  name: example-humioingesttoken
  namespace: logging
spec:
  managedClusterName: example-humiocluster
  name: example-humioingesttoken
  repositoryName: example-humiorepository
  parserName: example-humioparser
  tokenSecretName: example-humioingesttoken-token

By specifying tokenSecretName, the LogScale Operator will export the token as this secret name in Kubernetes. If you do not wish to have the token exported, omit this field from the spec.

HumioAlert

A HumioAlert resource tells the LogScale Operator to create a LogScale Alert. Any number of HumioAlert resources may be created and managed by the Operator.

The content of the yaml file will depend on how the LogScale Alert should be configured. The following shows an example of a HumioAlert resource.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAlert
metadata:
  name: example-alert
spec:
  managedClusterName: example-humiocluster
  name: example-alert
  viewName: humio
  query:
    queryString: "#repo = humio | error = true | count() | _count > 0"
    start: 24h
    end: now
    isLive: true
  throttleTimeMillis: 60000
  silenced: false
  description: Error counts
  actions:
    - example-email-action
HumioFilterAlert

A HumioFilterAlert resource tells the LogScale Operator to create a LogScale FilterAlert. Any number of HumioFilterAlert resources may be created and managed by the Operator.

The content of the yaml file will depend on how the LogScale FilterAlert should be configured. The following shows an example of a HumioFilterAlert resource.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioFilterAlert
metadata:
  name: example-filter-alert
spec:
  managedClusterName: example-humiocluster
  name: example-filter-alert
  viewName: humio
  queryString: "#repo = humio | error = true"
  throttleTimeSeconds: 3600
  throttleField: some-field
  enabled: true
  description: Error counts
  actions:
    - example-email-action
HumioAggregateAlert

A HumioAggregateAlert resource tells the LogScale Operator to create a LogScale AggregateAlert. Any number of HumioAggregateAlert resources may be created and managed by the Operator.

The content of the yaml file will depend on how the LogScale AggregateAlert should be configured. The following shows an example of a HumioAggregateAlert resource.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAggregateAlert
metadata:
  name: example-aggregate-alert
spec:
  managedClusterName: example-humiocluster
  name: example-aggregate-alert
  viewName: humio
  queryString: "#repo = humio | error = true | count()"
  queryTimestampType: "EventTimestamp"
  throttleTimeSeconds: 300
  triggerMode: "CompleteMode"
  searchIntervalSeconds: 3600
  throttleField: "@timestamp"
  description: "This is an example of an aggregate alert"
  enabled: true
  actions:
    - example-email-action
HumioScheduledSearch

A HumioScheduledSearch resource tells the LogScale Operator to create a LogScale ScheduledSearch. Any number of HumioScheduledSearch resources may be created and managed by the Operator.

The content of the yaml file will depend on how the LogScale ScheduledSearch should be configured. The following shows an example of a HumioScheduledSearch resource.

yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioScheduledSearch
metadata:
  name: example-scheduled-search
spec:
  managedClusterName: example-humiocluster
  name: example-scheduled-search
  viewName: humio
  queryString: "#repo = humio | error = true | count()"
  queryStart: "1d"
  queryEnd: "now"
  schedule: "0 0 * * *"
  timeZone: "UTC"
  backfillLimit: 3
  enabled: true
  description: Error counts
  actions:
    - example-email-action
HumioAction

A HumioAction resource tells the LogScale Operator to create a LogScale Action. Any number of HumioAction resources may be created and managed by the Operator.

The content of the yaml file will depend on how the LogScale Action should be configured. The following shows a examples of different types of HumioAction resources.

Email Action
yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAction
metadata:
  name: example-email-action
spec:
  managedClusterName: example-humiocluster
  name: example-email-action
  viewName: humio
  emailProperties:
    recipients:
      - example@example.com
    subjectTemplate: "{alert_name} has alerted"
    bodyTemplate: |-
      {alert_name} has alerted
      click {url} to see the alert
HumioRepository Action
yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAction
metadata:
  name: humio-humio-repository-action
spec:
  managedClusterName: example-humiocluster
  name: example-humio-repository-action
  viewName: humio
  humioRepositoryProperties:
    ingestToken: some-humio-ingest-token
OpsGenie Action
yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAction
metadata:
  name: example-humioaction
spec:
  managedClusterName: example-humiocluster
  name: example-ops-genie-action
  viewName: humio
  opsGenieProperties:
    genieKey: "some-genie-key"
PagerDuty Action
yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAction
metadata:
  name: humio-pagerduty-action
spec:
  managedClusterName: example-humiocluster
  name: example-pagerduty-action
  viewName: humio
  pagerDutyProperties:
    routingKey: some-routing-key
    severity: critical
Slack Post Message Action
yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAction
metadata:
  name: humio-slack-post-message-action
spec:
  managedClusterName: example-humiocluster
  name: example-slack-post-message-action
  viewName: humio
  slackPostMessageProperties:
    apiToken: some-oauth-token
    channels:
      - "#some-channel"
      - "#some-other-channel"
    fields:
      query: "{query}"
      time-interval: "{query_time_interval}"
Slack Action
yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAction
metadata:
  name: humio-slack-action
spec:
  managedClusterName: example-humiocluster
  name: example-slack-action
  viewName: humio
  slackProperties:
    url: "https://hooks.slack.com/services/T00000000/B00000000/YYYYYYYYYYYYYYYYYYYYYYYY"
    fields:
      query: "{query}"
      time-interval: "{query_time_interval}"
VictorOps Action
yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAction
metadata:
  name: humio-victor-ops-action
spec:
  managedClusterName: example-humiocluster
  name: example-victor-ops-action
  viewName: humio
  victorOpsProperties:
    messageType: critical
    notifyUrl: "https://alert.victorops.com/integrations/0000/alert/0000/routing_key"
Webhook Action
yaml
apiVersion: core.humio.com/v1alpha1
kind: HumioAction
metadata:
  name: humio-web-hook-action-managed
spec:
  managedClusterName: example-humiocluster
  name: example-web-hook-action
  viewName: humio
  webhookProperties:
    url: "https://example.com/some/api"
    headers:
      some: header
      some-other: header
    method: POST
    bodyTemplate: |-
      {alert_name} has alerted
      click {url} to see the alert