Bucket Storage

When using LogScale in production it is recommended that the instances running LogScale have local NVMe storage. Depending on the environment in which LogScale is being deployed these disks may be ephemeral, such as is the case with AWS's instance-store instances or Google's local SSD's. When utilizing ephemeral instances bucket storage is required for a production environment as it acts as the persistent storage for the cluster.

Kubernetes

The minimum supported Kubernetes version supported by the Humio Operator can be found at Humio Operator Version Matrix

Any Kubernetes platform is supported but implementation and usage of some features may be different.

Kafka

LogScale relies on Kafka as a fault tolerant event bus and internal cluster communication system. The minimum supported version of Kafka can be found here: Kafka Version

In general, we recommend using the latest Kafka version possible on the given environment.

TLS

By default the Humio Operator utilizes cert-manager to create an internal certificate authority for use by the LogScale cluster. In addition, support for provisioning certificates for external connectivity can be used in conjunction with cert-manager's external issuer support. If LogScale is configured to expose its APIs using HTTPS, which is the default, LogScale assumes Kafka connectivity will also use TLS; this is configurable. In some environments that employ service meshes that implement TLS or mTLS, TLS support can be disabled completely.

The components of a Kubernetes cluster will vary depending on the environment but this generally includes building blocks such as DNS, ingress, ingress controllers, networking, and storage classes. Depending on the different components of the Kubernetes cluster, some implementations details may vary and care must be taken during implementation. For example if the environment is using a service mesh that provides TLS connectivity for pods and services, the TLS provisioning feature built into the operator should be disabled.

For a production cluster, an ingress controller that has a routing rule to the humio service is required. The TLS configuration for this controller and for the Humio service will differ if the environment is utilizing an internal PKI implementation or using the built in automation the operator provides with cert-manager.

LogScale is a highly multi-threaded piece of software, and we recommend running a single cluster pod per underlying Kubernetes worker node. This can be achieved using Kubernetes pod anti-affinity. The reason for running one cluster pod per Kubernetes worker node is to let LogScale prioritize what threads gets priority, and to limit disk access so only one cluster pod is able to access the filesystem of a cluster pod. Multiple cluster pods must not use the same data directory, and ensuring they run on separate machines helps achieve that.

All Kubernetes worker nodes should have labels set for the zones they are located in. This should be done using the worker node label topology.kubernetes.io/zone. With this label available on the underlying worker nodes, it means that the LogScale cluster pods will become zone aware and use the zone information e.g. distribute/replicate data. It is important to consider the best strategy for placing worker nodes across zones so the LogScale cluster pods gets scheduled to worker nodes uniformly across multiple availability zones, data centers, or racks.

There's two overall paths for configuring disks for LogScale. One is a combination of ephemeral storage and bucket storage, and the other is to use network attached block storage. We recommend using ephemeral disks and bucket storage for production clusters if bucket storage is available. The operator does not partition, format, and mount the local storage on the worker instance, here is an example of using AWS's user-data script to accomplish this. The recommendation is to not mix instance types/sizes within the same pool of LogScale nodes that sharing the same configuration. It is not recommended to use spot instances as some cloud providers offer.

Ephemeral disk & bucket storage

Worker nodes running LogScale cluster pods must be configured with fast local storage, such as NVMe drives. If there are multiple NVMe drives attached to the machine, they can be combined using RAID-0. Preparing disks for use can be done using features like user data. It is important to configure LogScale to know that the disks are ephemeral by ensuring the environment variables include USING_EPHEMERAL_DISKS=true. This configuration makes LogScale able to make better and safer decisions managing data in the cluster. We can use hostPath mounts to access the locally-attached fast storage from the cluster pods, or potentially any other Kubernetes storage provider which grants direct unrestricted access to the underlying fast storage.

Local PVCs are supported by the Humio operator. This alternative to hostPath does not require the formatting and raiding of multiple disks but does require initial setup that is dependent on the Kubernetes environment.

It is technically possible to run with emptyDir volume types on the cluster pods, but that is not recommended since the lifecycle of emptyDir volumes follows the lifecycle of the pods. Upgrading LogScale clusters or changing configurations, will replace cluster pods which would wipe the data directories for LogScale cluster pods every time, which is not desired. The cluster should try and reuse ephemeral disks as much as possible, even if data is technically safe elsewhere, since performance will take a huge hit every upgrade/restart if data directories are not reused because LogScale would need to fetch everything from bucket storage again.

Network block storage

Using this type of underlying storage for LogScale clusters is generally much slower and more expensive compared to the alternative above, i.e. using the combination of ephemeral disks and bucket storage. The typical use of network block storage for Kubernetes pods will dynamically create disks and attach them to the Kubernetes worker nodes as needed. Network block storage should not be used for I/O heavy LogScale nodes, like storage and digest. However, it can be used for ingest, query coordinators and UI/API nodes.

LogScale requires low latency access to a Kafka cluster to operate optimally, you must have an available Kafka cluster before deploying LogScale.

Non-kafka systems that are similar are not supported, for example Google PubSub or Azure EventHub are not supported.

Sub 50 millisecond ping times from the LogScale pods to the Kafka cluster will ensure data is ingested quickly and be available for search in less than a second.

In its default configuration LogScale will automatically create the Kafka topics and partitions required to operate. This functionality can be disabled by setting the KAFKA_MANAGED_BY_HUMIO value to false. Note that when enabled, LogScale will set up the ingest queue, global, and chatter topics in Kafka using reasonable default settings, but does not edit existing topics to conform to these defaults.

Running Kafka on Kubernetes can be accomplished a variety of ways. The Strimzi Kafka Operator is one such way and uses the same operator patterns that the Humio Operator uses to manage the life-cycles of both Kafka and KRaft nodes. In production setups, LogScale, Kafka, and KRaft should be run on separate worker nodes in the cluster. Both Kafka and KRaft must use persistent volumes provided by the Kubernetes environment, and they should not use ephemeral disks in a production deployment. Kafka brokers and KRaft instances should be deployed on worker nodes in different locations such as racks, data centers, or availability zone to ensure reliability. Kubernetes operators such, as the Strimzi operator, do not create worker nodes and label them, that task is left to the administrators.

For the latest configurations see the LogScale GitHub repository.

For more general information on configuring Strimzi please see their documentation.

The operator overall supports two types of custom resources:

Within the Kubernetes ecosystem it is very common to follow GitOps-style workflows. To help bridge the gap, even for clusters that aren't managed by the humio-operator, there is a HumioExternalCluster CRD. This CRD can be configured with a URL and token to any LogScale cluster, that you may want to manage e.g. alerts using "HumioAlert" resources on some cluster where the cluster pods themselves are not managed by the humio-operator. Since this uses the regular LogScale APIs, this can also be used by customers to manage resources on the LogScale cloud.

The following example provides the configuration for a basic cluster on AWS using ephemeral disks and bucket storage. Access to S3 is handled using IRSA (IAM roles for service accounts) and SSO is handled through a Google Workspace SAML integration (link). This example assumes IRSA and the Google Workspace are configured and can be provided in the configuration below.

Before the cluster is created the operator must be deployed to the Kubernetes cluster and three secrets created. For information regarding the installation of the operator please refer to the Install Humio Operator on Kubernetes.

Prerequisite secrets:

In practice it looks like this:

kubectl create secret --namespace example-clusters generic \
  basic-cluster-1-bucket-storage --from-literal=encryption-key=$(openssl rand -base64 64)
kubectl create secret --namespace example-clusters generic \
  basic-cluster-1-idp-certificate --from-file=idp-certificate.pem=./my-idp-certificate.pem
kubectl create secret --namespace example-clusters generic \
  basic-cluster-1-license --from-literal=data=licenseString

Once the secrets are created the following cluster specification can be applied to the cluster, for details on applying the specification see the operator resources Creating the Resource.

Once applied the HumioCluster resource is created along with many other resources, some of which depend on cert-manager. In the basic cluster example a single node pool is created with three pods performing all tasks.

The overall structure of the Kubernetes resources within a LogScale deployment looks like this:

Figure 13. Kubernetes Installation Cluster Definition

Any configuration setting for LogScale can be used in the cluster specification. For additional configuration options please see the Configuration Variables.

The humio-operator contains one built-in ingress implementation, which relies on ingress-nginx to expose the cluster to the outside of the Kubernetes cluster. The built-in support for ingress-nginx should be seen mostly as a good starting point and source of inspiration, if it does not match certain requirements, it is possible to point alternative ingress controllers to the "Service" resource(s) pointing to the cluster pods. The built-in support for ingress-nginx only works if there is a single node pool with all nodes performing all tasks.

In most managed Kubernetes environment ingress has been integrated with the providers environment to orchestrate the creation of load balancers to load balance traffic to the LogScale service. In AWS when using the AWS Load Balancer Controller add-on (installation documentation) the following ingress object can be created to load balance external traffic to the basic-cluster service that was shown previously.

Basic Traefik IngressRoute example

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: logging
  name: basic-cluster-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /api/v1/status
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:xxxxxx:999999999999:certificate/xxxxxxxxx
 
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: basic-cluster-1
              port:
                number: 8080

Pods running ingress can be scaled dynamically using the Kubernetes Horizontal Pod Autoscaler

Utilizing bucket storage enables the LogScale to operate on ephemeral disks. When an instance is replaced due to maintenance or hardware failure data will be pulled from object storage when a new instance replaces the old one. LogScale natively supports Amazon Bucket Storage and Google Cloud Bucket Storage. S3 compatible systems such as MinIO are supported.

When LogScale determines segment files should be deleted, either by hitting retention settings or someone deleting a repository, the segments get marked for deletion, turning them into "tombstones". While they are marked for deletion, any local copies stored on the LogScale nodes will be deleted, but the copy stored in bucket storage will be kept around for a while before it gets deleted. By default, segments are kept for 7 days in bucket storage before the segment files gets deleted from bucket storage. This means it is possible to undo the deletion of segments as long as it is within 7 days of the segments being marked for deletion.

Configuring any type of quota on the bucket storage system is not recommended. LogScale assumes bucket storage will always be able to keep all the files it is trying to store in it.

It is possible to configure LogScale to expose an endpoint that provides LogScale metrics in a format that is supported by Prometheus, a popular metrics solution in the Kubernetes ecosystem. The most common method is to make sure pods expose the metrics on a defined port, and then configure Prometheus to automatically discover pods that have explicitly marked a certain port to be scraped. If we use the basic cluster example, this is how we would do achieve that:

apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
...
spec:
...
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8401"
...

If LogScale cluster pods are added to a service mesh, there are a couple of items worth highlighting.

As shown above, the humio-operator project takes care of LogScale cluster management, but does not solve the task of shipping logs for Kubernetes containers to a LogScale cluster. To solve that, we have a separate Helm chart located in https://github.com/humio/humio-helm-charts/ which installs a log shipper and sends container logs to the specified LogScale cluster.

The built in Kubernetes resource type HorizontalPodAutoscaler is not supported.

An example for a more complex ingress controller.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: basic-cluster-1-externally-trusted-certificate
  namespace: example-clusters
spec:
  commonName: basic-cluster-1.logscale.local
  secretName: basic-cluster-1-externally-trusted-certificate
  dnsNames:
    - basic-cluster-1.logscale.local
  issuerRef:
    name: letsencrypt-traefik-prod
    kind: ClusterIssuer
---
apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name: basic-cluster-1-transportconfig
  namespace: example-clusters
spec:
  disableHTTP2: true
  insecureSkipVerify: false
  rootCAsSecrets:
  - basic-cluster-1
  serverName: basic-cluster-1.example-clusters
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: basic-cluster-1
  namespace: example-clusters
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`basic-cluster-1.logscale.local`)
      kind: Rule
      services:
        - kind: Service
          name: basic-cluster-1
          port: 8080
          scheme: https
          serversTransport: basic-cluster-1-transportconfig
  tls:
    secretName: basic-cluster-1-externally-trusted-certificate

To scale up a HumioCluster, increase the nodeCount value of the HumioCluster node pool, and the humio-operator will create the additional pods.

At the time of writing, scaling down a HumioCluster is more involved as there's no built in support for carrying this out. Lowering nodeCount value by itself will not immediately start evicting/removing pods.

Overall there's a few different strategies depending on the type/role of node/pod we want to remove.