The following prerequisite compoentns are required before installation:

Before using this guide, the following requirements must be met:

CPU Architecture

As of now, Humio Operator only supports the x86_64/amd64 CPU architecture. Therefore, using Graviton CPU instances in AWS will not work for LogScale.

Bucket Storage

AWS provides NVMe storage in the form of local SSDs, which are directly attached to the virtual machine instances. Local SSDs offer high Input/Output Operations Per Second(IOPS) and low latency. When utilizing ephemeral instances bucket storage is required for a production environment as it acts as the persistent storage for the cluster.

Kubernetes

The minimum Kubernetes version supported by the LogScale Operator, see Version Matrix.

Kafka

LogScale relies on Kafka as a fault tolerant event bus and internal cluster communication system. You must have an available Kafka cluster before deploying LogScale.

AWS MSK is a great option for this, and a provisioning can be done using the provided terraform components code.

TLS

By default the LogScale Operator utilizes cert-manager to create an internal certificate authority for use by the LogScale cluster. In addition, support for provisioning certificates for external connectivity can be used in conjunction with cert-manager's external issuer support. If LogScale is configured to expose its APIs using HTTPS, which is the default, LogScale assumes Kafka connectivity will also utilize TLS, this is configurable. In some environments that employ service meshes that implement TLS or mTLS, TLS support can be disabled completely.

Nginx Ingress

The humio-operator contains one built-in ingress implementation, which relies on ingress-nginx to expose the cluster to the outside of the Kubernetes cluster. The built-in support for ingress-nginx should be seen mostly as a good starting point and source of inspiration, so if it does not match certain requirements it is possible to point alternative ingress controllers to the Service resource(s) pointing to the cluster pods. The built-in support for ingress-nginx only works if there is a single node pool with all nodes performing all tasks.

SSD RAID 0 setup

As long as we use Kafka (MSK) and bucket storage with our LogScale cluster, using RAID 0 on our Kubernetes workers is a safe option for ensuring very fast data processing, as long as Kafka is stable and bucket storage is working. To achieve a RAID0 setup. We're using AWS's i4i instances, coupled with TopoLVM for an easier volume management.

The following guide explains how to quickly set up a LogScale cluster in AWS using the LogScale Operator. We will create the required AWS resources, install the LogScale Operator, and setting up a simple LogScale cluster, all in a couple of easy steps using Terraform. For production installations, it is recommended to follow the full AWS Installation Guide and Reference Architecture and decide how running LogScale fits into your infrastructure.

Private EKS cluster:

Private clusters offer increased isolation by utilizing private IP addresses for nodes and providing private or public endpoints for control plane nodes, which can be further isolated. With private clusters, we can still access AWS APIs through while Pods are segregated from both inbound and outbound communication, establishing a cluster perimeter. The directional flows of these communications can be managed by exposing services through load balancing and a NAT Gateway.

SSM:

The worker nodes can be access via AWS Systems Manager via the AWS management console

Cert Manager and Let's Encrypt issuer:

Cert Manager is a popular Kubernetes tool used for managing the TLS certificates. It is combined with Let's Encrypt issuer to automate the process of obtaining and renewing SSL/TLS certificates for LogScale application. Cert Manager is responsible for certificate management within the EKS cluster, it can generate renew and keep track of TLS certificates. Let's encrypt is a certificate authority that provides TLS/SSL certificates.

NAT Gateway:

To enable outbound internet connectivity for the pods within the private EKS cluster, you can set up a NAT Gateway in your VPC. A NAT Gateway translates the private IP addresses of the pods to public IP addresses, allowing them to access the internet. It's configured for LogScale egress traffic.

Data Flow:

When sizing LogScale, your choice depends on your usage patterns, so we recommend first doing an example setup to see how LogScale works with your data. Examples of recommended deployment configurations are documented in Instance Sizing.

All the specific values for a particular size are described within the cluster_size.tpl file.

There are 3 types of LogScale cluster types:

Selecting the appropriate cluster type is important, as going with a complex type such as Internal-Ingest for a small cluster will significantly increase the costs, while going with the basic type for a large cluster can create performance issues.

The Terraform configuration sets up essential AWS infrastructure and services to support the deployment of LogScale.

The default Terraform backend configuration is using an S3 bucket for state storage, along with a DynamoDB table for state locking, allowing multiple engineers to safely work on the same infrastructure. Workspace support is also used for the S3 bucket state key so before applying the Terraform configuration, make sure to create and switch to a new Terraform Workspace using these commands

Initialize a working directory containing terraform configuration file.

terraform init

Create a new terraform workspace if it does not exist and switch to your new workspace.

terraform workspace new <workspace_name>
terraform workspace select <workspace_name>

You will also be required to supply the LogScale license key as a Terraform environment variable.

export TF_VAR_humiocluster_license="<LogScale License Key>"

Setting up Amazon Web Services

There are two steps to getting the data in, configuring the DNS and accessing the LogScale user interface.

k get ingress -n logging 
NAME                           CLASS  HOSTS   ADDRESS                                                                  PORTS     AGE
cluster_name>-ingest-ingress   none   *       internal-k8s-logging-6a23961160-1639354112.us-west-2.elb.amazonaws.com   80, 443   42h
cluster_name>-ui-ingress       none   *       k8s-logging-0ef6576a8e-517396864.us-west-2.elb.amazonaws.com             80        42h

kubectl get secret <cluster_name>-single-user-password -n logging -o=template --template={{.data.password}} | base64 -D

There are two steps for getting data into the cluster:

Using LogScale UI

To send data to the cluster, we will create a new repository, obtain the ingest token, and then configure fluentbit to gather logs from all the pods in our Kubernetes cluster and send them to LogScale.

Create a repo using LogScale's UI:

Click on Add new button and create a new repo

Create an ingest token:

Go to the test repo you've created and in the settings tab, select Ingest tokens and create a new Ingest token with any available parsers.

Ingesting logs into the cluster

Now we'll install fluentbit into the Kubernetes cluster and configure the endpoint to point to our $INGRESS_HOSTNAME, and use the $INGEST_TOKEN that was just created.

helm repo add humio https://humio.github.io/humio-helm-charts
helm repo update

Using a simple text editor, create a file named, humio-agent.yaml and copy the following lines into it:

humio-fluentbit:
  enabled: true
  humioHostname: <INGRESS_HOSTNAME> # Typically found in Route53 under your Zone
  es:
    tls: true
    port: 443
    inputConfig: |-
      [INPUT]
           Name             tail
           Path             /var/log/containers/*.log
           Parser           docker
           # The path to the DB file must be unique and
           # not conflict with another fluentbit running on the same nodes.
           DB               /var/log/flb_kube.db
           Tag              kube.*
           Refresh_Interval 5
           Mem_Buf_Limit    512MB
           Skip_Long_Lines  On
    resources:
      limits:
        cpu: 100m
        memory: 1024Mi
      requests:
        cpu: 100m
        memory: 512Mi

Install the chart by passing it the generated ingest token and the humio-agent.yaml value file.

helm install test humio/humio-helm-charts \ --namespace logging \ --set humio-fluentbit.token=$INGEST_TOKEN \ --values humio-agent.yaml

Verify logs are ingested:

• Go to the LogScale UI and click on the quickstart-cluster-logs repository

• In the search field, enter kubernetes.container_name = humio-operator and click Run

• Verify you can see the LogScale Operator logs

AWS Cloud Reference Architecture