Sending data to the cluster

There are two steps for getting data into the cluster:

Using LogScale UI

To send data to the cluster, we will create a new repository, obtain the ingest token, and then configure fluentbit to gather logs from all the pods in our Kubernetes cluster and send them to LogScale.

Create a repo using LogScale's UI:

Click on Add new button and create a new repo

Screenshot for creating a new repository

Create an ingest token:

Go to the test repo you've created and in the settings tab, select Ingest tokens and create a new Ingest token with any available parsers.

Screenshot showing how create a new ingest token
Ingesting logs into the cluster

Now we'll install fluentbit into the Kubernetes cluster and configure the endpoint to point to our $INGRESS_HOSTNAME, and use the $INGEST_TOKEN that was just created. 

helm repo add humio https://humio.github.io/humio-helm-charts
helm repo update

Using a simple text editor, create a file named, humio-agent.yaml and copy the following lines into it: 

humio-fluentbit:
  enabled: true
  humioHostname: <INGRESS_HOSTNAME> # Typically found in Route53 under your Zone
  es:
    tls: true
    port: 443
    inputConfig: |-
      [INPUT]
           Name             tail
           Path             /var/log/containers/*.log
           Parser           docker
           # The path to the DB file must be unique and
           # not conflict with another fluentbit running on the same nodes.
           DB               /var/log/flb_kube.db
           Tag              kube.*
           Refresh_Interval 5
           Mem_Buf_Limit    512MB
           Skip_Long_Lines  On
    resources:
      limits:
        cpu: 100m
        memory: 1024Mi
      requests:
        cpu: 100m
        memory: 512Mi

Install the chart by passing it the generated ingest token and the humio-agent.yaml value file. 

helm install test humio/humio-helm-charts \ --namespace logging \ --set humio-fluentbit.token=$INGEST_TOKEN \ --values humio-agent.yaml

Verify logs are ingested:

  • Go to the LogScale UI and click on the quickstart-cluster-logs repository

  • In the search field, enter kubernetes.container_name = humio-operator and click Run

  • Verify you can see the Humio Operator logs