Kubernetes Log Format

This guide describes how to send logs from your Kubernetes cluster to your Humio cluster. If you're looking to install Humio on Kubernetes, see our Installation Guide.

When it comes to managing micro-services in a Kubernetes cluster, Humio is a great way to get insights into your applications. While other log shippers are supported, we focus on using Fluent Bit for forwarding log messages to Humio.

Helm

Take advantage of Humio with your Kubernetes setup.

We'll start with Helm, the Kubernetes package manager. Directions for installing Helm for your particular OS are on the Helm GitHub page.

Once you've installed Helm, update the main Helm chart repository. This main repository contains subcharts for Humio.

humio
helm repo add humio https://humio.github.io/humio-helm-charts
helm repo update

Next, create a file named humio-agent.yaml with the following content:

humio
humio-fluentbit:
  enabled: true
  humioHostname: $YOUR_HUMIO_URL
  es:
    tls: true

If using OpenShift, it is necessary to create the following humio-agent.yaml with the extra fields which will create a SecurityContextConstraints resource as well as run fluent-bit in a privileged securityContext:

humio
humio-fluentbit:
  enabled: true
  humioHostname: $YOUR_HUMIO_URL
  es:
    tls: true
  securityContext:
    privileged: true
  scc:
    enabled: true

If using CRI rather than docker, it is necessary to create the following humio-agent.yaml with an inputConfig and parserConfig so that logs are correctly parsed by fluentbit.

humio
humio-fluentbit:
  enabled: true
  humioHostname: $YOUR_HUMIO_URL
  es:
    tls: true
  parserConfig: |-
    [PARSER]
      Name cri
      Format regex
      Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
      Time_Key time
      Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  inputConfig: |-
    [INPUT]
      Name             tail
      Path             /var/log/containers/*.log
      Parser           cri
      Tag              kube.*
      Refresh_Interval 5
      Mem_Buf_Limit    256MB
      Skip_Long_Lines  On

Replace $YOUR_HUMIO_URL with the hostname of your Humio installation, not including the port number. E.g. $YOUR_HUMIO_URL may become cloud.humio.com. Don't forget to enable the ELASTIC_PORT property. Take your ingest token from your Humio Repository page.

Token

Figure 221. Token


-- and replace $INGEST_TOKEN with the ingest token when running the install command below. We recommend running in a separate namespace, in our example the logging namespace.

humio
# Helm v3+
helm install humio humio/humio-helm-charts \
  --namespace logging \
  --set humio-fluentbit.token=$INGEST_TOKEN \
  --values humio-agent.yml

# Helm v2
helm install humio/humio-helm-charts --name humio \
  --namespace logging \
  --set humio-fluentbit.token=$INGEST_TOKEN \
  --values humio-agent.yaml

Once this is in place, the Kubernetes logs should be flowing into the Humio cluster.

Parsing Logs

Humio will look for the pod label humio-parser and use that parser on the log line. That way you can control how Humio parses the data by configuring your pods. If the humio-parser label is not set, no parser will be applied to the log lines.

Additional Filters

In some cases you might want to make some changes to the Fluent Bit configuration. The easiest way to do that is by updating the values.yaml with the filter and updating the Helm chart.

First, update the values.yaml by adding a customFluentBitConfig section containing custom Fluent Bit configurations:

ini files
humio-fluentbit:
  enabled: true
  humioHostname: $YOUR_HUMIO_URL
  es:
    tls: true
  customFluentBitConfig:
    custom-filter-kubernetes.conf: |-
      [FILTER]
       Name     modify
       Match    *
       Rename   log rawstring

Then upgrade the helm chart

humio
helm upgrade humio humio/humio-helm-charts \
  --values humio-agent.yaml

Uninstalling

humio
helm delete --purge humio

And optionally clean up the namespace

humio
kubectl delete namespace logging --cascade=true

For more information see Advanced log routing with Fluent Bit 1.4.