Elastic Beats
The OSS Elastic Beats are a group of log shippers which are cross-platform, lightweight, and can ship data to a number of tools including Humio.
All Beats are built using the libbeat library. Along with the official Beats, there are a growing number of community Beats.
As of version 6.7.0 of the libbeat only the OSS versions can ship to Humio. The non-OSS Beats check that the server is a licensed elastic server due to this change to the beats client library: "Check license x-pack".
There are currently five official Beats. The Elastic documentation site and Humio's documentation offer resources that describe how to use each of them.
Filebeat — Ships regular log files.
Metricbeat — Ships metrics from your OS and common services.
Packetbeat — Analyzes network packets and common protocols like HTTP.
Winlogbeat — Ships Windows event logs
Heartbeat — Checks system status and availability. See their Getting Started Guide.
Configuration
All Beats are built using the libbeat library and share output configuration. Humio supports parts of the ElasticSearch ingest API, so to send data from Beats to Humio, you use the ElasticSearch output (the documentation is identical for all Beats).
You can use the following elasticsearch
output
configuration template:
output:
elasticsearch:
hosts: ["$YOUR_HUMIO_URL/api/v1/ingest/elastic-bulk"]
username: any-organization
password: $INGEST_TOKEN
Where:
$YOUR_HUMIO_URL variable is the base URL of your Humio server.
$INGEST_TOKEN
variable is the ingest token for your repository.
Starting from version 7.7.0, all beats clients default to using the Elasticsearch index template feature which does not work with Humio. This needs to be disabled in the configuration:
output:
elasticsearch:
setup.template.enabled: false
To optimize performance for the data volumes you want to send, and to
keep shipping latency down, change the default settings for
compression_level
, worker
,
bulk_max_size
and flush_interval
.
Don't raise bulk_max_size
much: 100 – 300 is
the appropriate range. While doing so may increase throughput of ingest
it has a negative impact on search performance of the resulting events
in Humio.
Adding Fields
All Beats also have a fields
section in their
configuration. You can add fields to all events by specifying them in
the fields
section:
fields:
service: user-service
datacenter: dc-a
Fields can be turned into tags by including a @tags
field that lists the names of fields to turn into tags. This applies
to fields both from the fields sections and from the events being
shipped. Refer to Data Sources for
information on tags.
It is possible to specify a global parser that can Ingesting Data to Multiple Repositories.