Elastic Beats

The OSS Elastic Beats are a group of log shippers which are cross-platform, lightweight, and can ship data to a number of tools including LogScale.

All Beats are built using the libbeat library. Along with the official Beats, there are a growing number of community Beats.

As of version 6.7.0 of the libbeat only the OSS versions can ship to LogScale. The non-OSS Beats check that the server is a licensed elastic server due to this change to the beats client library: "Check license x-pack".

There are currently five official Beats. The Elastic documentation site and LogScale's documentation offer resources that describe how to use each of them.

Configuration

All Beats are built using the libbeat library and share output configuration. LogScale supports parts of the ElasticSearch ingest API, so to send data from Beats to LogScale, you use the ElasticSearch output (the documentation is identical for all Beats).

You can use the following elasticsearch output configuration template:

yaml
output:
  elasticsearch:
    hosts: ["$YOUR_LOGSCALE_URL/api/v1/ingest/elastic-bulk"]
    username: any-organization
    password: $INGEST_TOKEN

Where:

  • $YOUR_LOGSCALE_URL variable is the base URL of your LogScale server.

  • $INGEST_TOKEN variable is the ingest token for your repository.

Starting from version 7.7.0, all beats clients default to using the Elasticsearch index template feature which does not work with LogScale. This needs to be disabled in the configuration:

yaml
output:
  elasticsearch:
    setup.template.enabled: false

To optimize performance for the data volumes you want to send, and to keep shipping latency down, change the default settings for compression_level, worker, bulk_max_size and flush_interval. Don't raise bulk_max_size much: 100 – 300 is the appropriate range. While doing so may increase throughput of ingest it has a negative impact on search performance of the resulting events in LogScale.

Adding Fields

All Beats also have a fields section in their configuration. You can add fields to all events by specifying them in the fields section:

yaml
fields:
  service: user-service
  datacenter: dc-a

Fields can be turned into tags by including a @tags field that lists the names of fields to turn into tags. This applies to fields both from the fields sections and from the events being shipped. Refer to LogScale Internal Architecture for information on tags.

It is possible to specify a global parser that can Ingesting Data to Multiple Repositories.