You must make the following changes to the configuration, see Configuration.

sudo systemctl restart filebeat

An important next step is choosing a Parsing Data for your Filebeat events.

The following example is a simple 8.1.0 Filebeat configuration that sends data to LogScale, assuming the standard LogScale API is hosted on port 8080 and the elasticsearch API is available on port 9200:

filebeat.inputs:
  - paths:
      - /var/log/apache/*.log
    encoding: utf-8
    fields:
      aField: value

queue.mem:
  events: 8000
  flush.min_events: 1000
  flush.timeout: 1s

output:
  elasticsearch:
    hosts: ["https://cloud.humio.com:8080/api/v1/ingest/elastic-bulk"]
    username: anything
    password: 750y0940-ec68-4889-9e3a-e7e78d5536er
    compression_level: 5
    bulk_max_size: 200
    worker: 5

setup.ilm.enabled: false

output.elasticsearch.allow_older_versions: true

The Filebeat configuration file is located at /etc/filebeat/filebeat.yml on Linux.

The section only aims to document the set of keys and value required to ship data to LogScale and therefore not all of the configuration options which are available in Filebeat are listed.

The source block configures the sources of data that will be sent to LogScale.

This object specifies all the configuration related to the output of the log shipper and where the data is sent.

Add the following to your filebeat.yml configuration file to enable debug logging:

logging:
  level: debug
  to_files: true
  to_syslog: false
  files:
    path: /var/log/filebeat
    name: filebeat.log
    keepfiles: 3

We do not recommend that you use the JSON parsing built into Filebeat. Instead, LogScale has it's own JSON support. Filebeat processes logs line by line, so JSON parsing will only work if there is one JSON object per line. By using LogScale's json you can get JSON fields extracted during ingest. You can also Example: Parsing JSON to get more control over the fields that are created.

It's possible to add fields with static values using the fields section. These fields will be added to each event.

Filebeat automatically sends the host (beat.hostname) and filename (source) along with the data. LogScale adds these fields to each event. The fields are added as @host and @source in order to not collide with other fields in the event.

To avoid having the @host and @source fields, specify @host and @source in the fields section with an empty value.

LogScale saves data in Data Sources. You can provide a set of Tags to specify which Data Source the data is saved in. See Event Tags for more information about tags and Data Sources.

If a type is configured in Filebeat it's always used as tag. Other fields can be used as tags by defining the fields as tagFields in the Parsing Data pointed to by the type. In LogScale tags always start with a #. When turning a field into a tag, the name of the field will be prepended with #.

By default, the Filebeat handling in LogScale keeps only a subset of the fields shipped by Filebeat since the default handling targets just getting the message from the input files into LogScale as @rawstring, not all the extra fields that Filebeat may add. If you want to get the full set of fields, for instance if you are using Processors in the Filebeat configuration, then turn off the default handling by adding these lines to you Filebeat configuration:

# Skip default Filebeat field handling in LogScale by
# not including the word ``filebeat`` in the index name.
# The parser then gets all fields added by Filebeat.

setup.template.name: "beat"
setup.template.pattern: "beat"
output.elasticsearch.index: "beat"

By default, Filebeat creates one event for each line in the in a file. However, you can also split events in different ways. For example, stack traces in many programming languages span multiple lines.

You can specify multiline settings in the Filebeat configuration. See Filebeat's multiline configuration documentation.

Often a log event starts with a timestamp, and we want to read all lines until we see a new line starting with a timestamp. In Filebeat that can be done like this:

multiline.pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
multiline.negate: true
multiline.match: after

The multiline.pattern should match your timestamp format.

Below is an example of all of this. The $YOUR_LOGSCALE_URL variable is the URL for your installation.

filebeat:
  inputs:
    - paths:
        - /var/log/nginx/access.log
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      fields:
        aField: value
    - paths:
        - humio_std_out.log
      fields:
        service: humio
      multiline:
        pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
        negate: true
       match: after

queue.mem:
  events: 8000
  flush.min_events: 1000
  flush.timeout: 1s

output:
  elasticsearch:
    hosts: ["https://$YOUR_LOGSCALE_URL:8080/api/v1/ingest/elastic-bulk"]
    username: from-me
    password: "some-ingest-token"
    compression_level: 5
    bulk_max_size: 200
    worker: 1

logging:
  level: info
  to_files: true
  to_syslog: false
  files:
    path: /var/log/filebeat
    name: filebeat.log
    keepfiles: 3