Logstash

Important

Logstash version 7.13 and later are known to be incompatible with LogScale when using the Elasticsearch output plugin, you may receive the error Could connect to a compatible version of Elasticsearch Please upgrade to one of the two connection options detailed on this page.

Logstash is an established open source tool for collecting logs, parsing them, and outputting them to other systems. You can use Logstash alongside LogScale to process and analyze logs together. In this scenario, you use Logstash as the log collection and parsing agent and instruct it to send the data to LogScale.

LogScale supports the Elasticsearch bulk insertion API, but due to recent changes by Elastic the Logstash Elasticsearch output plugin is no longer compatible. The best option for sending data to LogScale is to use the HTTP output option (built-in) and send the data to the Splunk HEC ingest API, or to maintain the original Elasticsearch output features use the OpenSearch output plugin (not built-in).

Installation & Configuration

To download Logstash visit the OSS Logstash downloads page.

You can find the complete documentation for Logstash at the Logstash website.

OpenSearch Output

The OpenSearch project has continued the development of the open source elasticsearch output plugin. This is not a builtin plugin for LogStash but can be easily installed.

Important

Only OpenSearch Logstash Output Plugin version 1.1.0 and above are known to be compatible with LogScale. 1.0.0 does not work.

For installation instructions for the plugin please reference the OpenSearch documentation, or in your Logstash bin directory you can run:

bash
./logstash-plugin install logstash-output-opensearch

The following example shows a very simple Logstash configuration that sends data to LogScale:

graphql
input{
  exec{
    command => "date"
    interval => "5"
  }
}

output{
  opensearch{
    hosts => ["$YOUR_LOGSCALE_URL:443/api/v1/ingest/elastic-bulk"]
    user => "$HUMIO_REPO_NAME"
    password => "$INGEST_TOKEN"
    ssl => true
    ssl_certificate_verification => true
    manage_template => false
    http_compression => true
  }
}

The $YOUR_LOGSCALE_URL variable is the base URL of your LogScale server ($YOUR_LOGSCALE_URL). The $INGEST_TOKEN is the Ingest Tokens for your repository, (i.e., a string such as 5413ab7e-3ae3-116b-9b90-da446e01e131).

Important

In the above example the port is specified as Logstash will default to port 9200 even for TLS connections. You should declare the correct port for your LogScale configuration.

The port should be the same as used for the LogScale web UI, typically 443 (https) or 80 (http), or commonly port 9200 if your LogScale administrator has configured that.

In the above example, Logstash calls the Linux date command every five seconds. It passes the output from this command to LogScale.

Field Mapping

When you use the ElasticSearch output, Logstash outputs JSON objects. The JSON for an event sent to LogScale with the above configuration looks like this:

json
{
  "@timestamp": "2016-08-25T08:34:37.041Z",
  "message": "Thu Aug 25 10:34:37 CEST 2016\n",
  "command": "date"
}

LogScale can map each JSON object into an Event. Each field in the JSON object becomes a field in the LogScale Event. You will need to pair your ingest token with a parser to process the Logstash timestamps and parse the fields from your events, see Parsing Data.

Create a parser logstash-json with the following content:

logscale
// Parse the event as json
parseJson()
// Convert the timestamp to unix time
| parseTimestamp("yyyy-MM-dd'T'HH:mm:ss[.SSS]XXX", field=@timestamp)

Alternatively, depending on the configuration, the timestamp can be the time at which Logstash handles the event, or the actual timestamp in the data. If the timestamp is present in the data you can configure logstash to parse it, for example, by using the date filter. Another option is to handle parsing the timestamp in LogScale by connecting a parser to the ingest token.

Adding Parsers in LogScale

LogScale can do further parsing/transformation of the data it receives by Assigning Parsers to Ingest Tokens. For more information on parsers,

Dropping Fields

Logstash often adds fields like host and @version to events. You can remove these fields using a filter and the drop_field function in Logstash.