HTTP Event Collector (HEC)

LogScale's HEC API is an alternative HTTP ingest endpoint. You will find it at /api/v1/ingest/hec and at /services/collector.

Note

The best practices described in Ingest via API Best Practices also apply to the HEC API.

Format of Data

Ingested data is a series of whitespace delimited JSON objects, containing one or more of the following elements. All elements are optional.

Member Description
time

Time in seconds since January 1, 1970 in UTC. This can be a integer or floating point number to support milliseconds. LogScale represents time with millisecond precision.

Times are interpreted as follows:

  • Integer — treated as seconds

  • Float/Double — treated as seconds plus milliseconds

  • Integer — parsed to an integer or float and then treated as above

timezone Can be used to describe the time zone in which the event happened. Defaults to Z (i.e., UTC).
index Optional name of the repository to ingest into. In public-facing API's this must — if present — be equal to the repository used to create the ingest token used for authentication. In private cluster setups, humio can be configured to allow these to be different. See below.
sourcetype Translated to #type inside LogScale. If set, this is used to choose which LogScale parser to use for extracting fields.
source Translated to the @source field in LogScale. Typically used to designate the path to the file that is being shipped to LogScale.
host Translated to the @host field in LogScale. Typically used to designate the origin host.
event This can be either a JSON Object or a String. Translated to the @rawstring field in LogScale. When this is a JSON Object, all members of the object will become accessible fields in humio with no further processing. If it is a string, the key/value parser is always applied to the string to extract elements. The key/value parser searches for key=value, key= ” value “ or key='value'.
fields JSON object containing extra fields to the event. This can be used if event is a string, boolean or number and it is pre-processed prior to ingest to extract fields. Tags #tags can be added to the event by specifying fields starting with #. It cannot be used if event is an array or an object.

Authentication

You will need to provide a Ingest Tokens in the HTTP Authorization header.

The ingest token contains the name of the repository the data is stored in, and ingested events will be stored in the repository corresponding to the ingest token.

If the LogScale configuration variable ALLOW_CHANGE_REPO_ON_EVENTS is set to true, then HEC allows ingest to any repository specified as "index": "<repository-name>" in the body of a message, as long as the ingest token is valid for any existing repository on the humio server. If the named repo does not exist then the event remains in the repo designated by the ingest token.

This is a potential security issue on a public API endpoint, so this option should only be used inside a trusted environment.

Example

Below is an example of the contents of a json file, events.json:

javascript
{
  "time" : 1537537729.0,
  "event" : "Fri, 21 Sep 2018 13:48:49 GMT - system started name=webserver",
  "source" : "/var/log/application.log",
  "sourcetype" : "applog",
  "fields" : { "#env" : "prod" }
}

{
  "time" : 1537535729.0,
  "event" : {
    "message" : "System shutdown",
    "host" : { "ip" : "127.0.0.1", "port" : 2222 }
  },
  "fields" : { "#datacenter" : "amazon-east1" }
}
logscale
curl $YOUR_LOGSCALE_URL/api/v1/ingest/hec \
  -X POST \
  -H "Content-Type: text/plain; charset=utf-8" \
  -H "Authorization: Bearer $INGEST_TOKEN" \
  --data "@events.json"

Note

The Content-Type supports both text/plain and application/json

You must make the following changes to the sample configuration

  • Add other fields in the fields section. These fields, and their values, will be added to each event.

  • Insert the URL containing the LogScale host in the $YOUR_LOGSCALE_URL Endpoints field.

  • $INGEST_TOKEN is the Ingest Tokens for your repository, (a string such as fS6Kdlb0clqe0UwPcc4slvNFP3Qn1COzG9DEVLw7v0Ii).

Raw HEC

LogScale's Raw HEC API is a simple line-delimited ingest endpoint for unstructured logs.

You will find it at /api/v1/ingest/hec/raw and at /services/collector/raw.

Simply send a POST to one of the two endpoints above. Each line in the input (separated by /n, /r, or /r/n) will be ingested as an event.

You can optionally add an X-Splunk-Request-Channel header or channel as a query parameter. These will be added as a field on the event named "channel".

Example

In a simple text editor, create a file named, events.txt and copy the following lines into it:

ini
Fri, 21 Sep 2018 13:48:49 GMT - system started name=webserver
System shutdown

Then execute the following from the command-line:

logscale
curl $YOUR_LOGSCALE_URL/api/v1/ingest/hec/raw?channel=foo \
  -X POST \
  -H "Content-Type: text/plain; charset=utf-8" \
  -H "Authorization: Bearer $INGEST_TOKEN" \
  --data-binary "@events.txt"

You must make the following changes to the sample configuration:

  • Insert the URL containing the LogScale host in the $YOUR_LOGSCALE_URL field.

  • $INGEST_TOKEN is the Ingest Tokens for your repository, (a string such as fS6Kdlb0clqe0UwPcc4slvNFP3Qn1COzG9DEVLw7v0Ii).