Parser API

Data sent to Humio usually has some structure. You can use parsers to extract this structure. For example, a standard web server log has the status code, method, and URL fields for each log line.

When sending data to Humio, for example using Filebeat, you must specify a parser telling Humio how to parse the incoming data.

Humio has some built-in parsers for common formats like access logs from Apache and Nginx web servers. It also allows for custom parsers.

The endpoints displayed on this page are deprecated starting with version 1.27 of Humio. Users are encouraged to use instead the corresponding queries and mutations provided by our GraphQL API.

List Parsers

Request

To list all parsers for a given repository:

shell
GET /api/v1/repositories/$REPOSITORY_NAME/parsers

Response

javascript
[
  {
    "builtIn": true,
    "parseKeyValues": false,
    "parser": "(?<client>\\S+)\\s+-\\s+(?<userid>\\S+)\\s+\\[(?<@timestamp>.*)\\]\\s+\"((?<method>\\S+)\\s+(?<url>\\S+)?\\s+(?<httpversion>\\S+)?|-)\"\\s+(?<statuscode>\\d+)\\s+(?<responsesize>\\S+)\\s+\"(?<referrer>.*)\"\\s+\"(?<useragent>.*)\"\\s*(?<responsetime>.+)?",
    "id": "accesslog",
    "dateTimeFields": [
      "@timestamp"
    ],
    "kind": "regex",
    "dateTimeFormat": "dd/MMM/yyyy:HH:mm:ss Z"
  },
  {
    "id": "json",
    "kind": "json",
    "parseKeyValues": false,
    "dateTimeFields": [
      "@timestamp"
    ]
  }
]

The output format is similar to the input format in [Add Parser](#add-parser), where $YOUR_HUMIO_URL is the URL for your Humio Cloud Account

Here’s an example:

shell
curl https://`$YOUR_HUMIO_URL`/api/v1/repositories/$REPOSITORY_NAME/parsers \
   -H "Authorization: Bearer $API_TOKEN"

Create or Update Parser

Request

To create a parser for a given repository:

shell
POST /api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_ID

or to updated an existing parser use

shell
PUT /api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_ID

The JSON request body has the following attributes:

Name

Type

Required

Description

kind

String

Yes

Controls which parser kind to create. You can set this to regex, or json.

parser

String

Yes

The parser specification.
The contents of this field vary depending on the type of parser you are creating. See the details below.

parseKeyValues

Boolean

No

Sets whether you want the parser to parse ‘key=value’ pairs in the log line.
The default value is false.

dateTimeFields

Array

Yes

Specifies the fields which contain the timestamp of the event.
You can specify multiple fields, for example, a date field and a time field. The values of these fields are concatenated with whitespaces.
Humio parses these fields with the format that you specify in the dateTimeFormat attribute.

dateTimeFormat

String

No

The format string that Humio should use to parse the fields identified by the dateTimeFields attribute.
This attribute uses the Java DateTimeFormatter syntax.
The default value is the ISO-8601 format, for example, yyyy-MM-dd'T'HH:mm:ss.SSSZ, with milliseconds as an optional addition.

timezone

String

No

This field is only used if the timestamp of the event is in localtime and does not have a timezone.
In that case, you can use it to set a timezone.
Do not use this field if the timezone is part of the dateTimeFormat.
Examples: UTC, Z, or Europe/Copenhagen.

tagFields

Array

No

Specify fields in events generated by this parser that should be turned into tags.<br/> For example it could be specified that the host field in the events from this parser should be treated as a tag.

Response

Standard HTTP response codes, where $YOUR_HUMIO_URL is the URL for your Humio Cloud Account.

Here’s an example:

shell
curl https://$YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME \
   -XPUT \
   -H "Authorization: Bearer $API_TOKEN" \
   -H "Content-Type: application/json" \
   -d '{"parser": "^(?<date>\\S*) (?<time>\\S*) (?<host>\\S*) (?<appname>\\S*):",
        "kind": "regex",
        "parseKeyValues": true,
        "dateTimeFormat": "yyyy-MM-dd HH:mm:ss.SSS",
        "dateTimeFields": ["date", "time"],
        "timezone": "UTC",
        "tagFields": ["host"]
       }'

Parser Types

Humio currently supports two types of parsers:

  • [RegExp Parsers](#regexp-parsers)

  • [JSON Parsers](#json-parsers)

JSON Parsers

When using the json parser type, Humio expects data to be in the JSON format.

The only required field is dateTimeFields. This designates which field has the timestamp.

Here’s an example:

shell
curl https://$YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME \
  -XPUT \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"kind": "json", "dateTimeFields": ["@timestamp"] }'

RegExp Parsers

The “regex” parser type allows you to specify parsers using regular expressions, where named capture groups specify fields.

This example shows how to parse Nginx access logs with regular expressions. Note the use of \ to escape backslashes:

shell
(?<client>\\S+)\\s+-\\s+(?<userid>\\S+)\\s+\\[(?<@timestamp>.*)\\]\\s+\"((?<method>\\S+)\\s+(?<url>\\S+)?\\s+(?<httpversion>\\S+)?|-)\"\\s+(?<statuscode>\\d+)\\s+(?<responsesize>\\S+)\\s+\"(?<referrer>.*)\"\\s+\"(?<useragent>.*)\"\\s*(?<responsetime>.+)?

Delete Parser

Request

To delete a parser from a given repository, make the following request:

shell
DELETE /api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME

Response

Standard HTTP response codes, where $YOUR_HUMIO_URL is the URL for your Humio Cloud Account.

Here’s an example:

shell
curl https://$YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME \
  -XDELETE \
  -H "Authorization: Bearer $API_TOKEN"