Adding Fields to Events

The documentation explains how to add new fields in LogScale through two primary methods: regular expression-based field extraction and function-based field creation using 'as' parameters. Additional topics covered include eval syntax for numeric computations, assignment operators for simplified field creation, and field operators for filtering specific fields with functions, providing users with comprehensive options for field manipulation and data processing.

New fields can be created in two ways:

Regular Expression-based Field Extraction

New fields can be extracted from text data using regular expressions and then their values can be tested. This allows access to data that LogScale did not parse when the data was indexed.

For example, if one or more log entries contain text such as ... disk_free=2000 ..., then a query like the following can be used to find entries that have less than 1000 free disk space.

logscale
regex("disk_free=(?<space>[0-9]+)")
| space < 1000

The first line uses regular expressions to extract the value after the equals sign and assigns it to the field space, then filters the events where the extracted field is greater than 1000.

The named capturing groups ((?<FIELDNAME>) are used to extract fields in regular expressions. This combines two principles, the usage of grouping in regular expressions using (), and explicit field creation.

The same result can be obtained written using the regex literal syntax:

logscale
@rawstring=/disk_free=(?<space>[0-9]+)/
| space < 1000

You can apply repeat to field extraction to yield one event for each match of the regular expression. This allows processing multiple values for a named field, or a field name that matches a pattern, as in the following example:

logscale
regex("value[^=]*=(?<someBar>\\S+)", repeat=true)
| groupBy(someBar)

On an input event with a field value:

accesslog
type=foo value=bar1 valueExtra=bar2 value=bar3

the groupBy() sees all three bar values.

Warning

In order to use field-extraction this way, the regular expression must be a top-level expression, that is, | between bars |. The following query does not work:

Invalid Example for Demonstration - DO NOT USE
logscale
// DON'T DO THIS - THIS DOES NOT WORK
type=FOO or /disk_free=(?<space>[0-9]+)/
| space < 1000

Note

Since regular expressions require additional processing, the recommendation is to complete as much simple filtering (both text and/or field matching) as possible earlier in the query chain before applying the regular expression function.

as Parameters

Fields can also be added using functions. Most functions set their result in a field that has the function name prefixed with an underscore (_) by default. For example, the count() function puts its results in a field named _count.

Most functions that produce fields have a parameter called as. By setting this parameter, you can specify the name of the output field. The following example assigns the result of the count() function to the field named cnt (instead of the default _count).:

logscale
count(as=cnt)

Many functions can be used to generate new fields using the as argument. For example the concat() function combines aidValue and cidValue into a single string:

logscale
concat([aidValue, cidValue], as=checkMe2)

The format() function can be used to format information and values into a new value, such as formatting two fields with a comma separator for use with a table:

logscale Syntax
format(format="%s,%s", field=[a, b], as="combined")
| table(combined)

See also the Assignment Operator for shorthand syntax for assigning results to a field.

Eval Syntax

The eval() function can assign fields while doing numeric computations on the input.

The := syntax is short for eval. Use the pipe (|) between assignments.

logscale Syntax
...
| foo := a + b
| bar := a / b
| ...

This is short for the following:

logscale Syntax
...
| eval(foo = a + b)
| eval(bar = a / b)
| ...

Assignment Operator

The operator := can be used with functions that take an as parameter. When the right side of the assignment contains a Function Call, the assignment is rewritten to specify the as= argument which, by convention, is the output field name:

logscale Syntax
...
| foo := min(x)
| bar := max(x)
| ...

The previous example is short for the following example:

logscale Syntax
...
| min(x, as=foo)
| max(x, as=bar)
| ...

Field Operator

The field operator filters a specific field with any function that has the field parameter, so that this:

logscale Syntax
...
| ip_addr =~ cidr(subnet="127.0.0.1/24")
| ...

Is synonymous with this:

logscale Syntax
...
| cidr(subnet="127.0.0.1/24", field=ip_addr)
| ...

This works with many functions, including regex() and replace().