New fields can be created in two ways:
Regular Expression-based Field Extraction
You can extract new fields from your text data using regular expressions and then test their values. This lets you access data that LogScale did not parse when it indexed the data.
For example, if your log entries contain text such as
... disk_free=2000 ..., then you can use
a query like the following to find the entries that have less than 1000
free disk space
regex("disk_free=(?<space>[0-9]+)") | space < 1000
The first line uses regular expressions to extract the value after the
equals sign and assign it to the field space, and
then filter the events where the extracted field is greater than
The named capturing groups
(?<FIELDNAME>) are used to
extract fields in regular expressions. This combines two principles, the
using of grouping in regular expressions using
(), and explicit field creation.
The same result can be obtained written using the regex literal syntax:
@rawstring=/disk_free=(?<space>[0-9]+)/ | space < 1000
You can apply repeat to field extraction to yield one event for each match of the regular expression. This allows processing multiple values for a named field, or a field name that matches a pattern, as in this example:
regex("value[^=]*=(?<someBar>\\S+)", repeat=true) | groupby(someBar)
On an input event with a field value of:
type=foo value=bar1 valueExtra=bar2 value=bar3
groupBy() sees all three bar values.
In order to use field-extraction this way, the regular expression must
be a top-level expression, that is,
|. The following doesn't
// DON'T DO THIS - THIS DOES NOT WORK type=FOO or /disk_free=(?<space>[0-9]+)/ | space < 1000
Since regular expressions require additional processing, it is recommended to do as much simple filtering (text or field matching) as possible earlier in the query chain before applying the regular expression function.
Fields can also be added by functions. Most functions set their result
in a field that has the function name prefixed with a
_ by default. For example the
count() puts its result in a field
Most functions that produce fields have a parameter called
as. By setting this parameter you can
specify the name of the output field, for example
Assigns the result of the
count() to the field
cnt (instead of the default
concat([aidValue, cidValue], as=checkMe2)
Combines the aidValue and cidValue into a single string.
format() can be used to format information and
values into a new value, for example, formatting two fields with a comma
separator for use with a table:
format(format="%s,%s", field=[a, b], as="combined") | table(combined)
See also the Assignment Operator for shorthand syntax for assigning results to a field.
eval() can assign fields while doing
numeric computations on the input.
:= syntax is short for eval. Use
| between assignments.
... | foo := a + b | bar := a / b | ...
This is short for the following:
... | eval(foo = a + b) | eval(bar = a / b) | ...
You can use the operator
functions that take an
When what's on the right hand side of the assignment is a function call,
the assignment is rewritten to specify the
as= argument which, by convention, is
the output field name. For example
... | foo := min(x) | bar := max(x) | ...
The above is short for this:
... | min(x, as=foo) | max(x, as=bar) | ...
You can use
attr =~ fun() with any
function that has a parameter named
field. It designates the
field=attr argument and lets you write:
... | ip_addr =~ cidr(subnet="127.0.0.1/24") | ...
The alternative is this:
... | cidr(subnet="127.0.0.1/24", field=ip_addr) | ...