You can extract new fields from your text data using regular expressions and then test their values. This lets you access data that LogScale did not parse when it indexed the data.

For example, if your log entries contain text such as ... disk_free=2000 ..., then you can use a query like the following to find the entries that have less than 1000 free disk space

regex("disk_free=(?<space>[0-9]+)")
| space < 1000

The first line uses regular expressions to extract the value after the equals sign and assign it to the field space, and then filter the events where the extracted field is greater than 1000.

The named capturing groups ((?<FIELDNAME>) are used to extract fields in regular expressions. This combines two principles, the using of grouping in regular expressions using (), and explicit field creation.

The same result can be obtained written using the regex literal syntax:

@rawstring=/disk_free=(?<space>[0-9]+)/
| space < 1000

You can apply repeat to field extraction to yield one event for each match of the regular expression. This allows processing multiple values for a named field, or a field name that matches a pattern, as in this example:

regex("value[^=]*=(?<someBar>\\S+)", repeat=true)
| groupby(someBar)

On an input event with a field value of:

type=foo value=bar1 valueExtra=bar2 value=bar3

the groupBy() sees all three bar values.

// DON'T DO THIS - THIS DOES NOT WORK
type=FOO or /disk_free=(?<space>[0-9]+)/
| space < 1000

Fields can also be added by functions. Most functions set their result in a field that has the function name prefixed with a _ by default. For example the count() puts its result in a field _count.

Most functions that produce fields have a parameter called as. By setting this parameter you can specify the name of the output field, for example

count(as=cnt)

Assigns the result of the count() to the field named cnt (instead of the default _count).

Many functions can be used to generate new fields using the as argument. For example concat():

concat([aidValue, cidValue], as=checkMe2)

Combines the aidValue and cidValue into a single string.

The format() can be used to format information and values into a new value, for example, formatting two fields with a comma separator for use with a table:

format(format="%s,%s", field=[a, b], as="combined")
| table(combined)

See also the Assignment Operator for shorthand syntax for assigning results to a field.

The function eval() can assign fields while doing numeric computations on the input.

The := syntax is short for eval. Use | between assignments.

...
| foo := a + b
| bar := a / b
| ...

This is short for the following:

...
| eval(foo = a + b)
| eval(bar = a / b)
| ...

You can use the operator := with functions that take an as parameter. When what's on the right hand side of the assignment is a LQL Grammar: Function arguments, the assignment is rewritten to specify the as= argument which, by convention, is the output field name. For example

...
| foo := min(x)
| bar := max(x)
| ...

The above is short for this:

...
| min(x, as=foo)
| max(x, as=bar)
| ...

You can use attr =~ fun() with any function that has a parameter named field. It designates the field=attr argument and lets you write:

...
| ip_addr =~ cidr(subnet="127.0.0.1/24")
| ...

The alternative is this:

...
| cidr(subnet="127.0.0.1/24", field=ip_addr)
| ...

This also works well with regex() and replace(). It's a shorthand, and very convenient.