Modifying Data

There are many different ways of modifying, altering and changing data for the purposes of processing, sorting, or identifying different elements from the incoming events.

Updating (Overwriting) Data

Field values can always be overwritten, and in many cases this can be used to your advantage when working with functions that operate on some fields without them needing to be explicitly called out. For example, to overwrite the value of a field just assign the new value:

logscale
transferValue := 4857465
| transferValue := (transferValue/8)

But when parsing a timestamp from input data, the timestamp might not be found and so we need to default to another value:

logscale
findTimestamp(addErrors=false, timezone=UTC)
| case { @timestamp != *
| @timestamp:=now(); * }

In this example, we try to extract the timestamp from the log line. If the timestamp is not populated by the findTimestamp() then we assign the current timestamp using now().

Numerical Calculations

Basic numerical expressions can be used without requiring special syntax. For example:

logscale
bytesTransferred := 4857465
| megabytesTransferred := (bytesTransferred/1024)

For more complex calculations there are a number of functions with the math: prefix. However, each stage of the calculation may need to be performed individually:

logscale
sidea := 3
| sideb := 4
| sideasq := math:pow(sidea,exponent=2)
| sidebsq := math:pow(sideb,exponent=2)
| diag := sideasq+sidebsq
| sidec := math:sqrt(diag)

In this example, we've calculated the length of the hypotenuese using Pythagoras's theorem.

String Operations

There are no implicit operations or typing for strings, so the expressions that might be used for manipulating numerical values do not work the same way for strings. For example, you cannot concatenate strings using an operator:

Invalid Example for Demonstration - DO NOT USE
logscale
hw := "Hello" + "World"

All string operations must be performed using functions, and in some cases there may be multiple functions to achieve the same or similar results:

How to… Solution
Concatenate two or more strings

Use concat(), for example, newstring := concat(string1,string2)

Use format(), for example, format(field=[string1,string2],format="%s%s)

Split a string

Use splitString() to divide up a string by a specific regular expression, for example, splitString(by=/,/,field=csvstring) to split by a comma

Do not ignore other functions for specific types of string-based constant, such as kvParse(), parseXml() and similar to process specific types.

Format or justify a string

Use format(), for example, format(field=[string1],format="%20s)

Change case Use the upper() and lower() functions

Most other string manipulation operations can be performed using regular expressions.