Data Manipulation Query Functions
LogScale's event and data manipulation functions allow event creation, modification and data manipulation of events and fields within the event.
Table: Data Manipulation Query Functions
| Function | Default Argument | Availability | Description |
|---|---|---|---|
accumulate([current], function) | function | Applies an aggregation function cumulatively to a sequence of events. | |
array:append(array, values) | array | Appends single or multiple values to an array, or creates a new array if it does not already exist. | |
array:contains(array, value) | array | Checks whether the given value matches any of the values of the array and excludes the event if no value matches. | |
array:dedup(array, [asArray]) | array |
array:dedup() removes duplicate elements from
an array. The ordering of the first occurrence of each unique
element is preserved.
| |
array:exists(array, condition, [var]) | array |
Filters events based on whether the given array contains an
element that satisfies a given condition (based on the array
argument). Recommended for flat arrays. Does not work on nested
arrays — use objectArray:exists()
instead.
| |
array:filter(array, [asArray], function, [var]) | array | Drops entries from the input array using the given filtering function. | |
array:reduceAll(array, function, var) | array | Computes a value from all events and array elements of the specified array. | |
array:reduceColumn(array, [as], function, var) | array | Computes an aggregate value for each array element with the same index. | |
array:rename(array, asArray) | array | Takes the name of an array and renames all fields of this array. | |
array:sort(array, [asArray], [order], [type]) | array | Sorts the elements of an array of values using the given sorting data type and order. | |
array:union(array, [as]) | array | Determines the set union of array values over input events. | |
bitfield:extractFlags(field, [onlyTrue], output) | Interprets an integer as a bit field and extracts the specified flags. | ||
bitfield:extractFlagsAsArray([asArray], field, flagNames) | Interprets an integer as a bit field and extracts the specified flags. Information is returned as an array, listed in ascending order, from lowest bit to highest bit. | ||
bitfield:extractFlagsAsString([as], field, flagNames, [separator]) | Interprets an integer as a bit field and extracts the specified flags. Information is returned as a string, listed in ascending order, from lowest bit to highest bit. | ||
coalesce([as], expressions, [ignoreEmpty]) | expressions | Selects the value of the first expression from a list of expressions | |
concat([as], field) | field | Concatenates the values of a list of fields into a value in a new field. | |
concatArray([as], field, [from], [prefix], [separator], [suffix], [to]) | field | Concatenates values of all fields with same name and an array suffix into a new field. | |
copyEvent(type) | type | Duplicates event so pipeline will see both events. | |
drop(fields) | fields | Removes specified fields from each event. | |
dropEvent() | Drops completely an event in parser pipeline to stop it from being ingested. | ||
eval() | Creates a new field by evaluating the provided expression. | ||
format([as], field, format, [timezone]) | format | Formats a string using printf-style. | |
getField([as], source) | Reads dynamically-named fields that are computed from an expression. | ||
json:prettyPrint([as], [field], [step], [strict]) | field | More readable output to a JSON field. | |
lowercase(field, [include], [locale]) | field | Changes field name or content to lowercase for parsers. | |
neighbor([direction], [distance], include, [prefix]) | include | Allows access to fields from a single neighboring event in a sequence. | |
parseCEF([field], [headerprefix], [keeplabels], [labelprefix], [prefix]) | field | Parses CEF version 0.x encoded messages. | |
parseCsv(columns, [delimiter], [excludeEmpty], field, [trim]) | field | Parses a CSV-encoded field into known columns. | |
parseHexString([as], [charset], field) | field | Parses input from hex encoded bytes, decoding resulting bytes as a string. | |
parseJson([exclude], [excludeEmpty], field, [handleNull], [include], [prefix], [removePrefixes]) | field | Parses specified fields as JSON. | |
parseLEEF([delimiter], [field], [headerprefix], [keeplabels], [labelprefix], [parsetime], [prefix], [timezone]) | field | Parses LEEF version 1.0 and 2.0 encoded messages. | |
partition(condition, function, [split]) | function | Splits a sequence of events into multiple partitions based on a condition. | |
readFile(file, [include], [limit]) | file |
Uses a .csv lookup file or ad-hoc table as
data input for the query.
| |
rename([as], field) | field | Renames one or more given fields. | |
replace([as], [field], [flags], regex, [replacement], [with]) | regex | Replaces each substring that matches given regular expression with given replacement. | |
sankey(source, target, [weight]) | Produces data compatible with Sankey widget. | ||
series(collect, [endmatch], [maxduration], [maxpause], [memlimit], [separator], [startmatch]) | collect | Collects a series of values for selected fields from multiple events into one or more events. | |
setField(target, value) | Sets fields whose names are not known but computed from an expression. | ||
slidingTimeWindow([current], function, span, [timestampfield]) | function | Applies an aggregation to a moving time-based window of events in a sequence. | |
slidingWindow([current], events, function) | function | Applies an aggregation to a moving window of a specified number of events in a sequence. | |
split([field], [strip]) | field | Splits an event structure created by a JSON array into distinct events. | |
splitString([as], by, [field], [index]) | field | Splits a string by specifying a regular expression by which to split. | |
stripAnsiCodes([as], field) | field | Removes ANSI color codes and movement commands. | |
text:contains(string, substring) | string | Tests if a specific substring is present within a given string. | |
transpose([column], [header], [limit], [pivot]) | pivot | Transposes a query results set by creating an event for each attribute. | |
unit:convert([as], [binary], field, [from], [keepUnit], [to], [unit]) | field | Converts values between different units. | |
upper([as], field, [locale]) | field | Changes contents of a string field to upper case letters. | |
urlDecode([as], field) | field | URL-decodes the contents of a string field. | |
urlEncode([as], field, [type]) | field | URL-encodes the contents of a string field. | |
writeJson([as], [field]) | field | Writes data, including fields, as a JSON object. | |
xml:prettyPrint([as], field, [step], [strict], [width]) | field | Nicer output to an XML field. |