Data Manipulation Query Functions
LogScale's event and data manipulation functions allow event creation, modification and data manipulation of events and fields within the event.
Table: Data Manipulation Query Functions
| Function | Default Argument | Availability | Description |
|---|---|---|---|
accumulate([current], function) | function | Â | Applies an aggregation function cumulatively to a sequence of events. |
array:append(array, values) | array | Â | Appends single or multiple values to an array, or creates a new array if it does not already exist. |
array:contains(array, value) | array | Â | Checks whether the given value matches any of the values of the array and excludes the event if no value matches. |
array:dedup(array, [asArray]) | array | Â |
array:dedup() removes duplicate elements from
an array. The ordering of the first occurrence of each unique
element is preserved.
|
array:exists(array, condition, [var]) | array | Â |
Filters events based on whether the given array contains an
element that satisfies a given condition (based on the array
argument). Recommended for flat arrays. Does not work on nested
arrays — use objectArray:exists()
instead.
|
array:filter(array, [asArray], function, [var]) | array | Â | Drops entries from the input array using the given filtering function. |
array:reduceAll(array, function, var) | array | Â | Computes a value from all events and array elements of the specified array. |
array:reduceColumn(array, [as], function, var) | array | Â | Computes an aggregate value for each array element with the same index. |
array:rename(array, asArray) | array | Â | Takes the name of an array and renames all fields of this array. |
array:sort(array, [asArray], [order], [type]) | array | Â | Sorts the elements of an array of values using the given sorting data type and order. |
array:union(array, [as]) | array | Â | Determines the set union of array values over input events. |
bitfield:extractFlags(field, [onlyTrue], output) | Â | Â | Interprets an integer as a bit field and extracts the specified flags. |
bitfield:extractFlagsAsArray([asArray], field, flagNames) | Â | Â | Interprets an integer as a bit field and extracts the specified flags. Information is returned as an array, listed in ascending order, from lowest bit to highest bit. |
bitfield:extractFlagsAsString([as], field, flagNames, [separator]) | Â | Â | Interprets an integer as a bit field and extracts the specified flags. Information is returned as a string, listed in ascending order, from lowest bit to highest bit. |
coalesce([as], expressions, [ignoreEmpty]) | expressions | Â | Selects the value of the first expression from a list of expressions |
concat([as], field) | field | Â | Concatenates the values of a list of fields into a value in a new field. |
concatArray([as], field, [from], [prefix], [separator], [suffix], [to]) | field | Â | Concatenates values of all fields with same name and an array suffix into a new field. |
copyEvent(type) | type | Â | Duplicates an event so that the pipeline will see both events. |
drop(fields) | fields | Â | Removes specified fields from each event. |
dropEvent() | Â | Â | Drops completely an event in parser pipeline to stop it from being ingested. |
eval() | Â | Â | Creates a new field by evaluating the provided expression. |
format([as], field, format, [timezone]) | format | Â | Formats a string using printf-style. |
getField([as], source) | Â | Â | Reads dynamically-named fields that are computed from an expression. |
json:prettyPrint([as], [field], [step], [strict]) | field | Â | More readable output to a JSON field. |
lowercase(field, [include], [locale]) | field | Â | Changes field name or content to lowercase for parsers. |
neighbor([direction], [distance], include, [prefix]) | include | Â | Allows access to fields from a single neighboring event in a sequence. |
parseCEF([field], [headerprefix], [keeplabels], [labelprefix], [prefix]) | field | Â | Parses CEF version 0.x encoded messages. |
parseCsv(columns, [delimiter], [excludeEmpty], field, [trim]) | field | Â | Parses a CSV-encoded field into known columns. |
parseHexString([as], [charset], field) | field | Â | Parses input from hex encoded bytes, decoding resulting bytes as a string. |
parseJson([exclude], [excludeEmpty], field, [handleNull], [include], [prefix], [removePrefixes]) | field | Â | Parses specified fields as JSON. |
parseLEEF([delimiter], [field], [headerprefix], [keeplabels], [labelprefix], [parsetime], [prefix], [timezone]) | field | Â | Parses LEEF version 1.0 and 2.0 encoded messages. |
partition(condition, function, [split]) | function | Â | Splits a sequence of events into multiple partitions based on a condition. |
readFile(file, [include], [limit]) | file | Â |
Uses a .csv lookup file or ad-hoc table as
data input for the query.
|
rename([as], field) | field | Â | Renames one or more given fields. |
replace([as], [field], [flags], regex, [replacement], [with]) | regex | Â | Replaces each substring that matches given regular expression with given replacement. |
sankey(source, target, [weight]) | Â | Â | Produces data compatible with Sankey widget. |
series(collect, [endmatch], [maxduration], [maxpause], [memlimit], [separator], [startmatch]) | collect | Â | Collects a series of values for selected fields from multiple events into one or more events. |
setField(target, value) | Â | Â | Sets fields whose names are not known but computed from an expression. |
slidingTimeWindow([current], function, span, [timestampfield]) | function | Â | Applies an aggregation to a moving time-based window of events in a sequence. |
slidingWindow([current], events, function) | function | Â | Applies an aggregation to a moving window of a specified number of events in a sequence. |
split([field], [strip]) | field | Â | Splits an event structure created by a JSON array into distinct events. |
splitString([as], by, [field], [index]) | field | Â | Splits a string by specifying a regular expression by which to split. |
stripAnsiCodes([as], field) | field | Â | Removes ANSI color codes and movement commands. |
text:editDistance([allowTranspositions], [as], [ignoreCase], maxDistance, reference, target) | Â | Â | Returns the edit distance (Levenshtein distance) between a target string and a reference string. |
text:editDistanceAsArray([allowTranspositions], [asArray], [ignoreCase], maxDistance, references, target) | Â | Â | Returns the edit distance (Levenshtein distance) between a target string and a list of reference strings as an object array. |
text:contains(string, substring) | string | Â | Tests if a specific substring is present within a given string. |
text:endsWith(string, substring) | string | Â | Tests if a specific substring is present at the end of a given string. |
text:length([as], string) | string | Â | Computes the length of a string. |
text:positionOf([as], [begin], character, [occurrence], string) | string | Â | Computes the position of a given character or substring within a string. |
text:startsWith(string, substring) | string | Â | Tests if a specific substring is present at the start of a given string. |
text:substring([as], [begin], [end], string) | string | Â | Extracts a substring from a string given a pair of positions into the string. |
transpose([column], [header], [limit], [pivot]) | pivot | Â | Transposes a query results set by creating an event for each attribute. |
unit:convert([as], [binary], field, [from], [keepUnit], [to], [unit]) | field | Â | Converts values between different units. |
upper([as], field, [locale]) | field | Â | Changes contents of a string field to upper case letters. |
urlDecode([as], field) | field | Â | URL-decodes the contents of a string field. |
urlEncode([as], field, [type]) | field | Â | URL encodes the contents of a string field. |
writeJson([as], [field]) | field | Â | Writes data, including fields, as a JSON object. |
xml:prettyPrint([as], field, [step], [strict], [width]) | field | Â | Nicer output to an XML field. |