Query Functions
Manipulating and formatting functions for extracting information from your event data
LogScale's query functions take a set of events, parameters, or configurations. From this, they produce, reduce, or modify values within that set, or in the events themselves within a query pipeline.
Functions broadly fall into two categories, Transformation and Aggregate:
Transformation functions, sometimes referred to as Filter functions, transform or filter data and may add, remove or modify fields.
Aggregate functions combine events into a new results — often a single number or row.
Below is an alphabetical listing of all of LogScale query functions.
Table: Query Functions
| Function | Type | Default Argument | Availability | Description |
|---|---|---|---|---|
array:contains(array, value) | array, data-manipulation, filter | array | introduced in 1.62 | Checks whether the given value matches any of the values of the array and excludes the event if no value matches |
array:filter(array, [asArray], function, var) | data-manipulation, array | array | introduced in 1.39 | Drops entries from the input array using the given filtering function. |
array:reduceAll(array, function, var) | data-manipulation, array | array | introduced in 1.77 | Computes a value from all events and array elements of the specified array. |
array:regex(array, [flags], regex) | regular-expression, array, filter | array | introduced in 1.62 | Checks whether the given pattern matches any of the values of the array and excludes the event from the search result |
asn([as], [field]) | network | field | Determines autonomous system number and organization associated. | |
avg([as], field) | aggregate | field | Calculates the average for a field of a set of events. | |
base64Decode([as], [charset], field) | math | field | Performs Base64 decoding of a field. | |
beta:param(as, parameter) | data-manipulation | parameter | Reads given parameter and assigns the value to a field in the event. | |
beta:repeating([as], [interval]) | data-manipulation | interval | Marks the live query the function is used in as repeating. | |
bitfield:extractFlags(field, output) | data-manipulation | introduced in 1.76 | Interprets an integer as a bit field and extracts the specified flags | |
bucket([buckets], [field], [function], [limit], [minSpan], [span], [timezone], [unit]) | aggregate | span | Extends the groupBy function for grouping by time. | |
callFunction([as], field, function) | aggregate | function | Calls the named function on a field over a set of events. | |
cidr([column], [field], [file], [negate], [subnet]) | network, filter | field | Filters events using CIDR subnets. | |
collect(fields, [limit], [multival], [separator]) | aggregate | fields | Collects fields from multiple events into one event. | |
communityId([as], destinationip, [destinationport], [icmpcode], [icmptype], proto, [seed], sourceip, [sourceport]) | network | introduced in 1.33 | Computes the Community ID, a standard for hashing network flows. | |
concat([as], field) | string, data-manipulation | field | Concatenates the values of a list of fields into a value in a new field. | |
concatArray([as], field, [from], [prefix], [separator], [suffix], [to]) | string, data-manipulation | field | Concatenates values of all fields with same name and an array suffix into a new field. | |
copyEvent(type) | event | type | Duplicates event so pipeline will see both events. | |
count([as], [distinct], [field]) | statistics, aggregate | field | Counts given events. | |
counterAsRate([as], field) | aggregate | field | Calculates the rate for a counter field. | |
createEvents([rawstring]) | event, aggregate-testing | Generates temporary events as part of the query. | ||
default(field, [replaceEmpty], value) | event | value | Creates a field to given parameter and given value. | |
drop(fields) | event | fields | Removes attributes or columns from a result set. | |
dropEvent() | event | Drops completely an event in parser pipeline to stop it from being ingested. | ||
end([as]) | time-date | as | Assign the end of the search time interval to the field provided by parameter as. | |
eval() | data-manipulation | Creates a new field by evaluating the provided expression. | ||
eventFieldCount([as]) | event | Returns number of fields event uses internally for the values. | ||
eventInternals([prefix]) | event | Add a set of fields describing the storage locations of this event. | ||
eventSize([as]) | event | Returns the number of bytes that this event uses internally for the values, not counting the bytes for storing the field names. | ||
fieldset() | event | Retrieves a list of available fields. | ||
fieldstats() | event | Retrieves stats about fields. | ||
findTimestamp([addErrors], [as], [field], [timezone], [timezoneAs]) | time-date | Finds timestamp in given field and parses, trying multiple timestamp formats. | ||
format([as], field, format, [timezone]) | formatting, data-manipulation | format | Formats a string using printf-style. | |
formatDuration([as], field, [from], [precision]) | formatting | field | Formats a duration into a more readable string. | |
formatTime(as, [field], format, [locale], [timezone]) | formatting | format | Formats a string according to strftime(). | |
geography:distance([as], lat1, lat2, lon1, lon2) | geolocation | Calculates the distance between two geographical coordinates along an ideal earth surface | ||
geohash([as], [lat], [lon], [precision]) | geolocation | geohash | Calculates a geohash value given two fields representing latitude and longitude. | |
groupBy(field, [function], [limit]) | aggregate | field | Groups events by specified fields and executes aggregate functions on each group. | |
hash([as], field, [limit], [seed]) | event | field | Computes a non-cryptographic hash of a list of fields. | |
hashMatch([bits], [field], [hash], input, [salt]) | comparison, filter | input | Calculates a secure hash of a field and uses it to match events as a filter. | |
hashRewrite([as], [bits], field, [hash], [replaceInRawstring], salt) | comparison, data-manipulation | field | Calculates a secure hash of a field for storing in the event. | |
head([limit]) | aggregate | limit | Returns the oldest events. | |
holtwinters([alpha], [beta], [gamma], period) | aggregate | period | removed in 1.73 | Used to generate a trendline for a periodic dataset. |
in(field, values) | comparison, filter | field | Filters records by values where field is in given values. | |
ioc:lookup([confidenceThreshold], field, [include], [prefix], [strict], type) | geolocation | field | Look up IOCs (indicators of compromise). | |
ipLocation([as], [field]) | geolocation | field | Determines country, city, longitude, and latitude for given IP address. | |
join([end], field, [include], [key], [limit], [live], [max], [mode], [query], [repo], [start], [view]) | join | query | Join two LogScale searches. | |
json:prettyPrint([as], [field], [step], [strict]) | formatting, data-manipulation | field | Nicer output to a JSON field. | |
kvParse([as], [excludeEmpty], [field], [override], [separator], [separatorPadding]) | parsing | field | Key-value parse events. | |
length([as], field) | string | field | Returns the number of characters in a string field. | |
linReg([prefix], x, y) | aggregate | linreg | Computes linear relationship model between two variables using least-squares fitting. | |
lookup(from, [include], on) | format | from | deprecated in 1.30.0 | Enhances events with metadata. |
lower([as], field, [type]) | formatting | field | Changes text of a given string field to lower-case letters. | |
lowercase(field, [include], [locale]) | string, data-manipulation, formatting | field | Changes field name or content to lowercase for parsers. | |
match([column], field, file, [glob], [ignoreCase], [include], [mode], [strict]) | string | file | Searches text using a CSV or JSON file and can enhance entries. | |
math:abs([as], field) | math | field | Calculates the absolute value of a field. | |
math:arccos([as], field) | math | field | Calculates the arc cosine of a field. | |
math:arcsin([as], field) | math | field | Calculates the arc sine of a field. | |
math:arctan([as], field) | math | field | Calculates the arc tangent of a value. | |
math:arctan2([as], xfield, yfield) | math | Calculates the arc tangent of a value. | ||
math:ceil([as], field) | math | field | Rounds field value to smallest integer that's larger than or equal to it. | |
math:cos([as], field) | math | field | Calculates the cosine of a field. | |
math:cosh([as], field) | math | Returns the hyperbolic cosine of a double field. | ||
math:deg2rad([as], field) | math | field | Converts angles from degrees to radians. | |
math:exp([as], field) | math | field | Calculates Euler's number e raised to the power of a double value in a field. | |
math:expm1([as], field) | math | field | Returns the exponential value of a number minus 1. | |
math:floor([as], field) | math | field | Returns largest integer value not greater than the field value given. | |
math:log([as], field) | math | field | Calculates the natural logarithm (base e) of the value in a double field. | |
math:log10([as], field) | math | field | Calculates the base 10 logarithm of a double field. | |
math:log1p([as], field) | math | field | Calculates the natural logarithm of the sum of field's value and 1. | |
math:log2([as], field) | math | field | Calculates the base 2 logarithm of a double field. | |
math:mod([as], divisor, field) | math | field | Calculates the floor modulus of field value and the divisor. | |
math:pow([as], exponent, field) | math | field | Calculates the field value to the exponent power. | |
math:rad2deg([as], field) | math | field | Converts angles from radians to degrees. | |
math:sin([as], field) | math | field | Calculates the sine of a field. | |
math:sinh([as], field) | math | field | Calculates the hyperbolic sine of a double field. | |
math:spherical2cartesian([as], azimuth, polar, radius) | math | Calculates the average for a field of a set of events. | ||
math:sqrt([as], field) | math | field | Calculates the rounded positive square root of a double field. | |
math:tan([as], field) | math | field | Calculates the trigonometric tangent of an angle in a field. | |
math:tanh([as], field) | math | field | Calculates the hyperbolic tangent of a field. | |
max([as], field, [type]) | statistics | field | Finds the largest number for the specified field over a set of events. | |
min([as], field, [type]) | statistics | field | Finds the smallest number for the specified field over a set of events. | |
now([as]) | time-date | as | Assign the current time to the field provided by parameter as. | |
parseCEF([field], [prefix]) | data-manipulation, parsing | field | Parses CEF version 0.x encoded messages. | |
parseCsv(columns, [delimiter], [excludeEmpty], field) | data-manipulation, parsing | field | Parses a CSV-encoded field into known columns. | |
parseFixedWidth(columns, [field], [trim], widths) | parsing | field | Parses a fixed width-encoded field into known columns. | |
parseHexString([as], [charset], field) | data-manipulation, parsing | field | Parses input from hex encoded bytes, decoding resulting bytes as a string. | |
parseInt([as], [endian], field, [radix]) | parsing | field | Converts an integer from any radix or base to base-ten, decimal radix. | |
parseJson([exclude], [excludeEmpty], field, [handleNull], [include], [prefix], [removePrefixes]) | parsing, data-manipulation | field | Parses specified fields as JSON. | |
parseLEEF([delimiter], [field], [parsetime], [prefix], [timezone]) | data-manipulation, parsing | field | Parses LEEF version 1.0 and 2.0 encoded messages. | |
parseTimestamp([addErrors], [as], [caseSensitive], field, [format], [timezone], [timezoneAs]) | parsing, time-date | Parses a string into a timestamp. | ||
parseUrl([as], [field]) | parsing | field | Extracts URL components from a field. | |
parseXml(field, [prefix], [strict]) | parsing | field | Parses specified field as XML. | |
percentile([accuracy], [as], field, [percentiles]) | statistics | field | Returns one event with a field for each percentile specified. | |
range([as], field) | statistics | field | Finds numeric range between smallest and largest numbers for field over a set of events. | |
rdns([as], field, [server]) | network | field | Events using RDNS lookup. | |
regex([field], [flags], [limit], regex, [repeat], [strict]) | regular-expression, string | regex | Extracts new fields using a regular expression. | |
rename(as, field) | data-manipulation, event | field | Renames a given field. | |
replace([as], [field], [flags], regex, [replacement], [with]) | string, data-manipulation, regular-expression | regex | Replaces each substring that matches given regular expression with given replacement. | |
round([as], field, [how]) | math | field | Rounds an input field up or down, depending on which is nearest. | |
sample([field], [percentage]) | statistics, filter | percentage | Samples the event stream. | |
sankey(source, target, [weight]) | data-manipulation, widget | sankey | Produces data compatible with Sankey widget. | |
select(fields) | event | fields | Used to specify a set of fields to select from each event. | |
selectFromMax(field, include) | aggregate | Selects event with the largest value for the specified field. | ||
selectFromMin(field, include) | aggregate | Selects event with the smallest value for the specified field. | ||
selectLast(fields) | aggregate | fields | Specify fields to select from events, keeping value of most recent event for each field. | |
selfJoin([collect], field, [limit], [postfilter], [prefilter], [select], where) | join | field | Used to collate data from events that share a key. | |
selfJoinFilter(field, [prefilter], where) | filter, join | field | Runs query to determine IDs, and then gets all events containing one of them. | |
series(collect, [endmatch], [maxduration], [maxpause], [memlimit], [separator], [startmatch]) | aggregate, data-manipulation | collect | Collects a series of values for selected fields from multiple events into one or more events. | |
session([function], [maxpause]) | statistics | function | Collects events into sessions, and aggregates them. | |
shannonEntropy([as], field) | network | field | Calculates a entropy measure from a string of characters. | |
sort([field], [limit], [order], [reverse], [type]) | aggregate | field | Sorts events by their fields. | |
split([field], [strip]) | data-manipulation, string, regular-expression | field | Splits an event structure created by a JSON array into distinct events. | |
splitString([as], by, [field], [index]) | data-manipulation, string | field | Splits a string by specifying a regular expression by which to split. | |
start([as]) | time-date | as | Assign the start of the search time interval to the field provided by parameter as. | |
stats([function]) | statistics | function | Used to compute multiple aggregate functions over the input. | |
stdDev([as], field) | statistics | field | Calculates the standard deviation for a field over a set of events. | |
stripAnsiCodes([as], field, [type]) | data-manipulation, string | field | Removes ANSI color codes and movement commands. | |
subnet([as], bits, field) | network | field | Computes a subnet from a IPV4 field. | |
sum([as], field) | aggregate | field | Calculates the sum for a field over a set of events. | |
table(fields, [limit], [order], [reverse], [sortby], [type]) | widget | fields | Used to create a widget to present the data in a table. | |
tail([limit]) | event | limit | Returns the newest events. | |
test(expression) | filter, event | expression | Evaluates boolean expression and filters events. | |
time:dayOfMonth([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the day of the month of a timestamp field. | |
time:dayOfWeek([as], field, [timezone], [timezoneField]) | time-date | field | Gets day of week from 1 (Mon) to 7 (Sun) of a timestamp. | |
time:dayOfWeekName([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the English display name of day of the week of a timestamp field. | |
time:dayOfYear([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the day of the year of a timestamp field, from 1 to 365, or 366 in a leap year. | |
time:hour([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the hour (24-hour clock) of a timestamp field. | |
time:millisecond([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the millisecond of a timestamp field. | |
time:minute([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the minute value of a timestamp field. | |
time:month([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the month of a timestamp field (from 1 to 12). | |
time:monthName([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the English name of month of a timestamp field (e.g., January). | |
time:second([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the second of a timestamp field. | |
time:weekOfYear([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the week number within a year of a timestamp, a value from 1 to 53. | |
time:year([as], [field], [timezone], [timezoneField]) | time-date | field | Gets the year of a timestamp field. | |
timeChart([buckets], [function], [limit], [minSpan], [series], [span], [timezone], [unit]) | widget | series | Used to draw a linechart where the x-axis is time. | |
tokenHash([as], field) | string | field | Calculates a structure hash which is equal for similarly structured input. | |
top([as], [error], field, [limit], [max], [percent], [rest], [sum]) | aggregate | field | Finds the top results based on a given field. | |
transpose([column], [header], [limit], [pivot]) | data-manipulation, string | pivot | Transposes a query results set by creating an event for each attribute. | |
unit:convert([as], [binary], field, [from], [keepUnit], [to], [unit]) | data-manipulation, math | field | Converts values between different units. | |
upper([as], field, [locale]) | formatting, data-manipulation | field | Changes contents of a string field to upper-case letters. | |
urlDecode([as], field) | network | field | URL-decodes the contents of a string field. | |
urlEncode([as], field, [type]) | data-manipulation, network | field | URL-encodes the contents of a string field. | |
window([buckets], [function], [span]) | aggregate | function | Computes aggregate functions over a sliding window of data. | |
worldMap([ip], [lat], [lon], [magnitude], [precision]) | widget | worldmap | Used to produce data compatible with the World Map widget. | |
writeJson([as], [field]) | data-manipulation, format | field | Writes data, including fields, as a JSON object. | |
xml:prettyPrint([as], field, [step], [strict], [width]) | data-manipulation | field | Nicer output to an XML field. |