Function Types

Different functions operate in a variety of different ways in order to process the incoming events. The function type affects:

  • Where in a query the function can be used; some functions must be used before or after other functions or function types.

  • What fields in the events the function operates on.

  • What the output of the function is (a single field, the event as a while, or adding or removing events).

  • How many times the data is processed.

  • Whether the function affects the operation or execution of the query as a whole.

Functions, their type, and operation are summarized in the table below:

Subqueries.  At runtime, written queries execute in multiple steps. Some steps run as independent subqueries, which execute their operations separately from other query components. Other steps require input from previous query results, making them dependent subqueries. The primary query combines all results to produce the final output.

Function Type Effect Usage Context
Transforming Modifies event fields or filters data. Filters or updates data.  
Aggregating Aggregates data. Summarizes and collates information by specific fields.  
Asynchronous Operates on data post-aggregation. Must be used after an aggregator.  
Streaming/Sequence Filters or collates sequences of information. Must be placed after the first aggregator in a query. Cannot be used in live queries.
Multi-pass Processes data multiple times. Must be the first aggregator. Cannot be used in live queries.
Self-join Joins data within the same event set. Must be at the top-level of a query before any aggregators. Cannot be used in live queries.
Join Joins two sets of data together. Must be used at the top-level of a query. Should not be used in live queries or parsers.
Preamble Sets query parameters, what can be used in the query, or how the query is executed. Must be at the top of the query before all other statements.  

  • Transforming

    Transforming functions alter (transform) the events in the event set. They are broadly subdivided into the following function types:

    • Pure filter functions — accept or reject events; for example in() which matches events against a field contains a specific value.

    • Pure transforming functions — always output the same number of events as they get, modifying the events in the process. For example, ipLocation().

    • Field computation functions — compute a single value based on one or more input fields to an output field. For example format(). Often these functions perform an operation that could be applied to the a field through the := assignment operator.

    • Filtering and transforming functions — filters data and optionally transforms matching events. For example match() looks up data in a lookup file, filtering events that don't match and augmenting events that do.

    • Multi-event functions — these emit more than one event for the their input event set. For example, regex(), split() or copyEvent().

    During operation each transforming or filtering type function can be identified by the input/output event counts:

    Sub-Type UML Relationship Scala Model
    Filtering 1:0..1 Event => Boolean
    Transforming 1:1 Event => Event
    Sub-category 1:0..1 Value computation function: one event in, zero or one values out (put in a field on the event) (Event => Option[String]
    Filtering and transforming 1:0..1 Event => Option[Event]
    Splitting 1:0..* Event => Seq[Event]

  • Aggregating

    Aggregating functions summarize or simplify data. Because aggregating functions change and alter the information within the event sequence, they affect the way any following statements or functions operation. Aggregating functions are grouped into:

    • Basic mathematical functions like count() or sum().

    • Value collecting functions like collect() and selectFromMax().

    • Event collecting functions such as tail() or table(). These functions may imply that they are mandatory and stop the execution of the query; for example tail() implies that the query should complete after the function.

    • Grouping functions that group by a given key or time such as groupBy(), bucket(), or timeChart().

  • Asynchronous

    Asynchronous functions are those that provide asynchronous operation usually through accessing an external resource.

    Operation

    • Must occur after an aggregator.

    • Must occur on the top level of the independent subquery of which it is a part.

    Examples reverseDns()

  • Streaming/Sequence

    Streaming functions/stream-processing functions iterate over their input stream twice or more with an input stream that occurs in a well-defined order. This means that they must be placed in the result pipeline, after an aggregator.

    Operation

    • Must be placed in the result pipeline; after an aggregator.

    Examples accumulate(), partition(), neighbor()

  • Multi-pass

    Multi-pass functions are aggregating functions that iterate over their input set of events two or more times. If there is a prefix before the multi-pass query, then it is included in the iteration query.

    Operation

    • Must occur as the first aggregator.

    • Must occur on the top level of the independent subquery of which it is a part.

    • Must be run in an execution context where iterated querying is allowed.

    Examples correlate()

  • Self-join

    Self-join functions spawn independent subqueries that include the prefix of the query that came before the self-join function. For example:

    logscale
    host = /*server.com/
| selfJoin(field=connid,where=[{from="192.168.1.7"},{to="10.0.0.12"}])

    Is equivalent to running:

    logscale
    host = /*server.com/
| from="192.168.1.7"

    And:

    logscale
    host = /*server.com/
| to="10.0.0.12"

    Note that the query prefix is identical with both subqueries.

    Because this is the way the queries are executed, self joins must be at the top level of the query overall.

    Operation

    • Must be top-level in its own query.

    • Must be preceded only by transforming steps.

    • Must be executed in a context where independent subqueries may be spawned.

    Examples selfJoin(), selfJoinFilter()

  • Join

    Join functions (primarily join()) spawns two independent subqueries but do not have limitations on where they can be used and how the syntax before the function applies to the executed subqueries.

    join() operates as a filter function in the context of other operations, as in it filters the event set according to the execution and matching of the two subqueries executed, filtering the events in the process.

    Operation

    • Must be executed in the context of a given subquery

    • Cannot be used within a case, match or other function-based subqueries

    Examples setTimeInterval()

  • Preamble

    Preamble functions either define values to be used within the query or configure or change the metadata about the query, such as the start and end times. The effects of these functions are static and performed before the primary query starts.

    Preamble functions must be placed at the start of the query.

    Operation

    • Must be placed at the start of the primary query

    Examples setTimeInterval()