Function Location

Different functions are suitable for use only within different parts of the overall query pipeline. These requirements go beyond the best practice approach for writing the most efficient queries, such as executing filter functions as early as possible in the pipeline, and instead relate to the data, format or content of the pipeline at the time the function is called.

Function locations can be required or influenced by:

  • Functions That Must Be First

    There are functions that must be the first used in a query (after any filtering or event creation):

    • For example, the correlate() actually iterates over the incoming event stream multiple times and therefore needs to operate on the original event data.

    • The selfJoin() function creates independent subqueries that include the prefix query content as part of the subquery. This effectively means that the query:

      logscale
      foo 
| selfJoin({bar})

      Executes the subquery:

      logscale
      foo 
| bar

    • The readFile() function effectively replaces the incoming event set with the contents of the file it reads.

    • The setTimeInterval() controls the timespan of the events, and must come before any other functions or statements to ensure that the correct time range is selected for the incoming events.

    • Other functions that must occur early in the query, including potentially being the first item are those that directly access the original event data, such as eventSize(), eventInternals() and eventFieldCount(). Note that these functions must also have direct access to an original stored event.

  • Functions That Must Come Before Other Functions

    Some functions and operations must process the events directly, or modify or alter the event set so much that they would completely rewrite the event set.

    The collect() function often has to be used before aggregators because it collects information before the event is modified by the aggregation. Calling it after the aggregation, the field values will have disappeared and cannot be collected.

  • Functions That Must Come After Other Functions

    There are some functions that can only be used after other functions or function types for them to operate. For example, most sequence functions such as accumulate() must be used after an aggregator. Often, this process is related to ensuring the data being processed has been summarized to a level that can be easily processed without hitting a limit within the query engine. Some functions need to keep the entire event set in memory in order to process the results; requiring that the operation is after an aggregator helps keep the event size within the limits.

  • Functions That Can Be Used Within Other Functions

    Some functions are used primarily within the confines of another function to help process the event. For example, single aggregation functions such as count() or sum() are best used within multi-aggregators like groupBy() or array:eval().

  • Functions That Work Differently Depending on Position

    The now() function for example, if placed before the first aggregate function, is evaluated the first time the query sees the event. If placed after the first aggregate function, it is evaluated continuously, and gives the live value of the current system time, which can divert between LogScale nodes.

This information is displayed as part of the function summary for each function to help with placement. Within the user interface, notifications will normally be provided to highlight a fault in the query.