Query Pipeline
Queries within LogScale consist of one or more expressions, with
each expression separated by a pipe (|).
The query operates as a pipeline evaluating the stream of incoming events,
and as the events proceed through the pipeline, the events may be changed,
filtered, or modified.
In Event Data During Queries, an overview of the sequence of events and processing is provided. The actual sequence and processing may be more complex than this, and some steps in the process may be repeated multiple times within the overall sequence:
Queries process a collection of events through the query pipeline.
Events are processed in order from the event stream; initially that order will be based on the @timestamp.
Each event in the event stream is processed individually by each query fragment.
The effect of this process is that each stage of the pipeline can perform a number of different operations:
Add or modify the data in each event, including completely replacing the event data with new fields and information.
Add or modify the set of events, including completely replacing the event set for a new list of events.
When event sets are processed, the input may be a set of raw data - but the output could contain the same, more, or fewer events. Furthermore, each event could contain the same, more, or fewer fields than the original data. Typically, the aim when querying is to reduce the original event set from thousands of millions of individual events to a simplified or summarized set of data.