Query Pipeline Sequence

Although all queries are different, many queries follow a typical order of operation and steps, and some may have significance in terms of how different syntax or functions operate.

flowchart LR A["Source Data"] B["Extract"] C["Filter"] D["Augment"] E["Aggregate"] F["Results"] A --Time Selection--> B B --Identify Data--> C C --Reduce Event Set--> D D --Format--> E E --Summarize--> F
flowchart LR A["Source Data"] B["Extract"] C["Filter"] D["Augment"] E["Aggregate"] F["Results"] A --Time Selection--> B B --Identify Data--> C C --Reduce Event Set--> D D --Format--> E E --Summarize--> F

Each of these steps have different semantics and implications during the query, and may affect both the functions that can be used, and the overall performance of the query. Taking each item in turn:

Step Description Output Fields Output Events
Data Source The selected repository or view, or another data source. Always includes all fields in the ingested event set. This could be a single repository, a view covering multiple repositories, or a view that filters the incoming events (using a prefilter). The list of events selected from the repository is dependent on the time selection either in the UI or search interface, or the API. Limits the number of events in the set according to the @timestamp or @ingesttimestamp field. All Filtered by time selection
Extract Extract additional data from the incoming raw files or existing fields like timestamps, converting JSON data, or extracting more specific data from existing fields. All, plus additional extracted data All [a]
Filter Filter individual events by selecting specific data, using a regular expression, or a text wildcard match. All Filtered
Augment Add or calculate new values and fields into the data to make processing, filtering or summarizing information earlier. This might also include looking up data in a lookup file, location information, or reverse DNS lookups. All, plus additional augmented data All [b]
Aggregate Summarize incoming events, grouping them together by one or more fields. New summarized fields Reduced set according to aggregation
Format Info Format the information for display, for example formatting dates or numbers. Updated field information All [c]
Results Choose an output or display method, for example assembling the content into buckets for display within a graph, or using the WorldMap or Sankey widgets. Updated field Simplified event set

[a] All events from the incoming event set

[b] All events from the incoming event set

[c] All events from the incoming event set

While most queries will follow most of these stages, others may only include a few, or may perform multiple iterations of each step, like formatting after filtering or aggregating before further aggregation and formatting is applied. An example of this is described in Query Statement Order. For background on how the query is processed internally, see Query Processing. For details on the different operations that can occur when processing, see Modifying Event Data, Modifying an Event Set, and Finalizing an Event Set.

For more information on different data sources that might be available for querying, see Event Data Sources.