Query Processing

A basic overview of how a query is processed that describes where the operations take place, and how this affects processing and performance of the query is shown below, with the different steps aligned to the Query Pipeline:

flowchart TD A["Node A"] B["Node B"] C["Node C"] CA["Node A"] CB["Node C"] D["Node B"] E["Query Results"] A --Extract,Augment,Filter--> CA B --Extract,Augment,Filter--> CA B --Extract,Augment,Filter--> CB C --Extract,Augment,Filter--> CB CA --Aggregate--> D CB --Aggregate--> D D --Format and Results--> E
flowchart TD A["Node A"] B["Node B"] C["Node C"] CA["Node A"] CB["Node C"] D["Node B"] E["Query Results"] A --Extract,Augment,Filter--> CA B --Extract,Augment,Filter--> CA B --Extract,Augment,Filter--> CB C --Extract,Augment,Filter--> CB CA --Aggregate--> D CB --Aggregate--> D D --Format and Results--> E

When a query is submitted, the request is sent to only one node within a cluster of multiple machines. That single machine becomes the Query Coordinator and orchestrates the other nodes in the cluster (based on where the data is located) to perform the search.

To describe the diagram and query sequence:

  1. Data is retrieved for the repository from each node on which the data is found. Data in a repository is distributed around the cluster and so a given time range of data may access data from multiple nodes.

  2. When possible, any initial filtering, extraction or augmentation in the query (before aggregation) is performed on nodes where the data resides. This distributes work and increases the performance of the raw data processing across multiple nodes. This process is called Mapping.

  3. The resulting extracted and augmented data is then processed by one or more nodes during processing and aggregation. This combines the data into single processable dataset that can be updated, or summarized. As the data is simplified and aggregated the dataset becomes smaller. This process is called Reduction.

  4. Once the aggregated and summarized dataset has been combined, the information is returned to the Query Coordinator, which then performs any final formatting or processing on the reduced dataset.

This processing model ensures the best performance by distributed expensive operations, such as adding fields, or changing or calculating the value across hundreds of original source events. Performing this on a single node across millions of events would be time-consuming. By distributing the process, the operation can take place in parallel across each node.

The technical detail of how a query is processed is described in more detail in LogScale Internal Architecture.