Query Statement Order
When filtering data, some individual rows will be filtered from the event data set. When augmenting events, additional fields will be added (in orange). When aggregating, the data is summarized and quantified.
It's important to note that even aggregated data is represented as a set of events with fields of data. Because the event set at every point within the query is in the same format, event and event fields can be adjusted at any time. This means that aggregated results can be further adjusted or filtered before the final result set is created.
For example, a query could perform the following actions on an HTTP access log, in this order:
Filter the original events to select the IP addresses.
Augment the data by formatting the time into the date format.
Augment the IP address by looking up the hostname.
Augment the data by adding a field with the IP address location.
Aggregate the data by counting the number of times the event occurred, summarized by the IP address.
However, because the data is always represented as a list of events, the order could be changed. The following demonstrates this:
Aggregate the data by counting the number of times the event occurred, summarized by the IP address.
Augment the data by formatting the time into the date format.
Augment the data by adding a field with the IP address location.
Filter the original events to select the IP addresses.
Augment the IP address by looking up the hostname.
While the result would be the same, the performance would be affected. For faster processing with less drain on resources, it's always better to filter events first to reduce the number of events that must be processed or augmented.
There are multiple potential ways of achieving the same result set, but the ordering can have a significant impact on the performance of the query.
These basic principles are used in all queries throughout LogScale in order to filter, augment, and structure the information for processing.