Modifying an Event Set

Some operations within a query will modify an event set. For example, the list of events may filter individual events, replace the event set, and/or restructure the events into a new set. Events may also be combined with other events, leading to a new unified set during the operation of a join.

Understanding that the entire event set can be changed during a query is important. When the event set changes, a number of other factors also change:

  • The number of events may be different, either increased or decreased.

  • The fields in each event may be different.

  • Both of the above may be true, which means the entire event set will look different.

The most common example of an event set changing is the aggregation of data. The function groupBy() can completely convert the content of the event set. However, there are several different types of operations that make changes, including the following:

  • Filtering

    Filters allow for a basic process of selecting individual events from the data. Filters may be text filters, like the following example:

    logscale
    /Incident/

    This example filters events that include this string.

    Or you can filter by a specific parsed field:

    logscale
    eventName="Incident"

    A regular expression or wildcard is also an option:

    logscale
    eventName=/Malicious.*/

    This can be thought of as filtering out individual events from the event set, resulting in a smaller, more manageable set.

    flowchart LR %%{init:{'flowchart':{'nodeSpacing': 10, 'rankSpacing': 50,"defaultRenderer": "elk"}}}%% subgraph One direction LR a["event1"] b["event2"] c["event3"] aa["event4"] end f["Filter"] subgraph Two direction LR d["event1"] e["event2"] end One --> f --> Two style a height:40,line-height:1; style b height:40,line-height:1; style c height:40,line-height:1; style aa height:40,line-height:1; style d height:40,line-height:1; style e height:40,line-height:1;
    flowchart LR %%{init:{'flowchart':{'nodeSpacing': 10, 'rankSpacing': 50,"defaultRenderer": "elk"}}}%% subgraph One direction LR a["event1"] b["event2"] c["event3"] aa["event4"] end f["Filter"] subgraph Two direction LR d["event1"] e["event2"] end One --> f --> Two style a height:40,line-height:1; style b height:40,line-height:1; style c height:40,line-height:1; style aa height:40,line-height:1; style d height:40,line-height:1; style e height:40,line-height:1;

    Effect on Events:

    • The effect this has on events is that it removes entire events according to the filter (to include, or exclude) based on a value in the event.

  • Aggregation

    Aggregation combines data with one or more fields and optionally performs an aggregating function across fields, such as adding (sum), counting, or average value over time.

    Aggregation can be modeled as shown in the figure below:

    flowchart LR %%{init:{'flowchart':{'nodeSpacing': 10, 'rankSpacing': 50,"defaultRenderer": "elk"}}}%% subgraph source a["KeyA"] b["KeyB"] c["KeyB"] d["KeyC"] e["KeyA"] end agg["Aggregator"] h["KeyA"] j["KeyB"] k["KeyC"] a --> agg b --> agg c --> agg d --> agg e --> agg agg --> h agg --> j agg --> k linkStyle 1,2,6 stroke:#f00,stroke-width:4px linkStyle 3,7 stroke:#0f0,stroke-width:4px linkStyle 0,4,5 stroke:#00f,stroke-width:4px style a height:40,line-height:1; style b height:40,line-height:1; style c height:40,line-height:1; style d height:40,line-height:1; style e height:40,line-height:1; style h height:40,line-height:1; style j height:40,line-height:1; style k height:40,line-height:1;
    flowchart LR %%{init:{'flowchart':{'nodeSpacing': 10, 'rankSpacing': 50,"defaultRenderer": "elk"}}}%% subgraph source a["KeyA"] b["KeyB"] c["KeyB"] d["KeyC"] e["KeyA"] end agg["Aggregator"] h["KeyA"] j["KeyB"] k["KeyC"] a --> agg b --> agg c --> agg d --> agg e --> agg agg --> h agg --> j agg --> k linkStyle 1,2,6 stroke:#f00,stroke-width:4px linkStyle 3,7 stroke:#0f0,stroke-width:4px linkStyle 0,4,5 stroke:#00f,stroke-width:4px style a height:40,line-height:1; style b height:40,line-height:1; style c height:40,line-height:1; style d height:40,line-height:1; style e height:40,line-height:1; style h height:40,line-height:1; style j height:40,line-height:1; style k height:40,line-height:1;

    Effect on Events:

    • Removes the original event set.

    • Creates a new event set, with a new selection of fields according to the selected aggregating values.

  • Joins

    Joins combine data according to a given rule that matches values across fields on both sides, creating a new event set:

    flowchart LR %%{init:{'flowchart':{'nodeSpacing': 10, 'rankSpacing': 10,"defaultRenderer": "elk"}}}%% subgraph queryA a["EventA"] b["EventB"] c["EventC"] end subgraph queryB x["EventX"] y["EventY"] z["EventZ"] end join["Join"] h["EventA+EventY"] j["EventC+EventX"] a --> join b --> join c --> join x --> join y --> join z --> join join --> h join --> j linkStyle 0,5,6 stroke:#f00,stroke-width:4px linkStyle 3,2,7 stroke:#00f,stroke-width:4px style a height:40,line-height:1; style b height:40,line-height:1; style c height:40,line-height:1; style x height:40,line-height:1; style y height:40,line-height:1; style z height:40,line-height:1; style h height:40,line-height:1; style j height:40,line-height:1;
    flowchart LR %%{init:{'flowchart':{'nodeSpacing': 10, 'rankSpacing': 10,"defaultRenderer": "elk"}}}%% subgraph queryA a["EventA"] b["EventB"] c["EventC"] end subgraph queryB x["EventX"] y["EventY"] z["EventZ"] end join["Join"] h["EventA+EventY"] j["EventC+EventX"] a --> join b --> join c --> join x --> join y --> join z --> join join --> h join --> j linkStyle 0,5,6 stroke:#f00,stroke-width:4px linkStyle 3,2,7 stroke:#00f,stroke-width:4px style a height:40,line-height:1; style b height:40,line-height:1; style c height:40,line-height:1; style x height:40,line-height:1; style y height:40,line-height:1; style z height:40,line-height:1; style h height:40,line-height:1; style j height:40,line-height:1;

    Effect on Events:

    • Augments the original event set.

    • Creates a new event that may or may not be related to existing events (depending on the join type).