Filter Nested Array Elements and Label Missing Values

Filter an array of objects by key and set a fallback value when no element matches using the objectArray:eval() function

Query

flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[\Add Field/] result{{Result Set}} repo --> 1 1 --> result
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[\Add Field/] result{{Result Set}} repo --> 1 1 --> result
logscale
objectArray:eval("incidents[]", var=incident, asArray="highIds[]", function={ incident.severity = "high" | highIds := incident.id })
| default(field="highIds[0]", value="none")
| firstHighIncident := highIds[0]
| select([host, firstHighIncident])

Introduction

The array:filter() function works only on flat arrays. To filter an array of objects, use objectArray:eval() with a filtering non-aggregate pipeline in the function argument.

In this example, the objectArray:eval() function filters the incidents[] array for incidents with the severity high. The default() function then labels events without a high-severity incident as none.

Example incoming data might look like this:

@rawstring@timestamp@timestamp.nanoshostincidents[0].idincidents[0].severityincidents[1].idincidents[1].severityincidents[2].idincidents[2].severity
"host"=web-01 "incidents[0].id"=INC-101 "incidents[0].severity"=low "incidents[1].id"=INC-102 "incidents[1].severity"=high17749500257880web-01INC-101lowINC-102high<no value><no value>
"host"=web-02 "incidents[0].id"=INC-103 "incidents[0].severity"=low17749500257880web-02INC-103low<no value><no value><no value><no value>
"host"=web-03 "incidents[0].id"=INC-104 "incidents[0].severity"=medium "incidents[1].id"=INC-105 "incidents[1].severity"=low "incidents[2].id"=INC-106 "incidents[2].severity"=high17749500257880web-03INC-104mediumINC-105lowINC-106high
"host"=web-0417749500257880web-04<no value><no value><no value><no value><no value><no value>
"host"=web-05 "incidents[0].id"=INC-107 "incidents[0].severity"=high17749500257880web-05INC-107high<no value><no value><no value><no value>

Each event has a different number of incidents, and the high severity incident sits at a different position in the array. One event has no incidents at all.

Step-by-Step

  1. Starting with the source repository events.

  2. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[\Add Field/] result{{Result Set}} repo --> 1 1 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[\Add Field/] result{{Result Set}} repo --> 1 1 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    objectArray:eval("incidents[]", var=incident, asArray="highIds[]", function={ incident.severity = "high" | highIds := incident.id })
    | default(field="highIds[0]", value="none")
    | firstHighIncident := highIds[0]
    | select([host, firstHighIncident])

    Iterates over each element of incidents[] and assigns it to the variable incident. The filter incident.severity = "high" passes only matching elements. Only those elements write a value to the output array highIds[]. Elements that fail the filter produce no output.

    If no element matches, or the array does not exist, the output array does not exist. The default() function then sets highIds[0] to none. The final assignment copies the first entry into the firstHighIncident field. The select() function returns only the host and the first high-severity incident.

  3. Event Result set.

Summary and Results

The query returns one row per event. The high-severity incident is found at any position in the array, and events without one show none.

Sample output from the incoming example data:

hostfirstHighIncident
web-01INC-102
web-02none
web-03INC-106
web-04none
web-05INC-107