Count Nested Array Elements

Count elements in nested arrays using the objectArray:eval() function with array:length()

Query

flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1>Augment Data] 2>Augment Data] 3[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 3 fill:#2ac76d; click 3 #examples-array-length-count-nested-array-elements-3
logscale
objectArray:eval(array="arrayToCount[]", asArray="tmpCountArray[]", function={ tmpCountArray:=true })
| array:length("tmpCountArray[]")
| array:drop("tmpCountArray[]")

Introduction

The array:length() function can be used to count the number of elements in a flat array field. However, when working with nested arrays — that is, arrays where each element is itself an array or an object containing multiple sub fields - the array:length() function cannot directly count the elements. In such cases, a workaround is needed.

The workaround uses the objectArray:eval() function to iterate over each element of the nested array and assign a temporary marker value to a new flat array. Once the flat temporary array has been populated with one marker per nested element, the array:length() function can find the length of the flat temporary array, effectively returning the number of elements in the original nested array. The temporary array is then removed by using the array:drop() function to keep the output clean.

In this example, the objectArray:eval() function is used to iterate over a nested array field named arrayToCount[] and populate a temporary flat array named tmpCountArray[] with a marker value of true for each element. This enables array:length() to count the number of nested array elements, which it otherwise cannot do directly on object arrays.

Example incoming data might look like this:

@rawstring@timestamp@timestamp.nanosarrayToCount[0].namearrayToCount[0].valuearrayToCount[1].namearrayToCount[1].valuearrayToCount[2].namearrayToCount[2].valuearrayToCount[3].namearrayToCount[3].valuehost
"arrayToCount[0].name"=alpha "arrayToCount[0].value"=10 "arrayToCount[1].name"=beta "arrayToCount[1].value"=20 "arrayToCount[2].name"=gamma "arrayToCount[2].value"=30 host="server-01"17749500257880alpha10beta20gamma30<no value><no value>server-01
"arrayToCount[0].name"=delta "arrayToCount[0].value"=5 "arrayToCount[1].name"=epsilon "arrayToCount[1].value"=15 "arrayToCount[2].name"=zeta "arrayToCount[2].value"=25 "arrayToCount[3].name"=eta "arrayToCount[3].value"=35 host="server-02"17749500257880delta5epsilon15zeta25eta35server-02
"arrayToCount[0].name"=theta "arrayToCount[0].value"=100 "arrayToCount[1].name"=iota "arrayToCount[1].value"=200 host="server-03"17749500257880theta100iota200<no value><no value><no value><no value>server-03
"arrayToCount[0].name"=kappa "arrayToCount[0].value"=7 host="server-04"17749500257880kappa7<no value><no value><no value><no value><no value><no value>server-04
"arrayToCount[0].name"=lambda "arrayToCount[0].value"=42 "arrayToCount[1].name"=mu "arrayToCount[1].value"=84 "arrayToCount[2].name"=nu "arrayToCount[2].value"=126 "arrayToCount[3].name"=xi "arrayToCount[3].value"=168 host="server-05"17749500257880lambda42mu84nu126xi168server-05
"arrayToCount[0].name"=omicron "arrayToCount[0].value"=3 "arrayToCount[1].name"=pi "arrayToCount[1].value"=6 "arrayToCount[2].name"=rho "arrayToCount[2].value"=9 host="server-06"17749500257880omicron3pi6rho9<no value><no value>server-06
"arrayToCount[0].name"=sigma "arrayToCount[0].value"=11 "arrayToCount[1].name"=tau "arrayToCount[1].value"=22 host="server-07"17749500257880sigma11tau22<no value><no value><no value><no value>server-07
"arrayToCount[0].name"=upsilon "arrayToCount[0].value"=55 "arrayToCount[1].name"=phi "arrayToCount[1].value"=110 "arrayToCount[2].name"=chi "arrayToCount[2].value"=165 "arrayToCount[3].name"=psi "arrayToCount[3].value"=220 host="server-08"17749500257880upsilon55phi110chi165psi220server-08

Each event contains a nested array field arrayToCount[], where each element is an object with sub-fields such as arrayToCount[n].name and arrayToCount[n].value. Because the array elements are objects rather than scalar values, array:length() alone cannot count them. The workaround described in this example resolves this limitation.

Step-by-Step

  1. Starting with the source repository events.

  2. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1>Augment Data] 2>Augment Data] 3[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 3 fill:#2ac76d; click 3 #examples-array-length-count-nested-array-elements-3 style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    objectArray:eval(array="arrayToCount[]", asArray="tmpCountArray[]", function={ tmpCountArray:=true })

    The objectArray:eval() function iterates over each element of the nested object array specified by the array parameter, here set to arrayToCount[].

    For each element found, it evaluates the expression provided in the function parameter — in this case assigning the value true to a field named tmpCountArray, which will be an element of the output array. Note that the field name must match the array name specified in the asArray parameter.

    The results are collected into a new flat array specified by the asArray parameter, named tmpCountArray[].

    After this step, tmpCountArray[] contains one entry of true per element in the original nested array arrayToCount[], effectively creating a flat, countable representation of the nested array's length. Without this intermediate step, array:length() would be unable to count the elements because it does not operate on object-type (nested) arrays.

    Note that the temporary array (in this example tmpCountArray[]) must have a unique name for each searched event. If the name is not unique, LogScale overwrites and drops existing fields from the event.

  3. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1>Augment Data] 2>Augment Data] 3[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 3 fill:#2ac76d; click 3 #examples-array-length-count-nested-array-elements-3 style 2 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    | array:length("tmpCountArray[]")

    The array:length() function counts the number of elements in the flat array tmpCountArray[] that was created in the previous step. Because tmpCountArray[] is a flat array with one true entry per original nested element, this count accurately reflects the number of elements in the original nested array arrayToCount[]. The function returns the result in a new field named _length by default.

  4. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1>Augment Data] 2>Augment Data] 3[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 3 fill:#2ac76d; click 3 #examples-array-length-count-nested-array-elements-3 style 3 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    | array:drop("tmpCountArray[]")

    Removes the temporary flat array tmpCountArray[] from the event. Since this array was only created as an intermediate step to enable counting, it is no longer needed once array:length() has returned the count in the _length field. Dropping it keeps the output clean and avoids polluting the results with intermediate fields.

  5. Event Result set.

Summary and Results

The query is used to count the number of elements in a nested object array field by using the objectArray:eval() function to create a temporary flat array of marker values, then applying the array:length() function to find the length of the flat temporary array, and finally removing the temporary array with array:drop().

This query is useful, for example, to determine how many sub-objects are present in a nested array field per event — such as counting the number of items in a shopping cart, the number of network interfaces on a host, or the number of findings in a security alert — in cases where the array:length() function alone cannot operate on object arrays.

Sample output from the incoming example data:

_lengtharrayToCount[0].namearrayToCount[0].valuearrayToCount[1].namearrayToCount[1].valuearrayToCount[2].namearrayToCount[2].valuearrayToCount[3].namearrayToCount[3].valuehost
3alpha10beta20gamma30<no value><no value>server-01
4delta5epsilon15zeta25eta35server-02
2theta100iota200<no value><no value><no value><no value>server-03
1kappa7<no value><no value><no value><no value><no value><no value>server-04
4lambda42mu84nu126xi168server-05
3omicron3pi6rho9<no value><no value>server-06
2sigma11tau22<no value><no value><no value><no value>server-07
4upsilon55phi110chi165psi220server-08

Note that the original nested array fields arrayToCount[] are still present in the output alongside host and _length — only the intermediate tmpCountArray[] field has been removed by array:drop().

Also note that the count in _length reflects only the number of populated elements in the nested array per event, so events with fewer sub-objects will return a lower count accordingly, as seen with server-04 which contains only a single nested array element and therefore returns a _length value of 1.