This function removes duplicate values from an array. The ordering of the first occurrence of each unique element is preserved.

Function Summary
Signaturearray:dedup(array, [asArray])
Output DataZero or One Event per Input Event
Function TypeTransformation
Parameters for array:dedup()
ParameterTypeInterpreted TypeRequiredDefault ValueDescription
array[a]arrayarrayrequired   The array to deduplicate elements in.
asArraystringstringoptional[b] input array Where to output the deduplicated array.

[a] The parameter name array can be omitted.

[b] Optional parameters use their default value unless explicitly set.

Hide omitted argument names for this function

Show omitted argument names for this function

For instance, given a rawstring with array a[]:

Raw Events
{ "a":["horse", "fish", "horse", "cow", "cow"] }

Specify the name of the array field from which to remove duplicates:

logscale
parseJson()
| array:dedup("a[]")

The duplicate item will be removed:

a[0]a[1]a[2]
horsefishcow

array:dedup() Examples

Click + next to an example below to get the full details.

Deduplicate Values in Array

Remove duplicate values in an array using array:dedup()

Query
logscale
parseJson()
| array:dedup("emails[]")
Introduction

In this example, the function removes duplicates in the array emails[].

Example incoming data might look like this:

Raw Events
{"emails": ["john@mail.com", "admin@mail.com", "jane@mail.com", "admin@mail.com"]}
Step-by-Step

  1. Starting with the source repository events.

  2. logscale
    parseJson()

    Parses the incoming data to identify JSON values and converts them into a usable field.

  3. logscale
    | array:dedup("emails[]")

    Removes duplicate values in the array emails[].

  4. Event Result set.

Summary and Results

The query is used to remove duplicate values in an array.

Sample output from the incoming example data:

emails[0]emails[1]emails[2]
john@mail.comadmin@mail.comjane@mail.com