Deduplicate Values in Array

Remove duplicate values in an array using array:dedup()

Query

logscale
parseJson()
| array:dedup("emails[]")

Introduction

The array:dedup() function can be used to remove duplicate values in an array.

In this example, the function removes duplicates in the array emails[].

Example incoming data might look like this:

Raw Events
{"emails": ["john@mail.com", "admin@mail.com", "jane@mail.com", "admin@mail.com"]}

Step-by-Step

  1. Starting with the source repository events.

  2. logscale
    parseJson()

    Parses the incoming data to identify JSON values and converts them into a usable field.

  3. logscale
    | array:dedup("emails[]")

    Removes duplicate values in the array emails[].

  4. Event Result set.

Summary and Results

The query is used to remove duplicate values in an array.

Sample output from the incoming example data:

emails[0]emails[1]emails[2]
john@mail.comadmin@mail.comjane@mail.com