Drop Events Based on Parsing JSON Value

Query

logscale
case {
@rawstring="#*"
| dropEvent();
* }

Introduction

When parsing incoming data, it is sometimes the case that the data includes 'commented' data, where,for example, the # character is used to identify comments in files rather than real data. This example removes those lines from the ingest process during parsing using the dropEvent() function to drop the entire event from the ingest pipeline.

Step-by-Step

  1. Starting with the source repository events.

  2. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0[/Filter/] 1[/Filter/] result{{Result Set}} repo --> 0 0 --> 1 1 --> result style 0 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    case {
@rawstring="#*"
| dropEvent();

    Starts a case statement, with the first matching expression looking for the hash symbol in a line to indicate that it could be removed, then dropping the entire event using dropEvent()

  3. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0[/Filter/] 1[/Filter/] result{{Result Set}} repo --> 0 0 --> 1 1 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    * }

    For all other lines, the case expression matches all other events and lets them through.

  4. Event Result set.

Summary and Results

This query is used to remove data at ingestion, in this example data that matches a typical source construct (the comment). When used within the parser pipeline, the dropEvent() function ensures that the data is removed entirely from the query output, meaning that the event data will not be stored in LogScale.