Drop Events Based on Parsing JSON Value
Query
case {
@rawstring="#*"
| dropEvent();
* }
Introduction
When parsing incoming data, it is sometimes the case that the data
includes 'commented' data, where,for example, the
#
character is used to identify
comments in files rather than real data. This example removes
those lines from the ingest process during parsing using the
dropEvent()
function to drop the entire event
from the ingest pipeline.
Step-by-Step
Starting with the source repository events.
- flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0[/Filter/] 1[/Filter/] result{{Result Set}} repo --> 0 0 --> 1 1 --> result style 0 fill:#ff0000,stroke-width:4px,stroke:#000;logscale
case { @rawstring="#*" | dropEvent();
Starts a
case
statement, with the first matching expression looking for the hash symbol in a line to indicate that it could be removed, then dropping the entire event usingdropEvent()
- flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0[/Filter/] 1[/Filter/] result{{Result Set}} repo --> 0 0 --> 1 1 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;logscale
* }
For all other lines, the
case
expression matches all other events and lets them through. Event Result set.
Summary and Results
This query is used to remove data at ingestion, in this example
data that matches a typical source construct (the comment). When
used within the parser pipeline, the
dropEvent()
function ensures that the data
is removed entirely from the query output, meaning that the
event data will not be stored in LogScale.