Ignore Commented Lines During Parsing by Dropping Events
Query
parseJson()
| case { someField = "some_value"
| dropEvent(); * }
| parseTimestamp(field=@timestamp)
Introduction
The dropEvent()
function is often used within
parsers to drop events during parsing that do not need to be
ingested. The following example shows how to filter events as part
of a parser by matching a particular field value from being
ingested.
Step-by-Step
Starting with the source repository events.
- flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0[/Filter/] 1[/Filter/] 2[/Filter/] result{{Result Set}} repo --> 0 0 --> 1 1 --> 2 2 --> result style 0 fill:#ff0000,stroke-width:4px,stroke:#000;logscale
parseJson()
Parses the incoming data to identify JSON values and converts them into a usable field.
- flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0[/Filter/] 1[/Filter/] 2[/Filter/] result{{Result Set}} repo --> 0 0 --> 1 1 --> 2 2 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;logscale
| case { someField = "some_value" | dropEvent(); * }
Starts a
case
statement, with the first matching expression identifying a field value in the extracted JSON field from the returned results. Then drops the event. This has the effect of terminating the parsing for this event, as there is no more data to be processed. - flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0[/Filter/] 1[/Filter/] 2[/Filter/] result{{Result Set}} repo --> 0 0 --> 1 1 --> 2 2 --> result style 2 fill:#ff0000,stroke-width:4px,stroke:#000;logscale
| parseTimestamp(field=@timestamp)
Parses the timestamp from the @timestamp field for all other events that do not match the JSON value.
Event Result set.
Summary and Results
This query is used to drop events at ingestion. When used within
the parser pipeline, the dropEvent()
is a
simple and practical way of eliminating events during the
parsing of incoming data.