Filter Out Based on a Non-Matching Regular Expression (Function Format)

Query

logscale
responsesize > 2000
| not regex("/falcon-logscale-.*/",field=url)

Introduction

Typically a regular expression is used to filter events based on a value that the regular expression matches. The opposite can also be achieved, filtering events by those that do not match the regular expression by using the function form of a regula expression match.

This example searches weblog data looking for large log entries that are larger than a specified size but not in a specific directory.

Step-by-Step

  1. Starting with the source repository events.

  2. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0[/Filter/] 1[/Filter/] result{{Result Set}} repo --> 0 0 --> 1 1 --> result style 0 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    responsesize > 2000

    Fine

  3. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0[/Filter/] 1[/Filter/] result{{Result Set}} repo --> 0 0 --> 1 1 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    | not regex("/falcon-logscale-.*/",field=url)

    Negates the regular expression match, here filtering out any filename that contains the prefix falcon-logscale, but returning all other matching URLs.

  4. Event Result set.

Summary and Results

For example, given the following events:

@timestamp#repo#type@id@ingesttimestamp@rawstring@timestamp.nanos@timezoneclienthttpversionmethodresponsesizestatuscodeurluserid
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_0_6401_17199827432024-07-03T04:59:41192.168.1.240 - - [03/07/2024:04:59:03 +0000] "GET /js/htmllinkhelp.js HTTP/1.1" 200 230Z192.168.1.240HTTP/1.1GET23200/js/htmllinkhelp.js-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_0_6400_17199827432024-07-03T04:59:41192.168.1.24 - - [03/07/2024:04:59:03 +0000] "GET /data-analysis-1.100/css-images/external-link.svg HTTP/1.1" 200 10720Z192.168.1.24HTTP/1.1GET1072200/data-analysis-1.100/css-images/external-link.svg-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_0_6399_17199827432024-07-03T04:59:41192.168.1.209 - - [03/07/2024:04:59:03 +0000] "GET /js/htmllinkhelp.js HTTP/1.1" 304 -0Z192.168.1.209HTTP/1.1GET-304/js/htmllinkhelp.js-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_0_6398_17199827432024-07-03T04:59:41192.168.1.39 - - [03/07/2024:04:59:03 +0000] "GET /data-analysis/js/java.min.js HTTP/1.1" 304 -0Z192.168.1.39HTTP/1.1GET-304/data-analysis/js/java.min.js-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_0_6397_17199827432024-07-03T04:59:41192.168.1.62 - - [03/07/2024:04:59:03 +0000] "GET /falcon-logscale-cloud/js/php.min.js HTTP/1.1" 200 63970Z192.168.1.62HTTP/1.1GET6397200/falcon-logscale-cloud/js/php.min.js-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_0_6396_17199827432024-07-03T04:59:41192.168.1.206 - - [03/07/2024:04:59:03 +0000] "GET /integrations/js/theme.js HTTP/1.1" 200 148450Z192.168.1.206HTTP/1.1GET14845200/integrations/js/theme.js-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_0_6395_17199827432024-07-03T04:59:41192.168.1.1 - - [03/07/2024:04:59:03 +0000] "GET /data-analysis/js/json.min.js HTTP/1.1" 200 4960Z192.168.1.1HTTP/1.1GET496200/data-analysis/js/json.min.js-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_0_6394_17199827432024-07-03T04:59:41192.168.1.252 - - [03/07/2024:04:59:03 +0000] "GET /falcon-logscale-cloud/js/java.min.js HTTP/1.1" 200 27390Z192.168.1.252HTTP/1.1GET2739200/falcon-logscale-cloud/js/java.min.js-

Might return the following values:

@timestamp#repo#type@id@ingesttimestamp@rawstring@timestamp.nanos@timezoneclienthttpversionmethodresponsesizestatuscodeurluserid
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_2_6541_17199827432024-07-03T05:03:48192.168.1.231 - - [03/07/2024:04:59:03 +0000] "GET /logscale-repo-schema/js/corp.js HTTP/1.1" 200 186450Z192.168.1.231HTTP/1.1GET18645200/logscale-repo-schema/js/corp.js-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_2_6538_17199827432024-07-03T05:03:48192.168.1.69 - - [03/07/2024:04:59:03 +0000] "GET /data-analysis-1.100/images/dashboards.png HTTP/1.1" 200 1525900Z192.168.1.69HTTP/1.1GET152590200/data-analysis-1.100/images/dashboards.png-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_2_6535_17199827432024-07-03T05:03:47192.168.1.154 - - [03/07/2024:04:59:03 +0000] "GET /integrations/js/theme.js HTTP/1.1" 200 148450Z192.168.1.154HTTP/1.1GET14845200/integrations/js/theme.js-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_2_6534_17199827432024-07-03T05:03:47192.168.1.58 - - [03/07/2024:04:59:03 +0000] "GET /integrations/images/extrahop.png HTTP/1.1" 200 102610Z192.168.1.58HTTP/1.1GET10261200/integrations/images/extrahop.png-
2024-07-03T04:59:03weblogshttpsimpMqHKxw2QoBPZyNqbJRRs4ECC_2_6527_17199827432024-07-03T05:03:47192.168.1.164 - - [03/07/2024:04:59:03 +0000] "GET /integrations/images/zeek.png HTTP/1.1" 200 43920Z192.168.1.164HTTP/1.1GET4392200/integrations/images/zeek.png-