Detect Event A Happening X Times Before Event B

Detect event A happening X times before event B (brute force attack) using the partition() function combined with groupBy()

Query

logscale

head()
| groupBy(
    key,
    function = partition(
        condition=test(status=="success"),
        split="after",
        [
            { status="failure" | count(as=failures) }, 
            range(@timestamp, as=timespan), 
            max(@timestamp), 
            selectLast(status)
        ]
    )
)
| failures >= 3
| status = "success"

Introduction

In this example, the partition() function is used with the groupBy() function to detect event A happening X times before event B (brute force attack).

The query will detect instances where there were 3 or more failed attempts followed by a successful attempt within the specified 10-second window.

Note that the partition() function must be used after an aggregator function to ensure event ordering. Also note that the events must be sorted in order by timestamp to prevent errors when running the query. It is possible to select any field to use as a timestamp.

Example incoming data might look like this:

@timestamp	key	status
1451606300200	c	failure
1451606300400	c	failure
1451606300600	c	failure
1451606301000	a	failure
1451606302000	a	failure
1451606302200	a	failure
1451606302300	a	failure
1451606302400	b	failure
1451606302500	a	failure
1451606302600	a	success
1451606303200	b	failure
1451606303300	c	success
1451606303400	b	failure
1451606304500	a	<no value>
1451606304600	c	failure
1451606304700	c	failure
1451606304800	c	failure

Step-by-Step

Starting with the source repository events.
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0{{Aggregate}} 1{{Aggregate}} 2[/Filter/] 3{{Aggregate}} result{{Result Set}} repo --> 0 0 --> 1 1 --> 2 2 --> 3 3 --> result style 0 fill:#ff0000,stroke-width:4px,stroke:#000;
logscale
```
head()
```
Selects the oldest events ordered by time.
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0{{Aggregate}} 1{{Aggregate}} 2[/Filter/] 3{{Aggregate}} result{{Result Set}} repo --> 0 0 --> 1 1 --> 2 2 --> 3 3 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
logscale
```
| groupBy(
    key,
    function = partition(
        condition=test(status=="success"),
        split="after",
        [
            { status="failure" | count(as=failures) }, 
            range(@timestamp, as=timespan), 
            max(@timestamp), 
            selectLast(status)
        ]
    )
)
```
Groups the events by a specified key (for example, a user ID or IP address), then splits the sequence of events after each successful event (where the condition status=="success").
For each partition, it counts the number of failure in status and stores it in the field failures, finds the range of timestamps in the partition, finds the newest timestamp, and finds the latest status to show if the partition ended with a success.
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0{{Aggregate}} 1{{Aggregate}} 2[/Filter/] 3{{Aggregate}} result{{Result Set}} repo --> 0 0 --> 1 1 --> 2 2 --> 3 3 --> result style 2 fill:#ff0000,stroke-width:4px,stroke:#000;
logscale
```
| failures >= 3
```
Filters for partitions that contained 3 or more failures.
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 0{{Aggregate}} 1{{Aggregate}} 2[/Filter/] 3{{Aggregate}} result{{Result Set}} repo --> 0 0 --> 1 1 --> 2 2 --> 3 3 --> result style 3 fill:#ff0000,stroke-width:4px,stroke:#000;
logscale
```
| status = "success"
```
Filters for partitions with the value success in the status field to ensure that the final status is a success.
Event Result set.

Summary and Results

The query is used to detect instances where there are 3 or more failed attempts followed by a successful attempt. The query can be used to detect a brute force attack where an attacker tries multiple times before succeeding. Note that the effectiveness of this query depends on the nature of your data and the typical patterns in your system.

Sample output from the incoming example data: