How-To: How to Compare and Alert Historical Ingest

This article describes how to:

Measure billed LogScale SaaS ingest data in bytes
Compare recent ingested volumes to the same day of week

This will allow you to answer the question, "are my ingest volumes today normal, higher, or lower, when compared to the same day of week over the last few weeks?"

Step 1: Identify Billed Ingest Volumes

All queries should run in the humio-usage repo.

Falcon Data (FDR/1st Party Data)

For Falcon data, also called FDR (Falcon Data Replicator) data, you can identify billed ingest volume with this query:

logscale

#repo=humio-usage
| falconIngestAfterFieldRemovalSize > 0
| sum(falconIngestAfterFieldRemovalSize, as=bytes)
| unit:convert(field=bytes, from="B", to="GB")

3rd Party Data

For 3rd party (non-Falcon) data, you can identify billed ingest volume with this query:

logscale

#repo=humio-usage 
| ingestAfterFieldRemovalSize > 0
| sum(ingestAfterFieldRemovalSize)
| unit:convert(field=bytes, from="B", to="GB")

Falcon Data and 3rd Party Data

If you are ingesting both Falcon data and 3rd party data, you can use the below query to measure billed ingest volume across both data types:

logscale

repo=humio-usage
// create on billed_bytes field for both data types (Falcon and 3rd party)
ingestAfterFieldRemovalSize > 0 OR falconIngestAfterFieldRemovalSize > 0
 case{
falconIngestAfterFieldRemovalSize > 0 | billed_bytes :=
   falconIngestAfterFieldRemovalSize;
ingestAfterFieldRemovalSize > 0 | billed_bytes :=
   ingestAfterFieldRemovalSize;
billed_bytes := 0}

Step 2: Day of Week Benchmarking - Creating Historical Buckets

The next step is to create buckets so that your recent ingest volumes are compared to the same day of week. This is important for log volume patterns than change throughout the day, and throughout the week.

For example, if you alert on hourly ingest spikes, your ingest volume alert may fire every Monday morning when users or employees log on to systems and cause normal spikes in log volumes.

This approach accounts for these scenarios - for example, we will compare this Monday morning from 8am-9am to the average log volume on previous Monday mornings at 8am-9am, eliminating false positive alerting.

Case Statement for Buckets

We will use the case{} statement to create the buckets, looking at averages over the last 4 weeks.

Note

Note that this query is designed to scan the last 29 days (today + last 4 weeks of data)

logscale

case {
test(@timestamp > (now()-(duration("1d")))) | last_24hrs :=
   billed_bytes; // events less than 24 hours old
test(@timestamp > (now()-(duration("8d")))) | test(@timestamp <
   (now()-(duration("7d")))) | week1 := billed_bytes; // less than 8 day old, more than
   7 days old (same weekday last week)
test(@timestamp > (now()-(duration("15d")))) | test(@timestamp <
   (now()-(duration("14d")))) | week2 := billed_bytes; // less than 15 day old, more
   than 14 days old (same weekday 2 weeks ago)
test(@timestamp > (now()-(duration("22d")))) | test(@timestamp <
   (now()-(duration("21d")))) | week3 := billed_bytes; // less than 22 day old, more
   than 21 days old (same weekday 3 weeks ago)
test(@timestamp < (now()-(duration("29d")))) | test(@timestamp <
   (now()-(duration("28d")))) | week4 := billed_bytes; // less than 29 day old, more
   than 28 days old (same weekday 4 weeks ago)
}

Aggregate Across Buckets

logscale

// aggregate over the last 24 hours, and in the last 4 weeks on the same
   day of week
groupBy(field=repo,function=[
sum(field=last_24hrs,as=last_24hrs),
sum(field=week1,as=week1),
sum(field=week2,as=week2),
sum(field=week3,as=week3),
sum(field=week4,as=week4)
])

Historical Averages

We will now find the average historical values for the same day of week over each of the past 4 weeks

logscale

// find average bytes of the last 4 weeks, same day of week
| weekday_avg:=((week1+week2+week3+week4)/4)
// convert from bytes to GB
| unit:convert(field=last_24hrs, from="B", to="GB")
| unit:convert(field=weekday_avg, from="B", to="GB")
// compare GB from the last 24 hours to the average of the last 4 weeks, same day of week
| change := (last_24hrs-weekday_avg)
| pct_change := ((last_24hrs-weekday_avg)/weekday_avg)*100
// formatting
| pct_change := format("%.2f%%", field = pct_change)
| table([repo,last_24hrs,weekday_avg,change,pct_change])
| sort([last_24hrs,pct_change])
// optional - for alerting - add filtering conditions
//| last_24hrs >= 10 // at least 10 GB ingested in the last 24 hours
//| pct_change >= 50 OR pct_change <= 50

Total and Percent Change vs. Historical

We can then calculate the total and percent changes for recent data compared to the historical average:

logscale

| change := (last_24hrs-weekday_avg)
| pct_change := ((last_24hrs-weekday_avg)/weekday_avg)*100
// formatting
| pct_change := format("%.2f%%", field = pct_change)

Formatting

For readability and presentation, we will then apply formatting operators like table() and sort():

logscale

| table([repo,last_24hrs,weekday_avg,change,pct_change])
| sort([last_24hrs,pct_change])
// optional - for alerting - add filtering conditions
//| last_24hrs >= 10 // at least 10 GB ingested in the last 24 hours
//| pct_change >= 50 OR pct_change <= 50

Alerting Conditions

If you plan to run this for alerting purposes, you can use the below filter examples.

This example looks for where any repo that has ingested at least 10 GB
We can also add a filter for a spike or drop in billed volume of at least 50%

logscale

// optional - for alerting - add filtering conditions
//| last_24hrs >= 10 // at least 10 GB ingested in the last 24 hours
//| pct_change >= 50 OR pct_change <= 50

Step 3: Full Query and Alerting

Putting all the pieces together, we get the below query. We can set this up as an alert as shown below.

Scheduled Search Settings

It is recommended to use the following settings when alerting with this query:

This query works for LogScale SaaS clusters
Run this query in #repo=humio-usage
Time Interval: This query uses the last 29 days, you can update the query to only consider the last 1, 2, or 3 weeks, instead of 4, if preferred
This query can be scheduled to run once per hour, once per day, for example, or adjusted to any frequency you prefer
You can use a wait time (Max Wait Time field) of a few minutes to account for any ingest delays

logscale

#repo=humio-usage
// create on billed_bytes field for both data types (Falcon and 3rd party)
|  ingestAfterFieldRemovalSize > 0 OR falconIngestAfterFieldRemovalSize > 0
|case{
 falconIngestAfterFieldRemovalSize > 0 | billed_bytes := falconIngestAfterFieldRemovalSize;
 ingestAfterFieldRemovalSize > 0 | billed_bytes := ingestAfterFieldRemovalSize;
 billed_bytes := 0
 }
// bucket data into historical groups
| case {
 test(@timestamp > (now()-(duration("1d")))) | last_24hrs :=  billed_bytes; // events less than 24 hours old
 test(@timestamp > (now()-(duration("8d")))) | test(@timestamp <
    (now()-(duration("7d")))) | week1 := billed_bytes; // less than 8 day old, more than 7 days old (same weekday last week)
 test(@timestamp > (now()-(duration("15d")))) | test(@timestamp <
    (now()-(duration("14d")))) | week2 := billed_bytes; // less than 15 day old, more than 14 days old (same weekday 2 weeks ago)
 test(@timestamp > (now()-(duration("22d")))) | test(@timestamp <
    (now()-(duration("21d")))) | week3 := billed_bytes; // less than 22 day old, more than 21 days old (same weekday 3 weeks ago)
 test(@timestamp < (now()-(duration("29d")))) | test(@timestamp <
    (now()-(duration("28d")))) | week4 := billed_bytes; // less than 29 day old, more than 28 days old (same weekday 4 weeks ago)
 }
// aggregate over the last 24 hours, and in the last 4 weeks on the same day of week
| groupBy(field=repo,function=[
 sum(field=last_24hrs,as=last_24hrs),
 sum(field=week1,as=week1),
 sum(field=week2,as=week2),
 sum(field=week3,as=week3),
 sum(field=week4,as=week4)
 ])
// find average bytes of the last 4 weeks, same day of week
| weekday_avg:=((week1+week2+week3+week4)/4)
// convert from bytes to GB
| unit:convert(field=last_24hrs, from="B", to="GB")
| unit:convert(field=weekday_avg, from="B", to="GB")
// compare GB from the last 24 hours to the average of the last 4 weeks, same day of week
| change := (last_24hrs-weekday_avg)
| pct_change := ((last_24hrs-weekday_avg)/weekday_avg)*100
// formatting
| pct_change := format("%.2f%%", field = pct_change)
| table([repo,last_24hrs,weekday_avg,change,pct_change])
| sort([last_24hrs,pct_change])
// optional - for alerting - add filtering conditions
//| last_24hrs >= 10 // at least 10 GB ingested in the last 24 hours
//| pct_change >= 50 OR pct_change <= 50 
| change := (last_24hrs-weekday_avg)
| pct_change := ((last_24hrs-weekday_avg)/weekday_avg)*100
// formatting
| table([repo,last_24hrs,weekday_avg,change,pct_change])
| sort([last_24hrs,pct_change])
// optional - for alerting - add filtering conditions
| last_24hrs >= 10 // at least 10 GB ingested in the last 24 hours
| pct_change >= 10 OR pct_change <= -10 
// optional - for alerting - add filtering conditions
//| last_24hrs >= 10 // at least 10 GB ingested in the last 24 hours
//| pct_change >= 50 OR pct_change <= 50
// formatting
| pct_change := format("%.2f%%", field = pct_change)

A sample trigger configuration is shown below:

Knowledge Base