readPersistedAggregation() reads events from a repository (view) with tags that match the tag parameter. The function then filters the events by the specified time range and outputs the results.

Persisted Aggregation with Rollup Support is a variant of persisted aggregation. At read time, you can retrieve the aggregation at a coarser time resolution than the schedule interval provides. This flexibility is especially helpful for persisted aggregations that store metrics.

For example, you create a Persisted Aggregation with Rollup Support that stores average incoming bytes on an hourly schedule. From that same persisted aggregation, you can retrieve daily, weekly, and monthly average incoming bytes. To do so, pass 1d, 1w, or 1month as the span parameter. For a detailed explanation, see readPersistedAggregation() Syntax Examples.

For a Persisted Aggregation with Rollup Support, readPersistedAggregation() completes these actions in order:

This behavior resumes the persisted aggregation query from the first aggregator at the time resolution that you request.

This example demonstrates basic usage of readPersistedAggregation(), reading all events from a persisted aggregation by specifying a tag and view.

In this scenario, a persisted aggregation exists with the tag myTag in the view myView. The query retrieves all events stored under that tag. Only events matching the specified tag are returned; events from other tags in the same view are excluded.

readPersistedAggregation("myTag", view="myView")

If event data looks like this:

it would return:

This example demonstrates filtering events using an absolute time interval specified as Unix millisecond timestamps. Only events with a @timestamp value in the range [start, end) are returned.

In this scenario, the persisted aggregation contains four events with timestamps spread across a short interval. The query specifies an absolute start of 1451606301002 and an absolute end of 1451606301010. Events with timestamps before start or at and after end are excluded from the result.

readPersistedAggregation("myTag", view="myView", start=1451606301002, end=1451606301010)

If event data looks like this:

it would return:

This example demonstrates filtering events using relative durations for start and end. Relative values are interpreted as offsets back from the outer query's end time.

In this scenario, the outer query has an end time of 1451606301020. Specifying start=17ms and end=3ms translates to the absolute range [1451606301003, 1451606301017), that is, from 17 milliseconds before the query end time up to 3 milliseconds before the query end time. Events outside this range are excluded.

readPersistedAggregation("myTag", view="myView", start=17ms, end=3ms)

If event data looks like this:

it would return:

This example demonstrates the behaviour of readPersistedAggregation() when the specified persisted aggregation contains no events.

When the persisted aggregation identified by tag myTag in view myView has no stored events, the function returns an empty result set. No error is raised; the query simply produces no output rows.

readPersistedAggregation("myTag", view="myView")

If the persisted aggregation contains no events, the query returns an empty result set with no rows.

This example enriches the results of a persisted aggregation using a lookup file.

In this scenario, the persisted aggregation has the tag myTag in the view myView, and stores one row per service and humio_cluster_id. Because readPersistedAggregation() emits ordinary events, those rows can be enriched from a lookup file before being processed further. A per-cluster cost rate is looked up with match(), the strict parameter is set to false so that rows whose cluster is absent from the file are kept, default() supplies a fallback rate for those rows, and the costs are then summed per service.

The lookup file search_cost_rates.csv contains the following:

"humio_cluster_id","cost_per_cpu_hour"
"cluster-a","0.01"

readPersistedAggregation("myTag", view="myView")
| match(file="search_cost_rates.csv", field=humio_cluster_id, column=humio_cluster_id, include=[cost_per_cpu_hour], strict=false)
| default(value="0.004", field=cost_per_cpu_hour)
| total_cost_usd := total_cpu_ms / 3600000 * cost_per_cpu_hour
| groupBy(service, function=sum(total_cost_usd, as=cost_usd))

If event data looks like this:

it would return:

The search service ran for two CPU hours on cluster-a at the rate from the file, which is 0.01, giving 0.02. The cluster cluster-b is not in the file, so the ingest service is priced at the default rate of 0.004 for its single CPU hour. Without the strict parameter set to false, the ingest result would have been dropped instead.

This example rolls up an avg() aggregate to a coarser span.

In this scenario, a persisted aggregation with rollup support has the tag myTag in the view myView, runs on an hourly schedule, and stores avg(x). One event arrives at 00:01, in the first hour, and three events arrive at 01:01, 01:02 and 01:03, in the second hour. A span of 2h merges both hourly states into a single result. A rollup is not a plain addition of the results of each scheduled run: every aggregate is merged by its own rule, so that the rolled-up result is the same as the one the query would have produced had it been scheduled at the coarser interval. For avg(), this means that the rolled-up average is weighted by how many events each run saw. It is not an average of averages.

readPersistedAggregation("myTag", view="myView", span="2h", timezone="UTC")

If event data looks like this:

it would return:

Each scheduled run stored the following state, which is what a span of 1h would return:

"_avg","@timestamp"
"10","1451606400000"
"2","1451610000000"

The two hours hold four events with a total sum of 16, so the rolled-up average is 4. An average of the two hourly averages would instead have returned 6.

This example rolls up a groupBy() aggregate to a coarser span.

In this scenario, a persisted aggregation with rollup support has the tag myTag in the view myView, runs on a 10-minute schedule, and stores groupBy(x, function=count()). Events arrive at 00:01, 00:02 and 00:03 in the first run, at 00:11 and 00:12 in the second run, and at 00:21 and 00:22 in the third run, so the value alpha appears in the first two runs but not in the third. A span of 30m merges all three runs. Groups are merged by key, and keys that only some of the runs saw are carried through as they are.

readPersistedAggregation("myTag", view="myView", span="30m", timezone="UTC")

If event data looks like this:

it would return:

Each scheduled run stored the following state, which is what a span of 10m would return:

"x","_count","@timestamp"
"alpha","2","1451606400000"
"beta","1","1451606400000"
"alpha","1","1451607000000"
"beta","1","1451607000000"
"beta","2","1451607600000"

The value alpha was not seen in the last run, but it is still part of the rolled-up result.

This example rolls up a stats() aggregate to a coarser span.

In this scenario, a persisted aggregation with rollup support has the tag myTag in the view myView, runs on a 10-minute schedule, and stores stats([count(), sum(x)]). Events arrive at 00:01 and 00:02 in the first run, at 00:11 in the second run, and at 00:21 and 00:22 in the third run. A span of 30m merges all three runs into a single result. Each aggregate is merged by its own rule, at every level of nesting.

readPersistedAggregation("myTag", view="myView", span="30m", timezone="UTC")

If event data looks like this:

it would return:

Each scheduled run stored the following state, which is what a span of 10m would return:

"_count","_sum","@timestamp"
"2","30","1451606400000"
"1","5","1451607000000"
"2","3","1451607600000"

The _count aggregate is merged by addition and the _sum aggregate is merged by addition, giving 5 and 38 respectively. Aggregates may also be nested, as in groupBy(x, function=stats([count(), sum(y)])) or groupBy(x, function=groupBy(y, function=count())). Nested results are merged the same way, and group keys must match at every level for their results to be merged.

This example post-processes a rolled-up result in the reading query.

In this scenario, a persisted aggregation with rollup support has the tag myTag in the view myView, runs on a 10-minute schedule, and stores groupBy(x, function=count()), with events arriving at 00:01, 00:02 and 00:03 in the first run, at 00:11 and 00:12 in the second run, and at 00:21 and 00:22 in the third run. Everything after the first aggregate is re-run from the rolled-up result, so a persisted aggregation whose query continues past the first aggregate is post-processed at the requested span rather than at the schedule interval. Post-processing can also be done in the reading query, since readPersistedAggregation() emits ordinary events. Here, the rolled-up groups are sorted by count in descending order.

readPersistedAggregation("myTag", view="myView", span="30m", timezone="UTC")
| sort(_count, order=desc)

If event data looks like this:

it would return:

The sort() function is applied to the merged 30-minute result, not to the individual 10-minute runs.

This example reads a persisted aggregation for a time range in which no events were seen.

In this scenario, the persisted aggregation has the tag myTag in the view myView, runs on an hourly schedule, and stores count(). There are no events in the requested time range, so there is no event data. When the persisted aggregation saw no events in the requested time range, the query still runs on the empty state.

readPersistedAggregation("myTag", view="myView", span="1h", timezone="UTC")

it would return:

This example enriches a rolled-up result with a lookup file.

In this scenario, a persisted aggregation with rollup support has the tag myTag in the view myView, runs on an hourly schedule, and stores groupBy([service, humio_cluster_id], function=sum(cpu_ms, as=total_cpu_ms)). Events arrive at 00:01 and 00:02 in the first hour, and at 01:01 and 01:02 in the second hour. A span of 2h merges both hourly states before the match() enrichment is applied. Enrichment works on a rolled-up result too. The match() function here runs after the first aggregate, which is allowed in the reading query even though it is not allowed after the first aggregate inside the persisted aggregation itself.

The lookup file search_cost_rates.csv contains the following:

"humio_cluster_id","cost_per_cpu_hour"
"cluster-a","0.01"

readPersistedAggregation("myTag", view="myView", span="2h", timezone="UTC")
| match(file="search_cost_rates.csv", field=humio_cluster_id, column=humio_cluster_id, include=[cost_per_cpu_hour], strict=false)
| default(value="0.004", field=cost_per_cpu_hour)
| total_cost_usd := total_cpu_ms / 3600000 * cost_per_cpu_hour
| groupBy(service, function=sum(total_cost_usd, as=cost_usd))

If event data looks like this:

it would return:

Each scheduled run stored the following state, which is what a span of 1h would return:

"service","humio_cluster_id","total_cpu_ms","@timestamp"
"search","cluster-a","3600000","1451606400000"
"ingest","cluster-b","1800000","1451606400000"
"search","cluster-a","3600000","1451610000000"
"ingest","cluster-b","1800000","1451610000000"

Rolled up to two hours, the search service on cluster-a accounts for 7200000 ms and the ingest service on cluster-b for 3600000 ms, so the costs are the same as in the first example above. Enriching a rolled-up result therefore produces the same figures as enriching an equivalent pre-aggregated total.

Persisted aggregations are saved in the UTC timezone by default. Timezone conversion is not supported when reading persisted aggregations. The rollup operation therefore operates on UTC. Rolling data up to a day is equivalent to rolling up to a UTC day.

When you read a Persisted Aggregation with Rollup Support, you must explicitly provide timezone. Currently, only UTC is supported.

The span parameter currently accepts only fixed-duration time values. This means a day is always 24 hours, a week is always 7 days, a month is always 30 days, and a year is always 365 days, as described in Relative Time Syntax.

Calendar time is not supported when rolling up persisted aggregations.

Click + next to an example below to get the full details.

readPersistedAggregation("myTag", view="myView")

readPersistedAggregation("myTag", view="myView", span="2h", timezone="UTC")

avg(responseTime)