Scheduling Details

Understanding how persisted aggregations schedule and execute queries is important for configuring them correctly.

Available Intervals

Persisted aggregations run on a fixed schedule. You select from a set of predefined intervals in the Aggregation interval setting.

The interval you select must not exceed the retention period of the source repository. The retention period of the destination repository must be no shorter than the selected interval.

Lookback Period

The lookback period is not configurable. For each run, LogScale calculates results from the current interval's start time back by exactly one interval. For example, an hourly aggregation running at 14:00 covers the period from 13:00 to 14:00. The interval is inclusive of the start time and exclusive of the end time.

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% gantt dateFormat HH:mm axisFormat %H:%M tickInterval 1hour todayMarker off section Interval A Search window (13:00-14:00) :active, 13:00, 60m Execution at 14:00 :milestone, 14:00, 0m section Interval B Search window (14:00-15:00) :active, 14:00, 60m Execution at 15:00 :milestone, 15:00, 0m
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% gantt dateFormat HH:mm axisFormat %H:%M tickInterval 1hour todayMarker off section Interval A Search window (13:00-14:00) :active, 13:00, 60m Execution at 14:00 :milestone, 14:00, 0m section Interval B Search window (14:00-15:00) :active, 14:00, 60m Execution at 15:00 :milestone, 15:00, 0m

The diagram shows two consecutive hourly intervals. Each execution processes exactly one interval of data, with the search window ending at the execution time.

Offset

You can configure an offset to delay query execution and allow time for late-arriving events to be ingested before the interval is calculated. This is configured in the properties panel using the Wait time (seconds) field in Advanced settings. For example, a 5-minute interval with a 10-minute offset calculates results from 10 minutes ago rather than the current interval. Events that arrive after the offset window has passed are not captured for that interval. For data sources with unpredictable or long latency, ingest time is recommended instead.

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% gantt dateFormat HH:mm axisFormat %H:%M tickInterval 5minute todayMarker off section Without Offset Search window (5min) :active, 13:55, 5m Jitter :done, 14:00, 1m Query execution :crit, 14:01, 30s section With 10min Offset Search window (5min) :active, 13:45, 5m Wait time (10min offset) :done, 13:50, 10m Jitter :done, 14:00, 1m Query execution :crit, 14:01, 30s
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% gantt dateFormat HH:mm axisFormat %H:%M tickInterval 5minute todayMarker off section Without Offset Search window (5min) :active, 13:55, 5m Jitter :done, 14:00, 1m Query execution :crit, 14:01, 30s section With 10min Offset Search window (5min) :active, 13:45, 5m Wait time (10min offset) :done, 13:50, 10m Jitter :done, 14:00, 1m Query execution :crit, 14:01, 30s

The diagram compares a 5-minute interval aggregation scheduled at 14:00 with and without a 10-minute offset. The offset shifts the 5-minute data search window to end 10 minutes earlier (at 13:50 instead of 14:00), allowing late-arriving events to be ingested before the window closes. The system jitter and query execution occur at the same scheduled time in both cases.

The offset does not impact the system's execution jitter, which is applied independently to distribute query execution load evenly and ensure consistent execution times.

Backfill

For persisted aggregations, backfill duration refers to a one-time historical data generation process that runs when the aggregation is first created.

When you create a persisted aggregation, you can define a backfill duration using the Historical backfill period field in Advanced settings.