FAQ: How Does LogScale Handle Ingest Delays in Aggregate Alerts

Ingest Delays

Events are produced at an external system some time before they are sent to LogScale, which means there is a delay coming from outside of LogScale.

When an event is ingested into LogScale, it takes some time before it is available for searches. This is the ingest delay inside LogScale. There is no field on events reflecting the time when a log is available for searching. This ingest delay is the difference between event and ingest timestamps:

  • event timestamp — available in the @timestamp field, is when the event is produced that is, when it actually happens outside LogScale.

  • ingest timestamp — available in the @ingesttimestamp field, is when the event arrives in LogScale.

Queries and Ingest Delays

Normally, LogScale queries run on @timestamp, so they will be affected by ingest delays both outside and inside of LogScale. If the combined ingest delay is on average 20 minutes, that means that the newest event in LogScale will have @timestamp 20 minutes before now, on average. Therefore, any query running up until now will not see any data in the last 20 minutes of the search interval, and if it is a live query with a search interval of 20 minutes, it will miss events: this is because events with a larger combined ingest delay will be outside of the search interval when they are searchable in LogScale.

Queries running on @ingesttimestamp will only be affected by the ingest delay inside LogScale, but otherwise have the same problem.

Ingest Delays in Aggregate Alerts

For automated queries in general, there is a tradeoff between waiting for the full ingest delay to ensure a complete and correct query result, and fast results by acting on a non-empty query result as soon as it is available, no matter if it is complete.

When it comes to running queries for Aggregate Alerts in particular, LogScale handles alert triggers facing ingest delays as follows.

  • The UI allows to configure whether the query should run on @ingesttimestamp (when the event comes to LogScale) or on @timestamp (when the event happens outside). Read more about choosing the most appropriate timestamp for an aggregate alert at Alert Properties.

  • The way in which an aggregate alert chooses to trigger is set by the GraphQL API: it can be Immediate Mode or Complete Mode.

    • Immediate Mode means: the alert is triggered immediately even on incomplete results but will still wait up to 20 minutes for a result to trigger on; this will ensure that events that are up to 20 minutes delayed can still trigger the alert. With this mode, results might potentially be incomplete.

    • Complete Mode means: the alert waits for the internal ingest delay (no matter what timestamp is used) up to a maximum of 20 minutes before triggering, until there is a complete result to trigger on. When running the query on @ingesttimestamp, this will ensure complete results as long as the internal ingest delay is below 20 minutes.

  • Depending on the timestamp that's been configured in the UI, the API chooses a triggering mode by default:

    • When the alert runs on @timestamp, the default is Immediate Mode.

    • When the alert runs on @ingesttimestamp, the default is Complete Mode.

    The default triggering strategy can be changed through the API, by manipulating the triggerMode field in the createAggregateAlert or updateAggregateAlert GraphQL mutations.