Events are produced in an external system some time before they are sent to LogScale, which means there is a delay coming from outside of LogScale. This ingest delay is the difference between event and ingest timestamps:

When an event is ingested into LogScale, it takes some time before it is available for searches. This is the ingest delay inside LogScale. There is no field on events reflecting the time when a log is available for searching.

There are two metrics that are helpful when considering ingest delays outside and inside LogScale: external-ingest-delay and event-latency-repo. The metric external-ingest-delay shows the delay between log creation (@timestamp) and LogScale ingestion (@ingesttimestamp) for all events in a single repository within a 5 minute interval; in other words, the ingest delay outside of LogScale. If your triggers run on a view connected to multiple repositories, you need to aggregate the metric over all of the repositories.

The event-latency-repo metric shows ingest delay inside LogScale for a single repository, including time spent in parsers, updating live queries, and adding events to blocks for segment files. If you use this metric for triggers running on a view, you must aggregate the metric over all repositories to which the view is connected.

For automated queries in general, there is a tradeoff between waiting for the full ingest delay to ensure a complete and correct query result, and fast results by acting on a non-empty query result as soon as it is available, no matter if it is complete.

The UI allows you to configure whether the query should run on @ingesttimestamp (when the event comes to LogScale) or on @timestamp (when the event happens outside). Read more about choosing the most appropriate timestamp for a trigger at Trigger properties.

Normally, LogScale queries run on @timestamp, so they will be affected by ingest delays both outside and inside of LogScale. If, for example, the combined ingest delay is maximum 20 minutes, that means that the last 20 minutes of the search interval can be missing events.

Queries running on @ingesttimestamp will only be affected by the ingest delay inside LogScale, but otherwise have the same problem.

When it comes to running queries for Aggregate alerts in particular, LogScale handles alert triggers facing ingest delays as follows:

Based on whether CompleteMode or ImmediateMode is selected and the timestamp used for the aggregate alert, ingest delay is handled as follows:

In order for a scheduled search to find the events you want to see, you must consider the maximum ingest delay, the timestamp type used by the scheduled search, and what your scheduled search query is doing.

If, for example, you want to set up scheduled searches to monitor error rates on your Linux servers, consider the following scenario. You set up monitoring for the syslog server that receives audit logs from you Linux servers. It is then sent to LogScale through the Log Collector. When you look at the event search, you notice that it can take up to 15 minutes for a log captured on the Linux server to reach LogScale and get an assigned @ingesttimestamp as shown in the diagram below.

It can then take up to 5 minutes for it to be searchable in LogScale as shown below.

If the scheduled search or aggregate alert is running on @timestamp, you need to know the maximum ingest delay to be able to set an accurate value for Delay run and Time window on a scheduled search, or Time window on an aggregate alert. The following process describes how to find the maximum ingest delay.

Start by finding the maximum ingest delay for both ingest delay inside LogScale (event-latency-repo) and ingest delay outside LogScale (external-ingest-delay). To do this, run the following query in the humio-metrics repo using a search interval that covers a time period that is representative for the ingest delay of the cluster, changing the name for the metric to get delay for both:

name="event-latency-repo"
  | repo=<REPOSITORY>
  | max(max)

where you replace <REPOSITORY> with the name of the repository.

Once you have these two values, add them together to get the total maximum ingest delay. You will use this later in the process. If your query runs on a view connected to multiple repositories, you need to get the total maximum ingest delay of each repository.

Take the filter part of your query and append it with:

@error!=true
delay:=@ingesttimestamp - @timestamp
max(delay)

The @error!=true line is to exclude events with parser errors, as those errors could be errors parsing the timestamp, causing @timestamp to be wrong and thus provide an incorrect calculation. For the aggregate function, use max(), avg(), or percentile() depending on which value you want to know (maximum delay, average delay, or percentile of delay).

So for example, take this query:

event_platform=win
  | event_simpleName=ProcessRollup2
  | FileName=cmd.exe
  | CommandLine=*whoami*
  | count(as=execution_count)
  | execution_count >= 5
  | sort(execution_count)
  | head(10)

and add the code snippet above to the filter part of the query, so that it is:

event_platform=win
| event_simpleName=ProcessRollup2
| FileName=cmd.exe
| CommandLine=*whoami*
| @error!=true
| delay:=@ingesttimestamp - @timestamp
| max(delay)
...

Run the query with this calculation over a long time period. Use the produced value for the ingest delay outside LogScale to guide your decision for setting the Delay run and Time window on the scheduled search, or Time window on an aggregate alert. If you want all events, then it should be the produced value from the query above (difference between @ingesttimestamp and @timestamp) plus the event-latency-repo metric value which is the ingest delay inside of LogScale. This also means that results are delayed by that much. If you just want, e.g., 99% of events, use, for example, percentile(), max(), or avg() to specify that.