Calculate and Sort Ingest Lag Times

Analyze the time difference between event occurrence and ingestion using the select() function with sort()

Query

flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[[Expression]] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result

logscale

select([#repo, #Vendor, #type, @timestamp, @ingesttimestamp])
| ingest_lag_in_mins := ((@ingesttimestamp-@timestamp)/1000)/60
| sort(ingest_lag_in_mins, limit=20000)

Introduction

The select() function can be used to specify which fields to include in the results, allowing for focused analysis of specific data points. Combined with field creation and sorting, it enables detailed investigation of system performance metrics such as ingest lag.

In this example, the select() function is used to analyze the time difference between when events occurred and when they were ingested into LogScale, helping identify potential ingestion delays or performance issues.

Example incoming data might look like this:

@timestamp	@ingesttimestamp	#repo	#vendor	#type
2025-11-05T10:00:00.000Z	2025-11-05T10:01:30.000Z	windows-events	Microsoft	SecurityEvent
2025-11-05T10:00:15.000Z	2025-11-05T10:02:45.000Z	linux-syslog	Linux	SystemLog
2025-11-05T10:00:30.000Z	2025-11-05T10:01:15.000Z	network-logs	Cisco	FirewallLog
2025-11-05T10:00:45.000Z	2025-11-05T10:05:45.000Z	endpoint-logs	CrowdStrike	ProcessCreate
2025-11-05T10:01:00.000Z	2025-11-05T10:01:45.000Z	cloud-logs	AWS	CloudTrail
2025-11-05T10:01:15.000Z	2025-11-05T10:04:15.000Z	database-logs	Oracle	AuditLog
2025-11-05T10:01:30.000Z	2025-11-05T10:02:00.000Z	windows-events	Microsoft	LoginEvent
2025-11-05T10:01:45.000Z	2025-11-05T10:03:45.000Z	linux-syslog	Linux	AuthLog

Step-by-Step

Starting with the source repository events.
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[[Expression]] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
logscale
```
select([#repo, #Vendor, #type, @timestamp, @ingesttimestamp])
```
Selects the relevant fields for analysis: #repo, #vendor, #type, @timestamp, and @ingesttimestamp.
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[[Expression]] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 2 fill:#ff0000,stroke-width:4px,stroke:#000;
logscale
```
| ingest_lag_in_mins := ((@ingesttimestamp-@timestamp)/1000)/60
```
Creates a new field named ingest_lag_in_mins that calculates the time difference between @ingesttimestamp and @timestamp in minutes. The calculation first converts the millisecond difference to seconds (divided by 1000) and then to minutes (divided by 60).
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[[Expression]] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 3 fill:#ff0000,stroke-width:4px,stroke:#000;
logscale
```
| sort(ingest_lag_in_mins, limit=20000)
```
Sorts the results based on the ingest_lag_in_mins field in ascending order. The limit parameter is set to 20000 to ensure all relevant events are included in the analysis.
Event Result set.

Summary and Results

The query is used to calculate and analyze the time difference between when events occur and when they are ingested into LogScale, providing visibility into potential ingestion delays or performance issues.

This query is useful, for example, to troubleshoot correlation rule effectiveness, monitor data pipeline health, ensure real-time analysis capabilities, and identify potential bottlenecks in the data ingestion process.

Sample output from the incoming example data:

#repo	#vendor	#type	@timestamp	@ingesttimestamp	ingest_lag_in_mins
network-logs	Cisco	FirewallLog	2025-11-05T10:00:30.000Z	2025-11-05T10:01:15.000Z	0.75
windows-events	Microsoft	SecurityEvent	2025-11-05T10:00:00.000Z	2025-11-05T10:01:30.000Z	1.5
cloud-logs	AWS	CloudTrail	2025-11-05T10:01:00.000Z	2025-11-05T10:01:45.000Z	0.75
windows-events	Microsoft	LoginEvent	2025-11-05T10:01:30.000Z	2025-11-05T10:02:00.000Z	0.5
linux-syslog	Linux	SystemLog	2025-11-05T10:00:15.000Z	2025-11-05T10:02:45.000Z	2.5
linux-syslog	Linux	AuthLog	2025-11-05T10:01:45.000Z	2025-11-05T10:03:45.000Z	2.0
database-logs	Oracle	AuditLog	2025-11-05T10:01:15.000Z	2025-11-05T10:04:15.000Z	3.0
endpoint-logs	CrowdStrike	ProcessCreate	2025-11-05T10:00:45.000Z	2025-11-05T10:05:45.000Z	5.0

Note that the ingest lag is calculated in minutes for easier analysis. Lower values indicate better ingestion performance.

Examples Library