Event Timestamps
Event data ingested into LogScale will always contain a date and time in the form of a timestamp. Timestamps are critical to the way events are retrieved from a repository, and LogScale generates different timestamps as events are ingested:
The timestamp of the original event if this information was able to be extracted from the original event.
The timestamp of the original event if this information was able to be extracted from the original event, recorded at sub-millisecond level of precision. The timestamp is either extracted from the raw event log data (which will often contain a date and time) or be defined by the incoming data.
This field is only present if the accuracy of the timestamp during ingestion could be identified at this accuracy.
The ingest timestamp records when an event was ingested and stored on disk. Because data ingest involves reading, transferring, and parsing data, the ingest timestamp will typically be later than an event timestamp by at least a few milliseconds. This can be used to determine the delay between the original timestamp of an event and ingestion, which is a useful indicator of ingest performance.
When processing data, if a @timestamp cannot be extracted, then the @ingesttimestamp will be used instead.
During ingest, the @timestamp field is populated using the following rules:
The ingest timestamp is recorded during the ingest process. This is always generated and stored during ingest.
The parser will attempt to extract the timestamp from the original event data or log entry. Both external connectors, and different parsers within LogScale (such as
parseTimestamp()) will attempt to extract the timestamp of the original event.If the event timestamp cannot be extracted, then @timestamp will match the value of @ingesttimestamp. This ensures that there is always a @timestamp field for each event.
This process ensures that all events in a repository contain a @timestamp field to be used for ordering and querying data.
When performing certain searches or actions, whether a user employs @timestamp or @ingesttimestamp has an impact on what events are returned and in what sequence. When using @ingesttimestamp in place of @timestamp, consider the following:
Identifying Ingest Lag
Often, the data being ingested into LogScale will have been generated recently. This means the data will have been written to a log, then shipped to LogScale using either the Falcon LogScale Collector or another ingest method almost immediately. The difference between @timestamp and @ingesttimestamp is called ingest lag, which can be used to determine the ingest performance of the LogScale cluster.
For example, the query:
diff := (@ingesttimestamp - @timestamp) / duration("1m")
| timechart(span=1h, function=avg(field=diff))Will display a timechart of the ingest lag.