Event Time Selection and Sequence

When accessing information in a repository, data starts from the raw events in the repository. By default, when reading the data from the repository, the following event time selection and sequence concepts apply:

  • Time Window

    Queries are executed based on a time window defined outside of the query, either through the user interface or when making a query via the API. Whether querying through the user interface or an API, the events queries are always selected based on a timespan.

    If an explicit time is not given, the system default is to return the most recent events, allowing querying of both recent activity and also historical data. Querying recent activity is useful for security checks, while querying historical data is useful for correlations, trends, and/or historical analysis.

    Times selected are inclusive for both the start and end of the given time specification.

    For example, specifying the time 08:00:00.000 to -08:15:00.000 could show events including events with a time of 08:15:00.000.

    This also means that if you perform two different searches 08:00:00.000 to -08:15:00.000 and 08:15:00.000 to -08:30:00.000, you could get the same events twice.

    This approach is different compared to many other data environments where the query must define the selection of the event data.

    The time window is specified outside the query, and because queries are executed in a variety of different environments, the same query can be used in multiple scenarios without modification in order to run within a different timespan. This is particularly important when a query is executed as part of an alert or scheduled search - the time span of the events to be queried will be constantly adjusting to execute the query over recent events.

  • Time Specifications

    Time selection in queries can be relative or absolute.

    Events within the defined time window are always first supplied in time order, starting with the oldest event first.

    Typically the timestamp of the event (as parsed from the original log entry) is used, but LogScale can also use the ingest timestamp.

    When the timespan of a query is specified, the specification of that timespan is either Relative or Absolute:

    • Relative

      Relative time is defined as a time that is either relative to the current time, or within a certain absolute or anchor time. Relative times are always specified by declaring the quantity and measurement. For example, 7 days would indicate that the timespan would be from 7 days ago to now.

      When performing a query via the UI or an API, a relative time specifies a unit of time relative to now. Specifying an interval of 10 minutes will return events from 10 minutes ago to now, and a greater amount of relative time will return more events across the specified longer timespan. For example, specifying 1 month will return results be from one month ago until now.

      In queries and functions, relative time may be relative to another time set in the function, and may even be relative to another relative time.

      The relative time always accounts for the exact time. If the current time as 14:34:43 on 15th August 2025, then 7 days will select events starting from 14:34:43 on 8th August 2025.

      The time is significant; LogScale does not start from the beginning of the day, but strictly from the defined time span.

    • Absolute

      Absolute time specifies the full date and time of the query span, and are used when looking at historical information or when searching for data within a specific time range to identify a particular value. An example of what an absolute time range might look like is a range that starts at 04:30:03 on the 10th of August 2025, and ends at 04:35:00 on the 11th of August 2025.

  • Time Span

    The time is specific across an explicit start and end time (reklative or absolute), it defines the timespan of the original source events before any queries are executed against those events.

  • Event Sequence

    Events are returned in Time Order, from oldest event to newest according to the time selection.

    For example, when selecting data from the last 5 minutes at 12:44, events will be returned in order 12:39, then 12:40, 12:41, and so on.

    The order of events in this format may affect the way queries are executed or filters are applied, especially with functions or operations that make use of the sequence. This sequencing can be useful when looking for sequencing or ordering of events, such as a bad login before a successful one.

  • Performance Optimization for Time-based Queries

    __ls_shortname stores the most recently ingested data in the fastest available storage method, as opposed to older data, which is typically stored on slower storage systems. As a result, the most recent data is always the fastest to be returned, and optimizes querying the most recent information. This enables LogScale to execute searches, alerts, and queries on the most recent incoming data and with the best performance, while still allowing queries on older and/or historical data.

    Older data is stored in bucket storage and automatically recovered when it is being queried. If data cannot be stored or the data is beyond its retention time, the data is removed- retention time is determined by the data's timestamp. This ensures that LogScale has the latest data for analysis, and prioritizes the performance of this most recent data. For more information, see Data Retention and Expiration.

Advantages of the Time-Based Approach

There are some advantages of the time-based approach for repository data:

  • Performance Benefits

    • Quick access to recent data

      Because all data is timestamped, LogScale keeps the most recent data in memory and older data on local disk and in bucket storage (slower storage), therefore prioritizing the most recent information.

    • Efficient data retention management

      Stored data can have a retention specification that determines when data expires or will be deleted from the system. Because the data is organized by time, expiring the old data is easy to determine.

    • Optimized storage utilization

      LogScale intelligently uses different storage mechanisms, including memory, local disk/bucket storage, and automatic data transfer to comparatively slower storage forms to maximize system efficacy.

    • Faster query response times

      With the most recent data in memory, queries on local data can be executed very quickly.

  • Operational Benefits

    • Real-time monitoring capabilities

      Because the data is time-based, performing queries on the most recent or active data is easy.

    • Historical trend analysis

      With all data organized by time, looking at the same value or trend over minutes, hours, or days is straightforward.

    • Easy data lifecycle management

  • Analytical Benefits

    • Temporal correlation of events

      Data provided in a time series order makes finding relationships between events easier- for example, an event that occurs before another or determining whether a sequence of events occurred over a period of time.

    • Pattern detection over time

      Looking for explicit patterns over time is easier without having to search, filter, and then sort the data, particularly in a security or metric context. Likewise, looking for a specific pattern within a time range on specific data, or within a range that correlates with any event, is also much easier.

    • Incident timeline reconstruction

      Correlating sequences of events especially in a security or failure context. For example, identifying a bad login and then failed attack vector.

    • Capacity planning based on historical data

      Looking for increasing/decreasing values to identify a trend and make predictions relies on searching events in the sequence they were reported.

  • Business Benefits

    • Better incident response

      Because data can be identified by and associated with a specific time, investigating multiple events across different sources is a simpler process. Identifying specific incidents and correlated events and looking at live data/data from a specific identified point of time allows for querying to be applied to a very specific time segment. Finding the information for a specific time period can also be faster, allowing quicker responses to an incident.

    • Improved troubleshooting

      Knowing the time of an incident or problem makes tracing back from that time and following the sequence of events and/or metrics that led to that event easier. With a time-series structure, making those connections and tracking the changes is also more straightforward.

    • More accurate compliance reporting

      By correlating against a time, compliance and auditing processes provide an accurate trace of different events and when they occurred.

    • Enhanced security monitoring

      Timing in security monitoring is key. Knowing when the event occurred is critical, and timely efficient querying of the data once it has been identified is even more important. LogScale allows for quick access to information, efficient methods of querying, and identifying the event and sequence.