Queries

The LogScale Query Language (LQL) provides a full set of tools for selecting, filtering, and formatting data. The basic structure of the LQL is similar to the pipe-based processing within the Unix or Linux shell environment. The input to the query is a list of events, selected by a time span. That time span is either explicit, or relative, for example, the last 5 minutes.

Each set of events is then processed and either filtered, select events from the lists, augmented with additional data or information, or formatted. Formatting can include adding units to numerical values, formatting dates, or aggregated data to provide counts, sums or averages over the data.

For each statement in the query, an event list, consisting of a set of fields, is provided as an input. The query statement performs and action, and then outputs an updated list of events to provide the input for the next statement.

For example, when processing web access logs, you might run the query

logscale
url = /.js$/
| groupBy(statuscode)

The first line filters the incoming events, selecting only events where the requested URL ends in .js (JavaScript files). The result list of events is then provided as the input to the groupBy(), which is an aggregate function that summarizes the data, in this case using the statuscode. This will take a unique list of the statuscode values from the set of filtered events and then count the number of events with that value. The output might look like this:

statuscode_count
200288
304110
40411

Queries are used throughout LogScale to query and access data, including from the internal system repositories used to manage information. Internal usage data, for example, is stored in LogScale and is viewable within the Web UI.

Aggregations

Aggregation expressions are always function calls. These functions can combine their input into a new structure or emit new events into the output stream.

A query becomes an aggregation query if it uses at least one aggregate function like sum(), count() or avg().

For example, the query count() takes a stream of events as its input, and produces a single record containing a _count field.

Below are some examples:

logscale
loglevel = ERROR 
| timechart()
logscale
x := y * 2 
| bucket(function=sum(x))

Live Queries

Live queries are executed by LogScale during ingestion and are used to provide 'live' information from the ingested data. Because the data is processed when it is ingested, the results can be returned quicker than if the system waited for the data to be ingested and stored on disk. This is particularly useful in dashboards, but also for built-in alerting system.

When executing a live query, the time interval is always a time window relative to now, for example the last 5 minutes or the last day. Live queries are made up of the historical element and the live element, with the live element directly querying data that is being actively ingested.

Live queries are always being executed if they are being actively or regularly polled for results. This means that results are immediately available when opening a dashboard that is using live queries. If live queries are not polled the server will stop them at some point. The following describes when live queries are stopped:

  • Live queries on dashboards will be kept running on the server for 3 days if they are not polled.

  • Live queries are kept running for 1 hour if they are not polled.

  • All live queries that have not completed their historical search part will be closed if they are not polled for 30 seconds, like historical queries are.

The UI will try to stop live queries, when they are not used anymore. When submitting a new search, the previous one will be stopped.