Alerts

Alert Activities

Alerts use live-queries that run continuously, trigger one or more actions when the query returns a result. Using alerts enables automation for notifying analysts and administrators through different integrations such as email or forward to another repo. This means you don't have to rely on a routine of checking LogScale and executing queries manually or programmatically and can detect problems as soon as they occur

The Alert types that are available:

  • Standard Alerts are triggered by query that generates a result set. If the query is not already an aggregate query result, tail(200) is appended to the query to make it an aggregate query. Aggregate queries associated with a standard alert are run as live queries, and when the result set is executed and generates results, these are fed to the configured actions.

    Actions are triggered for the combined result set.

  • Filter Alerts are based on non-aggregate queries, and are configured to trigger the corresponding action at least once. Filter alerts also use a live query, but must not use an aggregate function for execution. Each event in the result set from the alert query triggers the actions associated with the alert.

The attributes of the two alert types can be compared using the following table:

Feature Standard Alerts Filter Alerts
Supports Aggregates Yes No
Supports Joins No, see warnings with live joins No
Needs search window Yes No
Triggered By Aggregate result Single event
Action invocation Sequentially Concurrently
Can be throttled Yes No (support will be added in a future release)
Can be used in packages Yes Yes

Standard Alerts

Standard alerts work by returning the results from an aggregate query; when the query returns an aggregate result with one or more rows, the alert is triggered. Standard alerts have the following attributes and behaviour:

  • Standard alerts execute a live query and returns the results from the query to act as the content (and data) for the alert

  • An alert is triggered against the query only when the query returns one or more results, and therefore the alert is triggered against a query result set of the events matching the aggregate query.

  • All the values within the result set from the query are available when triggering an action

  • Standard alerts can be throttled to prevent the query triggering a configured action too often or too frequently. See Setting Alert Throttle Period.

  • The environment variable ENABLE_ALERTS must be set to the true on every host in the cluster.

Filter Alerts

Filter alerts are designed to be triggered when the corresponding query filters an event; each matching event triggers the alert. Filter alerts have the following attributes and behaviour:

  • An alert is triggered for each matching event.

  • Events processed through a filter alert are recorded by the system so that they triggered only once during execution.

  • Filter alerts do not support either aggregate queries or joins.

  • The @id and @ingesttimestamp should be preserved during the filter query execution and thus cannot be overwritten or removed.

  • The _bucket field, if present in the query result will be removed when the alert query executes.

  • Filter alerts will process events, including catching up for past events, for up to 24 hours. This allows filter alerts to account for ingest and delays in processing actions. While catching up, the alert will not process new events; if a single event is causing the alert of actions to fail, the alert does not trigger until that event is outside the catch-up limit.

    The configuration value FILTER_ALERTS_MAX_CATCH_UP_LIMIT can be used to set the catch-up period using Relative Time Syntax.

  • Filter alerts will wait 10 minutes for query warnings about missing data to disappear. If they do not disappear within this time limit, the alert will give up and the data will not be triggered on.

    The configuration parameter FILTER_ALERTS_MAX_WAIT_FOR_MISSING_DATA can be used to set the wait time using Relative Time Syntax.

  • Throttle limits for filter alerts are not supported.

  • The environment variable ENABLE_FILTER_ALERTS must be set to true on every host in the cluster.

Alert Activities

Alerts can be managed using the following steps and procedures:

  • Creating Alerts

    Alerts can be created through the User Interface.

  • Setting Alert Throttle Period

    When creating standard alerts, you can keep them from triggering multiple times in a short period of time by setting a throttle period.

  • Managing Alerts

    The User Interface offers many ways to manage alerts that you've created.

  • Diagnosing Alerts

    Using alerts may sometimes present errors and warnings to handle.

  • Packages

    As part of the LogScale Alert system, you may integrate it with a security monitoring system. These systems can be used to notify your staff and allow for more detailed analysis of server security.