Alerts
Alert Activities
Alerts use live-queries that run continuously, trigger one or more actions when the query returns a result. Using alerts enables automation for notifying analysts and administrators through different integrations such as email or forward to another repo. This means you don't have to rely on a routine of checking LogScale and executing queries manually or programmatically and can detect problems as soon as they occur
The Alert types that are available:
Standard Alerts are triggered by query that generates a result set. If the query is not already an aggregate query result,
tail(200)
is appended to the query to make it an aggregate query. Aggregate queries associated with a standard alert are run as live queries, and when the result set is executed and generates results, these are fed to the configured actions.Actions are triggered for the combined result set.
Filter Alerts are based on non-aggregate queries, and are configured to trigger the corresponding action at least once. Filter alerts also use a live query, but must not use an aggregate function for execution. Each event in the result set from the alert query triggers the actions associated with the alert.
The attributes of the two alert types can be compared using the following table:
Feature | Standard Alerts | Filter Alerts |
---|---|---|
Supports Aggregates | Yes | No |
Supports Joins | No, see warnings with live joins | No |
Needs search window | Yes | No |
Triggered By | Aggregate result | Single event |
Action invocation | Sequentially | Concurrently |
Can be throttled | Yes | No (support will be added in a future release) |
Can be used in packages | Yes | Yes |
Standard Alerts
Standard alerts work by returning the results from an aggregate query; when the query returns an aggregate result with one or more rows, the alert is triggered. Standard alerts have the following attributes and behaviour:
Standard alerts execute a live query and returns the results from the query to act as the content (and data) for the alert
An alert is triggered against the query only when the query returns one or more results, and therefore the alert is triggered against a query result set of the events matching the aggregate query.
All the values within the result set from the query are available when triggering an action
Standard alerts can be throttled to prevent the query triggering a configured action too often or too frequently. See Setting Alert Throttle Period.
The environment variable
ENABLE_ALERTS
must be set to thetrue
on every host in the cluster.
Filter Alerts
Filter alerts are designed to be triggered when the corresponding query filters an event; each matching event triggers the alert. Filter alerts have the following attributes and behaviour:
An alert is triggered for each matching event.
Events processed through a filter alert are recorded by the system so that they triggered only once during execution.
Filter alerts do not support either aggregate queries or joins.
The @id and @ingesttimestamp should be preserved during the filter query execution and thus cannot be overwritten or removed.
The _bucket field, if present in the query result will be removed when the alert query executes.
Filter alerts will process events, including catching up for past events, for up to 24 hours. This allows filter alerts to account for ingest and delays in processing actions. While catching up, the alert will not process new events; if a single event is causing the alert of actions to fail, the alert does not trigger until that event is outside the catch-up limit.
The configuration value
FILTER_ALERTS_MAX_CATCH_UP_LIMIT
can be used to set the catch-up period using Relative Time Syntax.Filter alerts will wait 10 minutes for query warnings about missing data to disappear. If they do not disappear within this time limit, the alert will give up and the data will not be triggered on.
The configuration parameter
FILTER_ALERTS_MAX_WAIT_FOR_MISSING_DATA
can be used to set the wait time using Relative Time Syntax.Throttle limits for filter alerts are not supported.
The environment variable
ENABLE_FILTER_ALERTS
must be set totrue
on every host in the cluster.
Alert Activities
Alerts can be managed using the following steps and procedures:
Alerts can be created through the User Interface.
When creating standard alerts, you can keep them from triggering multiple times in a short period of time by setting a throttle period.
The User Interface offers many ways to manage alerts that you've created.
Using alerts may sometimes present errors and warnings to handle.
As part of the LogScale Alert system, you may integrate it with a security monitoring system. These systems can be used to notify your staff and allow for more detailed analysis of server security.