Monitoring Alert Execution through the humio-activity Repository

The humio/activity package provides a wealth of information about activity within LogScale and should be installed to help monitor alerts.

The execution and any errors generated when executing alerts can be tracked in the humio-activity repo by examining the category field:

  • Alert mark standard alerts

The subCategory then indicates whether the event relates to the execution of the Alert, Query or Action.

Checking the severity field will indicate the level of the event:

  • Info entries are used to indicate when an alert has been triggered

  • Warning indicates an issue either with the alert, reading the result, or triggering actions, or where an alert has not been triggered due to throttling

  • Error indicates an error, for example running the query or trigger

The following additional fields in each event contain more detailed information for each alert invocation or error; for a full example event, see Alert Raw Event Example:

Field Description
actionId ID of the triggered action; only set for the invocation of a specific action
actionIds List of action ids for when an alert trigger has been triggered
actionInvocationId Unique id for the invocation of an action, can be used to correlate logs, same commenas for actionId
actionInvocationIds List of action invocation ids for when an alert has been triggered
actionName Name of the action that generated an error or would have been triggered
actionName Name of the triggered action; only set for the invocation of a specific action
alertId ID of the alert
alertName Name of the alert
alertTime The timestamp when the alert was triggered.
dataspace Name of the repository or view
eventId The eventId when an alert trigger on an event
events The number of the events returned by the query; by default all queries return a maximum of 200. Where no events were returned by the query the value will be 0.
exceptionMessage A detailed error message that will include errors at the cluster-level that may have contributed; for example permission, API, or network issues
externalQueryId The external id of the running query
lastAlertTime The timestamp of the last time the alert triggered
message The error or warning message for the alert
query The alert query executed
queryProcessedEvents The number of events processed to return the final result set.
queryTimeMillis Time taken in milliseconds to execute the query. This value can be used to help indicate the load of the query (and therefore any optimization or refinement), or to find outliers during execution.
suggestion A guide to the warning or error and how to resolve or identify more information
user The user the query runs on behalf of (run-as-user)
viewId ID of the view for the alert