Monitor, diagnose, and troubleshoot triggers

Once created and configured, triggers do not need to be managed. The triggers will run continuously and trigger actions until disabled or deleted.

The list of triggers shows summary information for each configured trigger:

  • Status shows the current status of the trigger. High-level indication of status is by color:

    • A ⬤ grey circle indicates that the trigger is disabled or that no actions have been assigned.

    • A ⬤ green circle indicates that the trigger is functioning normally.

    • A ⬤ red circle indicates that an error has occurred.

  • The circle indicator is combined with a text description of the status:

    • Okay — Trigger is enabled with no errors

    • Disabled — Trigger is disabled

    • Error — An error has occurred; click ⋮ on the trigger and select Show status to see details.

    • Warning — A warning has occurred; click ⋮ on the trigger and select Show status to see details.

    • No actions assigned — no actions have been configured for the trigger.

    • Disabled actions — one or more actions assigned to the trigger are disabled.

  • Trigger type shows the trigger type and Actions shows the list of the currently configured actions.

The other columns provide detail on when the trigger was Last triggered, either showing the date and time stamp or indicating the trigger has never been triggered.

In addition, when editing a trigger (see Edit triggers), the status of the trigger, and its last trigger status will be displayed on the Edit alert page.

Diagnose trigger warnings and errors

When managing triggers, warnings and errors are reported through the user interface in the Triggers overview andNotifications, and through more detailed logs in the humio-activity repository. For more information about detailed logs, see Monitor Triggers with humio-activity Repository. In the case of notifications, one notification per trigger is sent. For information about notifications, see Notifications.

A good way to troubleshoot error messages is to use the dashboards in the humio/activity package to identify patterns. Some information that might be useful could be the cost over time of alerts or scheduled searches, the top alert messages over time, or scheduled search performance lags, for example. The humio/activity package contains useful dashboards to gather this information.

Errors or warnings are generated at different points in the execution, and cleared using different criteria:

Condition Aggregate Alert Filter Alert Legacy Alert Scheduled Search
Query start Yes, per alert Yes, per alert Yes, per alert  
Query poll Yes, per alert Yes, per alert Yes, per alert  
Trigger action Yes, per action against the aggregate result Yes, per action and event; Individual events track both whether they were triggered and whether an action was started successfully. Yes, per action against the aggregate result  

By default, scheduled searches, aggregate alerts, and filter alerts retry to send events to actions for up to 24 hours. Failure notifications will keep reappearing in the Notifications area for every failed alert for as long as the error stays on the alert.

When analyzing errors and warnings for triggers, consider the following additional factors:

  • Errors in scheduled searches, aggregate alerts, and legacy alerts where multiple Actions have been attached. If some of the actions fail to run, this is logged, but no error will be set on the alert. The alert is considered to have fired, and will be throttled as normal. It will only be considered an error if all actions fail.

    For filter alerts, this information is tracked for each event.

  • Warnings aimed at discouraging queries that include a live join() function in legacy alerts. For more information, see Errors when Using Live join() Functions. (Legacy alerts only)

  • Errors that require some user interaction, for instance an error in the trigger query or that the user has lost permissions to run the query.

  • Warnings that require some user interaction, for instance a warning on too many groups in a groupBy() function invocation in the trigger query. In such a case, the trigger will still trigger, if relevant, but you should consider rewriting the query.

  • Warnings due to the trigger query only returning partial results, which may fire the trigger when it should not have been triggered, or make the trigger only return some of the events it would otherwise have returned.

Clear errors and warnings

Errors and warnings are automatically cleared when the phase they occurred in (Query start, Query poll, or Trigger action) has been executed successfully later. What this means is:

  • Errors and warnings with starting the query are cleared when the query is later successfully started.

  • Errors and warnings with polling the query are cleared when the query is later successfully polled.

  • Errors and warnings with invoking actions are cleared when the actions are later successfully invoked.

Manual clearing of errors and warnings is primarily meant for the situation where an alert or scheduled search, which seldom triggers, encounters an error or warning with invoking actions. That will only be resolved automatically the next time the alert or scheduled search triggers, which can take a long time, so the user can manually clear that error or warning if they know it is resolved.

If an error or warning is manually cleared, but is still occurring because the problem has not be fixed, it will be automatically added again on the next run of the job.