A scheduled search is a static query, set to run on a schedule. At a scheduled interval, the query will run and if its result is non-empty, the scheduled search will trigger its associated actions.
While we recommend highly using scheduled searches, you can disable all scheduled searches from running by setting the environment variable, ENABLE_SCHEDULED_SEARCHES to false. Rather than disable all scheduled searches, though, you can disable an individual scheduled search from the Alerts tab in the User Interface.
Scheduled searches are related to Alerts and they are able to trigger the same actions. However, scheduled searches are applicable in other use cases than alerts, such as when:
You need to automatically report some search result on a schedule. For instance, you have stakeholders that expect to get an email every Monday at 10:00 containing the top most important security events for the previous week.
You have an ingest delay on some logs, which results in them never appearing in searches made by alerts. For instance, if an alert looks back in time using a
1htime window, it won’t trigger on logs ingested with a 12 hour delay. With a scheduled search, you can choose to run your search at a point in time, where you’re fairly certain that every log of interest has been ingested.
You need to take delayed action on search results. For instance, if you trigger user bans using an alert, offending users will be banned immediatly upon a transgression and can then easily figure out what triggered their ban. Using a scheduled search, you can choose to ban all offending users at the same time every day, as to obscure the conditions of a ban.
If your situation doesn’t fall into one of these use cases, you should probably use an alert instead. Alerts run as live queries, rather than historic ones, and should thus generally be considered more performant.
Fill out the fields prompted.
Alternatively you can use the GraphQL API to view, create, update and delete scheduled searches using the associated queries and mutations.
Scheduled searches, per default, will not trigger any action(s), if a query result contains a warning. You can alter this behavior by switching on the SCHEDULED_SEARCH_DESPITE_WARNINGS variable in the configuration file. Scheduled searches have the same behavior as alerts in regards to warnings and errors, see Errors and Warnings documentation.
In the following we discuss some of the fields you set on a scheduled search.
This field allows you to specify the schedule on which your scheduled search should be run. The schedule is defined using a UNIX cron expression, as known from the crontab file found in many UNIX-like systems. There are many online tools to help you generate UNIX cron expressions, that you can use if you need help writing up an expression for your use case.
The Coordinated Universal Time(UTC) offset defines the temporal offset from UTC in which the search is scheduled. For instance, with a schedule
0 6 * * * and an offset
UTC+01:00, the search will be scheduled for 5AM at UTC.
As for all searches, a time interval must be specified. For scheduled searches the time interval is given by a
end time relative to the scheduled execution time. For instance, if a scheduled search is executing at midnight Jan 2nd, with a time interval of
start = 24h and
end = now, the search will consider all logs within the time interval:
At times, a scheduled search can be prevented from running. If a scheduled search query is still running when the next scheduled search is planned to start, then the active query is allowed to keep running, and the next scheduled search is prevented from starting. For example, if a scheduled search is set to run every 5 minutes,
*/5 * * * *, but the query takes 12 minutes to complete, then it will prevent 2 searches from being run on time. Scheduled searches can also be prevented from running if Humio is down, or there is an error preventing an associated action from being triggered. When a scheduled search is no longer prevented from running, Humio will attempt to backfill searches, which were missed previously.
The backfilling behavior depends on the value given to the backfill limit, which determines how many missed searches will be executed before any new searches are scheduled.
Let us say that we schedule a search every hour,
0 * * * *, and Humio is down between 10:30 and 14:15. This means that the searches at 11:00, 12:00, 13:00 and 14:00 were missed.
Executing the most recent ‘missed’ search is not considered backfilling, as this can also occur under normal operation, if there is a slight delay within Humio. Thus, if the backfill limit is set to 0, as per default, only the search at 14:00 will be executed at startup.
If we increase the limit to 1 we would start off by executing the search scheduled at 13:00, if we increase the limit to 2 we start with the search at 12:00 and if we increase the limit to 3 we start with the search at 11:00. Increasing the value of the backfill limit beyond this point will not have any effect in this example. Note that the missed searches are executed in sequence from oldest to newest.
The backfill limit may not exceed the global maximum backfill limit. This is set using the configuration option, SCHEDULED_SEARCH_BACKFILL_LIMIT.
Humio will always attempt to run a search exactly according to schedule. This makes scheduled searches predictable, but also risks that many scheduled searches will be configured to run at the same time, which might cause delays. It is common to schedule many jobs for midnight, if they are to be run daily, but if you experience delays in search execution because of a sudden high search load, try to space the searches out over a larger span of time.
If you decide to run a search on another schedule, but wish to keep the same search window, you need to update
end on your scheduled search. For instance, if your search was running at midnight and searching through the previous day, you would have configured the interval parameters as
end=now. But if you need to reschedule this search run at 3AM instead, you would have to update the interval parameters as
end=3h to search within the same 24 hour time window.