Find Failed Requests

Display ingest requests that have failed due to throttling

Query

flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] result{{Result Set}} repo --> 1 1 --> result
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] result{{Result Set}} repo --> 1 1 --> result
logscale
#type=humio
#kind=logs
statuscode=503
/msg=(?<msg>Ingest parsing exceeded the acceptable amount of time[^\.]+)\. exception/

Introduction

A regular expression can be used together with field filters to narrow down log events to a specific subset based on both field values and message content. By combining status code filters with a regex pattern match on the message field, it is possible to surface only the events that represent a particular and identifiable failure condition from the broader stream of log events.

API throttling is the process of limiting the number of API requests that a user can make in a certain period. A throttling error indicates that you have exceeded the limit for your account. When searching Falcon LogScale logs in the humio repository, the tag #type, #kind and #vhost can be used. All the logs will have #type=humio, a #kind tag, and a #vhost tag.

This query finds individual ingest requests that fail due to being throttled.

Example incoming data might look like this:

@timestamp#type#kind#vhoststatuscodemsg@rawstring
1742032800000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-alpha. exception occurredIngest parsing exceeded the acceptable amount of time for repository repo-alpha. exception occurred
1742032860000humiologsingest.example.com200Ingest completed successfully for repo-alphaIngest completed successfully for repo-alpha
1742032920000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-beta. exception occurredIngest parsing exceeded the acceptable amount of time for repository repo-beta. exception occurred
1742032980000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-alpha. exception occurredIngest parsing exceeded the acceptable amount of time for repository repo-alpha. exception occurred
1742033040000humiologsingest.example.com429Too many requests for repo-betaToo many requests for repo-beta
1742033100000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-gamma. exception occurredIngest parsing exceeded the acceptable amount of time for repository repo-gamma. exception occurred
1742033160000humiometricsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-alpha. exception occurredIngest parsing exceeded the acceptable amount of time for repository repo-alpha. exception occurred
1742033220000humiologsingest.example.com200Ingest completed successfully for repo-gammaIngest completed successfully for repo-gamma
1742033280000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-beta. exception occurredIngest parsing exceeded the acceptable amount of time for repository repo-beta. exception occurred
1742033340000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-delta. exception occurredIngest parsing exceeded the acceptable amount of time for repository repo-delta. exception occurred
1742033400000humiologsingest.example.com404Repository not found for repo-deltaRepository not found for repo-delta
1742033460000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-alpha. exception occurredIngest parsing exceeded the acceptable amount of time for repository repo-alpha. exception occurred

Step-by-Step

  1. Starting with the source repository events.

  2. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] result{{Result Set}} repo --> 1 1 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] result{{Result Set}} repo --> 1 1 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    #type=humio
#kind=logs
statuscode=503
/msg=(?<msg>Ingest parsing exceeded the acceptable amount of time[^\.]+)\. exception/

    Filters on all events from all hosts of all kinds, and filters on all log status codes where the value in logstatuscode exceeds 503/msg and returns an exception/error. The value 503/msg is defined as the acceptable amount of time for parsing and ingest request (in this example).

  3. Event Result set.

Summary and Results

The query is used to find individual ingest requests that fail due to being throttled. From the returned results, you can decide whether to change the throttling period for the relevant alert or not. The best way to handle throttling is to reduce the number of concurrent requests. Be aware, that throttling is used to maintain the optimal performance and reliability of the system, as throttling limits the number of API calls or operations within a time window to prevent the overuse of resources.

This query is useful, for example, to identify which repositories are repeatedly hitting ingest parsing time limits, enabling operations teams to investigate and adjust throttling thresholds or ingest rates for affected repositories.

Sample output from the incoming example data:

@timestamp#type#kind#vhoststatuscodemsg
1742032800000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-alpha
1742032920000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-beta
1742032980000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-alpha
1742033100000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-gamma
1742033280000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-beta
1742033340000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-delta
1742033460000humiologsingest.example.com503Ingest parsing exceeded the acceptable amount of time for repository repo-alpha

Note that only events with #kind equal to logs, statuscode equal to 503, and a msg field matching the pattern Ingest parsing exceeded the acceptable amount of time followed by any characters up to a period before the word exception are retained.

The msg field in the output contains only the captured portion of the message up to the period before the word exception, as defined by the regex capture group.