Parsers Throttling

Monitoring parser resource usage thresholds over time

Query

flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2>Augment Data] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result
flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2>Augment Data] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result
logscale
#kind=logs class=/ParserLimitingJob/ "Top element for parser id"
pct:=100*costSum/threshold
timeChart(function=max(pct), minSpan=10s)

Introduction

The timeChart() function can be used to visualize aggregated values over time by grouping events into time-based buckets and applying an aggregate function to each bucket. By combining timeChart() with max(), it is possible to track the highest value reached within each time interval, making it straightforward to monitor whether a calculated metric is approaching a critical threshold over time.

Throttling is used to maintain the optimal performance and reliability of the system, as throttling limits the number of API calls or operations within a time window to prevent the overuse of resources.

In this example, the timeChart() function is used to show how close (in percentage) the system has been to start throttling any parser.

Example incoming data might look like this:

@timestamp#kindclasscostSumthreshold@rawstring
1742032800000logscom.humio.ParserLimitingJob6501000Top element for parser id repo-alpha cost 650
1742032810000logscom.humio.ParserLimitingJob8201000Top element for parser id repo-beta cost 820
1742032820000logscom.humio.ParserLimitingJob9101000Top element for parser id repo-alpha cost 910
1742032830000logscom.humio.ParserLimitingJob4301000Top element for parser id repo-gamma cost 430
1742032840000logscom.humio.ParserLimitingJob7801000Top element for parser id repo-beta cost 780
1742032850000logscom.humio.ParserLimitingJob5501000Top element for parser id repo-alpha cost 550
1742032800000logscom.humio.IngestPipeline9001000Ingest pipeline processing on repo-alpha
1742032810000metricscom.humio.ParserLimitingJob8801000Top element for parser id repo-beta cost 880
1742033400000logscom.humio.ParserLimitingJob7201000Top element for parser id repo-alpha cost 720
1742033410000logscom.humio.ParserLimitingJob9501000Top element for parser id repo-beta cost 950
1742033420000logscom.humio.ParserLimitingJob8601000Top element for parser id repo-gamma cost 860
1742033430000logscom.humio.ParserLimitingJob4901000Top element for parser id repo-alpha cost 490
1742033440000logscom.humio.ParserLimitingJob8101000Top element for parser id repo-beta cost 810
1742033450000logscom.humio.ParserLimitingJob6701000Top element for parser id repo-gamma cost 670
1742033410000logscom.humio.QueryService7501000Query service heartbeat on repo-beta

Step-by-Step

  1. Starting with the source repository events.

  2. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2>Augment Data] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2>Augment Data] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    #kind=logs class=/ParserLimitingJob/ "Top element for parser id"

    Filters on all logs in humio that are tagged with kind equal to logs and then returns the events where the class field has values containing /ParserLimitingJob/, and where the logline contains the string Top element for parser id.

  3. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2>Augment Data] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 2 fill:#ff0000,stroke-width:4px,stroke:#000;
    flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2>Augment Data] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 2 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    pct:=100*costSum/threshold

    Calculates the percentage of the values in the field costSum divided with values in the field threshold and returns the results in a new field named pct.

  4. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2>Augment Data] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 3 fill:#ff0000,stroke-width:4px,stroke:#000;
    flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2>Augment Data] 3{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> result style 3 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    timeChart(function=max(pct), minSpan=10s)

    Shows the calculated sum of the max values in the field pct in percentage in spans of 10 sec in a timechart.

  5. Event Result set.

Summary and Results

The query is used to show how close (in percentage) the system has been to start throttling any parser.

This query is useful, for example, to identify time periods where parser resource usage is approaching the throttling threshold, enabling operations teams to take preventive action before throttling occurs.

Sample output from the incoming example data:

_bucket_max
174203280000091
174203340000095

Note that only events with #kind equal to logs, class matching ParserLimitingJob, and @rawstring containing Top element for parser id are retained.

Within each 10-second bucket the pct value is calculated as 100*costSum/threshold for each event, and the max() function then returns the highest pct value across all qualifying events in that bucket. In the first bucket the highest pct value is 91, from the event with costSum of 910 and threshold of 1000. In the second bucket the highest pct value is 95, from the event with costSum of 950 and threshold of 1000.