Object-Level Metrics
Object-level metrics track the metric for a specific entity, for example a
repository or partition. These object level metrics are identified by a
further one or more fields within the humio-metrics
repository. For example, the
ingest-eventsize metric measures
the size of events ingested into a repository. The
repo field is used to
identify the repository. This means that you can obtain metrics for each
repository individually. For example, to get the average
mean_rate for all
repositories, the following query could be used:
logscale
name="ingest-eventsize"
| groupBy(repo,function=avg(mean_rate))Which would return a table like this:
| repo | _avg |
|---|---|
| humio | 38604.0022394165 |
| humio-activity | 144.314766291994 |
| humio-audit | 17.084220687823436 |
| humio-measurements | 0.7359642376509877 |
| humio-metrics | 1456.6950711819202 |
| humio-usage | 9.891309850915516 |
| large-ingest | 0.5448924963318023 |
The full list of object-level metrics is shown in Object-Level Metrics table.
Table: Object-Level Metrics
| Metric Name | Metric Type | Availability | Description |
|---|---|---|---|
actions | Timer | Time spent invoking actions from an alert or scheduled search | |
auxiliary-service-availability | Gauge | The availability status of the auxiliary services registered. true=1 false=0 | |
data-ingester-errors | Meter | Number of events that got an @error tag added to their fields during parsing | |
datasource-count | Gauge | Number of datasources | |
entity-store-operation-rate | Meter | introduced in 1.164.0 | Named entity store operations being called to observed entity stores, initiated by this node |
event-forwarding-errors | Counter | Number of events that were not forwarded for each forwarder | |
event-forwarding-events | Counter | Number of events that were forwarded for each forwarder | |
event-latency-partition | Histogram | Per-partition latency of the 'humio-ingest' topic of the ingest+digest pipeline including time spent in parsers, updating live queries and adding events to blocks for segment files | |
event-latency-repo | Histogram | For each repository, overall latency of ingest+digest pipeline including time spent in parsers, updating live queries and adding events to blocks for segment files | |
events-parsed | Meter | Number of input events a parser was applied to | |
fdr-inflight-message-count | Gauge | Approximate number of inflight messages present on an FDR feed's SQS queue | |
fdr-ingest-events | Meter | Number of FDR events ingested for each FDR feed | |
fdr-message-count | Gauge | Approximate number of messages present on an FDR feed's SQS queue | |
find-timestamp-failed | Counter |
Total number of events for which the findTimestamp()
function failed
| |
garbage-collection-time | Gauge | Time spend doing garbage collection | |
global-operation-rate | Meter | Named operations being applied to global data, initiated by this node | |
global-operation-time | Timer | Execution time for named operations being applied to global data, received and executed on this node. Allocated heap is counted as 1ns/byte | |
ingest-bytes | Meter | Number of bytes uncompressed in flushed blocks for segments being constructed | |
ingest-eventsize | Meter | Number of bytes uncompressed summed over individual events in blocks in progress | |
ingest-offset-lowest | Gauge | The lowest offset on the Kafka ingest queue partition that LogScale will ever need to read again (in failover scenarios). This metric is not updated very often. With the default configuration it is updated around every 40 minutes. It is important that this value keeps growing over time to show there is progress on all datasources. | |
ingest-parsing | Timer | Time spent parsing incoming events | |
ingest-parsing-allocation | Meter |
Amount of memory allocated by the parser during the parsing process, to
parse and ingest a given event for the parser in
repo. The memory allocated here can be an indication
of the average size of the incoming event processed during parsing. Since
it tracks across individual parsers, when summing the total across all
parsers the metric will indicate the memory load for parsing all events.
See also ingest-parsing to
monitor the time taken per event.
| |
ingest-queue-consumer | Timer | Time spent constructing segment file blocks in memory and writing them to disk, including updating live queries if any | |
ingest-queue-latency | Histogram | After the parser has completed, latency of the ingest queue from insert into queue and up to when the data is read but not yet processed in the digest node for each partition. | |
ingest-queue-read-offset | Gauge | Offset in kafka of last event read from the ingest queue | |
ingest-queue-write-offset | Gauge | Offset in Kafka of last event written to the ingest queue | |
ingest-reader-occupancy | Histogram | Percentage of time spent on work in local ingest reader threads on work that is not plain waiting for more records from the ingest queue in Kafka | |
ingest-reader-partition-bytes | Meter | Number of bytes read from Kafka as compressed events from the ingest queue | |
ingest-reader-partition-events | Meter | Number of events added to segment file blocks being constructed | |
ingest-writer-partition-bytes | Meter | Number of bytes written to Kafka as compressed events into the ingest queue in each partition | |
kafka-chatter-by-kind-bytes | Meter | Number of bytes written to Kafka on the chatter topic for each kind of chatter | |
kafka-chatter-by-kind-serialize | Timer | Time spent serializing value being written to Kafka when publishing to the chatter topic for each kind of chatter | |
live-events | Meter | Number of events processed by live queries | |
no-timestamp-found | Counter |
Total number of events for which the findTimestamp()
did not find a timestamp
| |
no-timezone-found | Counter |
Total number of events for which the findTimestamp()
function did not find a timezone and was not called with a default
timezone
| |
query-delta-cost | Histogram | 30s delta cost on queries per repo, for the entire cluster | |
query-millis | Meter | Number of milliseconds spent processing historical queries | |
repo-queries | Meter | Number of queries started per repo | |
segment-bytes-assigned-to-host-as-owner | Meter | The rate at which segment bytes are assigned to each host | |
segments-assigned-to-host-as-owner | Meter | Rate at which segments are assigned to each host | |
tcp-ingest-bytes | Meter | Number of bytes read by tcp ingest listener | |
thread-pool-occupancy | Gauge | Percentage utilisation of the threads in the pool | |
thread-pool-queue-size | Gauge | Number of tasks in queue for the given thread pool | |
time-spent-on-permission-checks | Histogram | Time in milliseconds spent on permission checks | |
udp-ingest-bytes | Meter | Number of bytes read by udp ingest listener | |
written-events | Meter | Number of events written to the ingest queue after being parsed |