Object-Level Metrics

Object-level metrics track the metric for a specific entity, for example a repository or partition. These object level metrics are identified by a further one or more fields within the humio-metrics repository. For example, the ingest-eventsize metric measures the size of events ingested into a repository. The repo field is used to identify the repository. This means that you can obtain metrics for each repository individually. For example, to get the average mean_rate for all repositories, the following query could be used:

logscale
name="ingest-eventsize"
| groupBy(repo,function=avg(mean_rate))

Which would return a table like this:

repo_avg
humio38604.0022394165
humio-activity144.314766291994
humio-audit17.084220687823436
humio-measurements0.7359642376509877
humio-metrics1456.6950711819202
humio-usage9.891309850915516
large-ingest0.5448924963318023

The full list of object-level metrics is shown in Object-Level Metrics table.

Table: Object-Level Metrics

Metric NameMetric TypeAvailabilityDescription
actionsTimer  Time spent invoking actions from an alert or scheduled search
auxiliary-service-availabilityGauge  The availability status of the auxiliary services registered. true=1 false=0
data-ingester-errorsMeter  Number of events that got an @error tag added to their fields during parsing
datasource-countGauge  Number of datasources
entity-store-operation-rateMeter  Named entity store operations being called to observed entity stores, initiated by this node
event-forwarding-errorsCounter  Number of events that were not forwarded for each forwarder
event-forwarding-eventsCounter  Number of events that were forwarded for each forwarder
event-latency-partitionHistogram  Per-partition latency of the 'humio-ingest' topic of the ingest+digest pipeline including time spent in parsers, updating live queries and adding events to blocks for segment files
event-latency-repoHistogram  For each repository, overall latency of ingest+digest pipeline including time spent in parsers, updating live queries and adding events to blocks for segment files
events-parsedMeter  Number of input events a parser was applied to
fdr-inflight-message-countGauge  Approximate number of inflight messages present on an FDR feed's SQS queue
fdr-ingest-eventsMeter  Number of FDR events ingested for each FDR feed
fdr-message-countGauge  Approximate number of messages present on an FDR feed's SQS queue
find-timestamp-failedCounter  Total number of events for which the findTimestamp() function failed
garbage-collection-timeGauge  Time spend doing garbage collection
global-operation-rateMeter  Named operations being applied to global data, initiated by this node
global-operation-timeTimer  Execution time for named operations being applied to global data, received and executed on this node. Allocated heap is counted as 1ns/byte
ingest-bytesMeter  Number of bytes uncompressed in flushed blocks for segments being constructed
ingest-eventsizeMeter  Number of bytes uncompressed summed over individual events in blocks in progress
ingest-offset-lowestGauge  The lowest offset on the Kafka ingest queue partition that LogScale will ever need to read again (in failover scenarios). This metric is not updated very often. With the default configuration it is updated around every 40 minutes. It is important that this value keeps growing over time to show there is progress on all datasources.
ingest-parsingTimer  Time spent parsing incoming events
ingest-parsing-allocationMeter  Amount of memory allocated by the parser during the parsing process, to parse and ingest a given event for the parser in repo. The memory allocated here can be an indication of the average size of the incoming event processed during parsing. Since it tracks across individual parsers, when summing the total across all parsers the metric will indicate the memory load for parsing all events. See also ingest-parsing to monitor the time taken per event.
ingest-queue-consumerTimer  Time spent constructing segment file blocks in memory and writing them to disk, including updating live queries if any
ingest-queue-latencyHistogram  After the parser has completed, latency of the ingest queue from insert into queue and up to when the data is read but not yet processed in the digest node for each partition.
ingest-queue-lowest-offset-lagGaugeintroduced in 1.167.0 The distance between the latest offset on the Kafka ingest queue partition, and the lowest offset that LogScale will ever need to read again (in failover scenarios).
ingest-queue-read-offsetGauge  Offset in kafka of last event read from the ingest queue
ingest-queue-write-offsetGauge  Offset in Kafka of last event written to the ingest queue
ingest-reader-occupancyHistogram  Percentage of time spent on work in local ingest reader threads on work that is not plain waiting for more records from the ingest queue in Kafka
ingest-reader-partition-bytesMeter  Number of bytes read from Kafka as compressed events from the ingest queue
ingest-reader-partition-eventsMeter  Number of events added to segment file blocks being constructed
ingest-writer-partition-bytesMeter  Number of bytes written to Kafka as compressed events into the ingest queue in each partition
kafka-chatter-by-kind-bytesMeter  Number of bytes written to Kafka on the chatter topic for each kind of chatter
kafka-chatter-by-kind-serializeTimer  Time spent serializing value being written to Kafka when publishing to the chatter topic for each kind of chatter
live-eventsMeter  Number of events processed by live queries
no-timestamp-foundCounter  Total number of events for which the findTimestamp() did not find a timestamp
no-timezone-foundCounter  Total number of events for which the findTimestamp() function did not find a timezone and was not called with a default timezone
query-delta-costHistogram  30s delta cost on queries per repo, for the entire cluster
query-millisMeter  Number of milliseconds spent processing historical queries
repo-queriesMeter  Number of queries started per repo
segment-bytes-assigned-to-host-as-ownerMeter  The rate at which segment bytes are assigned to each host
segments-assigned-to-host-as-ownerMeter  Rate at which segments are assigned to each host
tcp-ingest-bytesMeter  Number of bytes read by tcp ingest listener
thread-pool-occupancyGauge  Percentage utilisation of the threads in the pool
thread-pool-queue-sizeGauge  Number of tasks in queue for the given thread pool
time-spent-on-permission-checksHistogram  Time in milliseconds spent on permission checks
udp-ingest-bytesMeter  Number of bytes read by udp ingest listener
written-eventsMeter  Number of events written to the ingest queue after being parsed