Node-Level Metrics
Node-level metrics are recorded specific to each node within your LogScale cluster and will have a field, @host, that contains the hostname number within the cluster. To query across the nodes, a query will need to aggregate across all the records using a reference point, for example the time of the metric entry:
logscale
name="ingest-bytes"
| day := time:dayOfMonth(@timestamp)
| groupBy(day,function=sum(m1))The full list of node-level metrics is shown in Node-Level Metrics table.
Table: Node-Level Metrics
| Metric Name | Metric Type | Availability | Description |
|---|---|---|---|
azure-storage-read | Meter | introduced in 1.166.0 | Bytes fetched for raw segment files and aux files from azure to local data store |
azure-storage-write | Meter | introduced in 1.166.0 | Bytes stored for raw segment files and aux files using azure as data store |
bucket-storage-download-memory-allocated | Gauge | Used for internal debugging. Total size of on-heap byte-buffers allocated by the LogScale application for download from Bucket Storage. | |
bucket-storage-download-queue-free-slots | Gauge | Number of free slots for segment downloads in the Bucket Storage download queue | |
bucket-storage-download-requests-cap-size-hits | Gauge | Number of times the request download queue size cap is hit | |
bucket-storage-fetch-for-query-queue | Gauge | Count of segment files queued awaiting fetch from Bucket Storage to local data store due to being referred by a query | |
bucket-storage-in-progress-downloads | Gauge | Number of downloads in progress | |
bucket-storage-in-progress-uploads | Gauge | Number of uploads in progress | |
bucket-storage-max-concurrent-archive-stream-operations | Gauge | Max concurrent archive stream operations | |
bucket-storage-max-concurrent-delete-operations | Gauge | Max concurrent delete operations | |
bucket-storage-max-concurrent-download-operations | Gauge | Max concurrent download operations | |
bucket-storage-max-concurrent-listfile-operations | Gauge | Max concurrent listfile operations | |
bucket-storage-max-concurrent-upload-operations | Gauge | Max concurrent upload operations | |
bucket-storage-max-concurrent-upload-stream-operations | Gauge | Max concurrent upload stream operations | |
bucket-storage-pending-upload | Gauge | Total size of segment files pending upload to Bucket Storage | |
bucket-storage-pending-upload-underreplicated | Gauge | Total size of pending upload to Bucket Storage for segment files that are not known to have more than one replica in the local cluster | |
bucket-storage-pending-work | Gauge | Number of segments in pending work queue for bucket uploads | |
bucket-storage-request-upload | Meter | Total size of segment files requesting upload to Bucket Storage | |
bucket-storage-request-upload-queue-overflow | Meter | Number of times a segment file does not fit into Bucket Storage upload queue | |
bucket-storage-segment-downloads-in-progress | Gauge | Number of segments downloads currently in progress | |
bucket-storage-total-segment-size | Gauge | Total size of segment files stored in Bucket Storage | |
bucket-storage-upload-latency-max-ms | Gauge | The number of milliseconds that have elapsed since the oldest event that is not currently uploaded to Bucket Storage was ingested | |
bucket-transfer-manager-iteration-time | Histogram | introduced in 1.167.0 | Time elapsed following the start of a single iteration of the bucket transfer manager timer |
chatter-reader-occupancy | Histogram | introduced in 1.167.0 | Percentage of time the 'TransientChatter event loop' is busy. Busy is everything other than waiting for the next event from Kafka on the transientChatter-events topic. |
cluster-time-skew | Gauge | Largest time skew (in milliseconds) between this node and any other node in the cluster | |
compact-timestamp-found | Counter |
Total number of events for which the findTimestamp()
function found a timestamp in the compact format
| |
compressed-bytes-only-present-in-bucket-storage | Gauge | Total compressed bytes of segment files that are present in Bucket Storage, but not on any node disks | |
compressed-bytes-underreplicated-ignoring-bucket-storage | Gauge | Total compressed bytes of segment files that are present in Bucket Storage, but present on too few node disks to satisfy the replication factor | |
cross-query-builder-cache-hits | Meter | Number of times we got a hit looking up a key in the cross-query builder cache | |
cross-query-builder-cache-misses | Meter | Number of times we got a miss looking up a key in the cross-query builder cache | |
cross-query-builder-cache-size | Gauge | Current number of entries in the cross-query builder cache | |
currently-running-streaming-queries | Gauge | The amount of currently active streaming queries | |
day-month-year-timestamp-found | Counter |
Total number of events for which the findTimestamp()
function found a timestamp in the day-month-year format
| |
digest-active-datasources | Histogram | Number of active datasources | |
digest-buffer-target-latency | Histogram | Latency target of in-memory buffer after ingest queue in the digest pipeline | |
digest-live-latency | Histogram | Latency of live update part of digest pipeline for internal bulks in milliseconds | |
digest-segment-latency | Histogram | Latency of segment building part of digest pipeline for internal bulks in milliseconds | |
direct-memory-allocated | Gauge | Used for internal debugging. Amount of direct memory allocated by the Humio application. This does not account for every direct memory allocation in the JVM | |
dynamic-table-reference-controller-size-of-tables-in-use-by-queries | Gauge | Size of tables registered tables in use by queries | |
elastic-search-ingestion-events-in-bulk | Histogram | ||
elastic-search-ingestion-request-errors | Counter | Number of ingest errors in the elastic-search endpoint since the node started | |
elastic-search-ingestion-requests | Timer | Time spent ingesting a bulk request using the elasticsearch ingest protocol | |
event-collector-request-errors | Counter | Number of ingest errors in the http-event-collector endpoint since the node started | |
event-latency | Histogram | Overall latency of ingest queue and digest pipeline not including parsers, but from insert into ingest queue, then updating live queries and adding events to blocks for segment files | |
failed-http-checks | Meter | Number of nodes that appear to be unreachable using http as seen from this node. A healthy system has zero of these | |
federated-merge-latency-ms | Histogram | Time from having a result until it is fully merged and returned in a poll | |
federated-poller-latency-ms | Histogram | Time to return a result in a poll from the federation cluster to the remote clusters | |
federated-poller-session-count | Gauge | Number of active sessions in the federated poller | |
federated-query-count | Gauge | Number of federated queries active | |
federated-wasted-merges | Meter | Number of merges which are never returned in a poll | |
gcs-storage-read | Meter | Bytes fetched for raw segment files and aux files from gcs to local data store | |
gcs-storage-write | Meter |
Bytes stored for raw segment files and aux files
using GCS as data store
| |
global-allocations | Gauge | Global allocations | |
global-condition-index-hit | Gauge | Rate of hits on on global operations that may benefit from in-memory index | |
global-condition-index-miss | Gauge | Rate of misses on on global operations that may benefit from in-memory index | |
global-publish-wait-for-value | Timer | Time spent waiting to see the value being read back from Kafka when pushing an update to the global state | |
global-reader-occupancy | Histogram | introduced in 1.167.0 | Percentage of time the 'Global event loop' is busy. Busy is everything other than waiting for the next event from Kafka on the global-event topics. |
globalsnapshot-size | Gauge |
Size of global-snapshot.json file written
| |
handle-bucket-download-tasks-latency | Histogram | introduced in 1.167.0 | Time elapsed while running actions on already scheduled bucket download tasks |
handle-bucket-upload-tasks-latency | Histogram | introduced in 1.167.0 | Time elapsed while running actions on already scheduled bucket upload tasks |
hashfilter-included-blocks | Meter | Number of blocks included using hashfilters in queries and thus read from compressed blocks in segment files | |
hashfilter-skipped-blocks | Meter | Number of blocks skipped using informed filters in queries and thus not read from compressed blocks in segment files | |
http-requests | Timer | Timing of all inbound http requests | |
http-requests-external-size | Histogram | Size of external inbound http requests | |
http-requests-external-timing | Timer | Timing of external inbound http requests | |
http-requests-internal-size | Histogram | Size of internal inbound http requests | |
http-requests-internal-timing | Timer | Timing of internal inbound http requests | |
humio-ingestion-request-errors | Counter | Number of ingest errors in the humio ingestion endpoint since the node started | |
ingest-bytes-total | Meter | Number of bytes uncompressed in flushed blocks for segments being constructed across all repos | |
ingest-kafka-timeouts | Meter | Number of times Kafka has timed out when we tried to write to the ingest queue | |
ingest-listener-tcp-available | Gauge | TCP ingest listener free slots for lines to be processed (high when idle, zero when over-loaded) | |
ingest-reader-occupancy | Histogram | Percentage of time spent in local ingest reader threads on work that is not plain waiting for more records from the ingest queue in Kafka | |
ingest-reader-polltime | Timer | Time blocked waiting for next message from Kafka from ingest queue | |
ingest-request-delay | Histogram | Histogram of ingest request time spent being delayed due to exceeding limit on concurrent processing of ingest | |
ingest-writer-bulksize | Histogram | Histogram of size (bytes) of data for jobs that carry events. Some jobs are no-payload and are not included here | |
ingest-writer-compressed-bytes | Meter | Number of bytes written to Kafka as compressed events into the ingest queue in total | |
ingest-writer-jobs | Meter | Number of jobs pushed to in-memory job queue for digest writers | |
ingest-writer-queue-add | Meter | Number of times an ingest queue consumer pushes to in-memory job queue for digest writers, including when the operation fails due to the queue being full | |
ingest-writer-queue-empty | Meter | Number of times an ingest queue consumer hit an empty queue while pushing to in-memory job queue for digest writers | |
ingest-writer-queue-full | Meter | Number of times an ingest queue consumer hit a full queue while pushing to in-memory job queue for digest writers | |
ingest-writer-threads | Histogram | Number of ingest writer threads currently active | |
ingest-writer-uncompressed-bytes | Meter | Number of bytes written to Kafka before compression for events into the ingest queue in total | |
internal-poll-rate | Meter | Internal polls requests rate from this node | |
internal-throttled-poll-rate | Meter | The rate of internal polls during query execution that were throttled due to rate limiting. | |
internal-throttled-poll-wait-time | Timer | The delay per poll round and request in polling query workers due to rate limiting. | |
jvm-NON-heap-max-usage | Gauge | Maximum java virtual machine NON heap memory usage | |
jvm-NON-heap-usage | Gauge | Java Virtual Machine NON heap memory usage | |
jvm-heap-usage | Gauge | Java Virtual Machine heap memory usage | |
jvm-heap-usage-percent | Gauge | Java Virtual Machine heap memory usage in percent | |
jvm-hiccup-latency | Histogram | Latency of timed events inside Humio jvm | |
kafka-chatter-bytes | Meter | Number of bytes written to Kafka on the chatter topic | |
kafka-chatter-put | Timer |
Time waiting for getting ack from Kafka when publishing
to the chatter topic
| |
kafka-ingestqueue-put | Timer | Time waiting for getting ack when adding ingest events to the ingest queue | |
kafka-request-bytes | Histogram | Number of bytes written to Kafka as compressed events for the ingest queue | |
kafka-request-events | Histogram | Number of events written to Kafka as compressed events for the ingest queue | |
kafka-request-queue-fill-percentage | Gauge | Percentage of queue used for tasks waiting to send to the Kafka ingest queue | |
live-dashboard-query-count | Gauge | Number of live queries on dashboards | |
livequeries-canceled-due-to-digest-delay | Meter | Number of live queries that have been canceled due to excessive digest delay | |
livequeries-rate | Gauge | The rate of the cost of live queries, in cost/s | |
livequeries-rate-canceled-due-to-digest-delay | Gauge | The rate of the cost of live queries canceled due to excessive digest delay, in cost/s | |
livequery-count | Gauge | Number of live (real-time) queries active | |
load-segment-total | Timer | Time spent reading (waiting for) blocks from segment files | |
local-query-jobs-queue | Gauge | Count queries currently queued or active on node including exports | |
local-query-jobs-queue-exports-part | Gauge | Count queries currently queued or active on node for exports | |
local-query-jobs-wait | Histogram | Histogram of time in milliseconds that each query waited between getting any work done including exports | |
local-query-segments-queue | Gauge | Count of elements in queue as number of segments currently queued for query including exports | |
local-query-segments-queue-exports-part | Gauge | Count of elements in queue as number of segments currently queued for query for exports | |
logplex-ingestion-request-errors | Counter | Number of ingest errors in the logplex endpoint since the node started | |
lookup-tables-loading-tasks | Gauge | Maximum number of pending tasks queued for loading lookup tables since latest sample for metrics | |
lookup-tables-number-of-files | Gauge | Number of files registered in the uploaded file cache | |
lookup-tables-number-of-index-structures | Gauge | Number of index-structures registered in the lookup table coordinator | |
lookup-tables-number-of-lookup-tables | Gauge | Number of lookup tables registered in the lookup table coordinator | |
lookup-tables-number-of-queries | Gauge | Number of queries registered in the lookup table coordinator | |
lookup-tables-size-of-cached-files | Gauge | Size of files in the uploaded file cache | |
mapsegment | Timer | Time spent on 'map' phase while searching non-real time segment files | |
max-ingest-delay | Gauge | The maximum overall latency of ingest+digest pipeline across all hosts and partitions. Excludes time spent in parsers. | |
min-ingest-timestamp | Histogram | Lowest ingest timestamp seen by this node across partitions, relative to current time. | |
min-unacked-ingest-timestamp | Histogram | Lowest unacked-pending-ingest timestamp seen by this node across partitions, relative to current time. | |
mini-segment-created | Meter | Number of new mini-segment being created. The number gets incremented when the mini-segment gets closed and added to global | |
minisegment-blocks | Histogram | Number of blocks in segments created by pre-merging mini-segments into larger mini-segments | |
minisegment-compressed-size | Histogram | Size of the file for segments created by pre-merging mini-segments into larger mini-segments | |
minisegment-merge-cpu-time | Timer | CPU time spent pre-merging mini-segments | |
minisegment-uncompressed-size | Histogram | Number of bytes uncompressed for segments created by pre-merging mini-segments into larger mini-segments | |
missing-cluster-nodes | Gauge | Number of nodes that are considered dead by this node, or nodes that have been manually stopped. | |
missing-cluster-nodes-stateful | Gauge | Number of stateful nodes that are considered dead by this node, or nodes that have been manually stopped. | |
month-day-year-last-timestamp-found | Counter |
Total number of events for which the findTimestamp()
found a timestamp in the month-day-year(last) format
| |
month-day-year-timestamp-found | Counter |
Total number of events for which the findTimestamp()
found a timestamp in the month-day-year format
| |
primary-disk-usage | Gauge | Percent used on the primary disk | |
proxied-query-polls | Timer | Timing of internal requests due to polling of queries not hitting the server coordinating the query | |
queries | Counter | Total number of queries started since this node started | |
query | Timer | Measure how long it takes for queries to complete | |
query-compiler-mapper-task-time | Histogram | Time in milliseconds spent by the query compiler compiling a targeted query to a mapper task | |
query-compiler-parse-time | Histogram | Time in nanoseconds spent by the query compiler parsing from input text to AST | |
query-compiler-preprocess-time | Histogram | Time in milliseconds spent by the query compiler in the preprocessing phase | |
query-coordinator-latency | Histogram | Latency for responses on query state refreshes from nodes within the cluster | |
query-delta-total-cost | Histogram | 30s delta of total cost on queries for the entire cluster | |
query-delta-total-memory-allocation | Histogram | 30s delta of total memory allocation on queries for the entire cluster | |
query-deserialization-bytes | Gauge | Maximum total number of pending bytes in tasks queued for deserialization since the latest sample for metrics | |
query-deserialization-tasks | Gauge | Maximum number of pending tasks queued for deserialization since latest sample for metrics | |
query-fsm-snapshot-refresh-time | Timer | Measures the time between QueryFSM state snapshot updates | |
query-live-delta-cpu-usage | Histogram | 30s delta of CPU usage on live queries for the entire cluster | |
query-segments-count | Meter | Segment being queried that hit local files. Includes those fetched from remote once they arrive | |
query-segments-count-from-remote | Meter | Segments being queried that missed local, triggering a fetch from remote | |
query-static-cost-cache-hit | Meter | Part of static cost of queries coordinated by this host and completed in this time interval, which were based on results loaded from the query state cache | |
query-static-cost-cache-miss | Meter | Part of static cost of queries coordinated by this host and completed in this time interval, which were accumulated across the cluster refreshing the result | |
query-static-cost-total | Meter | Total static cost of queries coordinated by this host and completed in this time interval | |
query-static-delta-cpu-usage | Histogram | 30s delta of CPU usage on static queries for the entire cluster | |
query-thread-limit | Histogram | Number of threads allowed to execute historical parts of queries. Gets turned down if digest is unable to keep up | |
query-time-spent-starved-waiting-for-bucket-storage-avg | Gauge | Average milliseconds a query has spent not running due to waiting for data from Bucket Storage. Resets when logged. | |
query-time-spent-starved-waiting-for-bucket-storage-max | Gauge | Max milliseconds a query has spent not running due to waiting for data from Bucket Storage. Resets when logged. | |
querycache-disk-usage | Gauge | Sum of sizes of files in local query cache | |
querycache-max-age | Gauge | Age of the oldest cache entry that has not been reused or deleted yet. As the cache drops the least recently used, this is the age of the next item to be dropped from the cache | |
read-compressed-bytes | Meter | Number of bytes of read from compressed blocks in segment files | |
read-prefilter-bytes | Meter | Number of bytes of read from pre-filter files | |
recompress-millis | Meter | Number of milliseconds CPU time spent merging and re-compressing segment files | |
s3-archiving-bytes-per-second | Meter | Bytes archived in S3 per second | |
s3-archiving-errors-per-second | Meter | Errors per second archiving logs in S3 | |
s3-archiving-latency-max-ms | Gauge | The number of milliseconds that have elapsed since the oldest event that is not currently archived in Bucket Storage was ingested | |
s3-archiving-writes-per-second | Meter | Successful S3 archival writes per second | |
s3-storage-read | Meter |
Bytes fetched for raw segment files and aux files
from S3 to local data store
| |
s3-storage-write | Meter |
Bytes stored for raw segment files and aux files
using S3 as data store
| |
schedulesegments | Timer | Time spent scheduling segment files for the 'map' phase while searching non-real time segment files | |
secondary-disk-usage | Gauge | Percent used on the secondary disk. Only present if secondary disk is configured | |
segment-entity-compact | Gauge | Number of new segments using the compact format being added or updated in the in-memory cache | |
segment-entity-full | Gauge | Number of new segments requiring the full format being added or updated in the in-memory cache | |
segment-fetching-trigger-queue-hit-full-counter | Counter | introduced in 1.166.0 | Number of times the segment fetching trigger queue is full when adding tasks to it |
segment-fetching-trigger-queue-offer-counter | Counter | introduced in 1.166.0 | Number of times a task is added to the segment fetching trigger queue |
segment-fetching-trigger-queue-retry-offer-counter | Counter | introduced in 1.166.0 | Number of times a task is readded to the segment fetching trigger queue |
segment-fetching-trigger-queue-size | Meter | introduced in 1.166.0 | Size of the segment fetching trigger queue |
segment-merge-cpu-time | Timer | CPU time spent merging segments | |
segment-merge-latency-ms | Histogram | Histogram measuring the merge latency, which is defined as the latency between the last minisegment being written in a sequence with the same merge target | |
segment-merger-threads | Histogram | Number of segment merger threads currently active | |
segment-validator-threads | Histogram | Number of segment validator threads currently active | |
serialize-state-bytes | Histogram | Number of bytes serialized for internal query states | |
serialize-state-time | Timer | Time spent serializing internal query states | |
shafka-ingest-bytes | Meter | Number of bytes ingested from shafka | |
shafka-ingest-events | Meter | Number of events ingested from shafka | |
start-new-bucket-tasks-latency | Histogram | introduced in 1.167.0 | Time elapsed while running actions to schedule new bucket transfer tasks |
table-cache-calls-to-get | Gauge | Number of table fetches from the cache | |
table-cache-calls-to-load | Gauge | Number of tables loaded into the cache | |
table-cache-calls-to-put | Gauge | Number of tables put in the cache | |
table-cache-disk-usage | Gauge | Total size of tables cached on disk in bytes | |
table-cache-number-of-cached-tables-on-Disk | Gauge | Number of cached tables on disk | |
table-cache-reads-from-disk | Gauge | Number of tables loaded into the cache from disk | |
table-cache-writes-to-disk | Gauge | Number of cached tables written to disk | |
table-coordinator-calls-to-getTableFromCluster | Gauge | Number of calls to TableCoordinator.getTableFromCluster | |
table-coordinator-calls-to-getTableFromNode | Gauge | Number of calls to TableCoordinator.getTableFromNode | |
table-coordinator-calls-to-hasTableOnCluster | Gauge | Number of calls to TableCoordinator.hasTableOnCluster | |
table-coordinator-calls-to-hasTableOnNode | Gauge | Number of calls to TableCoordinator.hasTableOnNode | |
table-coordinator-calls-to-putTableOnCluster | Gauge | Number of calls to TableCoordinator.putTableOnCluster | |
table-coordinator-calls-to-putTableOnNode | Gauge | Number of calls to TableCoordinator.putTableOnNode | |
table-coordinator-calls-to-putTableOnRemoteCluster | Gauge | Number of calls to TableCoordinator.putTableOnRemoteCluster | |
table-registry-number-of-queries | Gauge | Number of queries in TableRegistry | |
target-segment-blocks | Histogram | Number of blocks in segments created by merging mini-segments | |
target-segment-compressed-size | Histogram | Size of the file for segments created by merging mini-segments | |
target-segment-created | Meter | Number of new segment targets being created. The number gets incremented when the target ID is chosen, before any of the mini-segments exist | |
target-segment-uncompressed-size | Histogram | Number of bytes uncompressed for segments created by merging mini-segments | |
temp-disk-usage-bytes | Gauge | Bytes used on the temp disk | |
time-digest | Gauge | CPU time used on digest as a fraction of wall time | |
time-livequery | Gauge | CPU time used on live queries as a fraction of wall time | |
time-only-timestamp-found | Counter |
Total number of events for which the findTimestamp()
function found a timestamp in the time only format
| |
timestamp-parsing-failed | Counter | Total number of timestamp strings that did not parse as a time stamp since start of the node | |
unix-epoch-timestamp-found | Counter |
Total number of events for which the findTimestamp()
found a timestamp in the unix epoch format
| |
uploaded-files-cache-entries | Gauge | Cached uploaded files. How many files are cached in memory | |
user-permissions-lookup | Counter | Total number of user permission lookups | |
user-permissions-lookup-cache-miss | Counter | Number of user permission lookups with a cache miss | |
written-events-after-queue | Meter | Number of events added to segment file blocks being constructed | |
year-month-day-timestamp-found | Counter |
Total number of events for which the findTimestamp()
found a timestamp in the year-month-day format
|