Node-Level Metrics

Node-level metrics are recorded specific to each node within your LogScale cluster and will have a field, @host, that contains the hostname number within the cluster. To query across the nodes, a query will need to aggregate across all the records using a reference point, for example the time of the metric entry:

logscale
name="ingest-bytes"
| day := time:dayOfMonth(@timestamp)
| groupBy(day,function=sum(m1))

The full list of node-level metrics is shown in Node-Level Metrics table.

Table: Node-Level Metrics

Metric NameMetric TypeAvailabilityDescription
bucket-storage-download-memory-allocatedGauge  Used for internal debugging. Total size of on-heap byte-buffers allocated by the LogScale application for download from Bucket Storage.
bucket-storage-download-queue-free-slotsGauge  Number of free slots for segment downloads in the Bucket Storage download queue
bucket-storage-download-requests-cap-size-hitsGauge  Number of times the request download queue size cap is hit
bucket-storage-fetch-for-query-queueGauge  Count of segment files queued awaiting fetch from Bucket Storage to local data store due to being referred by a query
bucket-storage-in-progress-downloadsGauge  Number of downloads in progress
bucket-storage-in-progress-uploadsGauge  Number of uploads in progress
bucket-storage-max-concurrent-archive-stream-operationsGauge  Max concurrent archive stream operations
bucket-storage-max-concurrent-delete-operationsGauge  Max concurrent delete operations
bucket-storage-max-concurrent-download-operationsGauge  Max concurrent download operations
bucket-storage-max-concurrent-listfile-operationsGauge  Max concurrent listfile operations
bucket-storage-max-concurrent-upload-operationsGauge  Max concurrent upload operations
bucket-storage-max-concurrent-upload-stream-operationsGauge  Max concurrent upload stream operations
bucket-storage-pending-uploadGauge  Total size of segment files pending upload to Bucket Storage
bucket-storage-pending-upload-underreplicatedGauge  Total size of pending upload to Bucket Storage for segment files that are not known to have more than one replica in the local cluster
bucket-storage-pending-workGauge  Number of segments in pending work queue for bucket uploads
bucket-storage-request-uploadMeter  Total size of segment files requesting upload to Bucket Storage
bucket-storage-request-upload-queue-overflowMeter  Number of times a segment file does not fit into Bucket Storage upload queue
bucket-storage-segment-downloads-in-progressGauge  Number of segments downloads currently in progress
bucket-storage-total-segment-sizeGauge  Total size of segment files stored in Bucket Storage
bucket-storage-upload-latency-max-msGauge  The number of milliseconds that have elapsed since the oldest event that is not currently uploaded to Bucket Storage was ingested
cluster-time-skewGauge  Largest time skew (in milliseconds) between this node and any other node in the cluster
compact-timestamp-foundCounter  Total number of events for which the findTimestamp() function found a timestamp in the compact format
compressed-bytes-only-present-in-bucket-storageGauge  Total compressed bytes of segment files that are present in Bucket Storage, but not on any node disks
compressed-bytes-underreplicated-ignoring-bucket-storageGauge  Total compressed bytes of segment files that are present in Bucket Storage, but present on too few node disks to satisfy the replication factor
cross-query-builder-cache-hitsMeter  Number of times we got a hit looking up a key in the cross-query builder cache
cross-query-builder-cache-missesMeter  Number of times we got a miss looking up a key in the cross-query builder cache
cross-query-builder-cache-sizeGauge  Current number of entries in the cross-query builder cache
currently-running-streaming-queriesGauge  The amount of currently active streaming queries
day-month-year-timestamp-foundCounter  Total number of events for which the findTimestamp() function found a timestamp in the day-month-year format
digest-active-datasourcesHistogram  Number of active datasources
digest-buffer-target-latencyHistogram  Latency target of in-memory buffer after ingest queue in the digest pipeline
digest-live-latencyHistogram  Latency of live update part of digest pipeline for internal bulks in milliseconds
digest-segment-latencyHistogram  Latency of segment building part of digest pipeline for internal bulks in milliseconds
direct-memory-allocatedGauge  Used for internal debugging. Amount of direct memory allocated by the Humio application. This does not account for every direct memory allocation in the JVM
dynamic-table-reference-controller-size-of-tables-in-use-by-queriesGauge  Size of tables registered tables in use by queries
elastic-search-ingestion-events-in-bulkHistogram  
elastic-search-ingestion-request-errorsCounter  Number of ingest errors in the elastic-search endpoint since the node started
elastic-search-ingestion-requestsTimer  Time spent ingesting a bulk request using the elasticsearch ingest protocol
event-collector-request-errorsCounter  Number of ingest errors in the http-event-collector endpoint since the node started
event-latencyHistogram  Overall latency of ingest queue and digest pipeline not including parsers, but from insert into ingest queue, then updating live queries and adding events to blocks for segment files
failed-http-checksMeter  Number of nodes that appear to be unreachable using http as seen from this node. A healthy system has zero of these
federated-merge-latency-msHistogram  Time from having a result until it is fully merged and returned in a poll
federated-poller-latency-msHistogram  Time to return a result in a poll from the federation cluster to the remote clusters
federated-poller-session-countGauge  Number of active sessions in the federated poller
federated-query-countGauge  Number of federated queries active
federated-wasted-mergesMeter  Number of merges which are never returned in a poll
gcs-storage-readMeter  Bytes fetched for raw segment files and aux files from gcs to local data store
gcs-storage-writeMeter  Bytes stored for raw segment files and aux files using GCS as data store
global-allocationsGauge  Global allocations
global-condition-index-hitGauge  Rate of hits on on global operations that may benefit from in-memory index
global-condition-index-missGauge  Rate of misses on on global operations that may benefit from in-memory index
global-publish-wait-for-valueTimer  Time spent waiting to see the value being read back from Kafka when pushing an update to the global state
globalsnapshot-sizeGauge  Size of global-snapshot.json file written
hashfilter-included-blocksMeter  Number of blocks included using hashfilters in queries and thus read from compressed blocks in segment files
hashfilter-skipped-blocksMeter  Number of blocks skipped using informed filters in queries and thus not read from compressed blocks in segment files
http-requestsTimer  Timing of all inbound http requests
http-requests-external-sizeHistogram  Size of external inbound http requests
http-requests-external-timingTimer  Timing of external inbound http requests
http-requests-internal-sizeHistogram  Size of internal inbound http requests
http-requests-internal-timingTimer  Timing of internal inbound http requests
humio-ingestion-request-errorsCounter  Number of ingest errors in the humio ingestion endpoint since the node started
ingest-bytes-totalMeter  Number of bytes uncompressed in flushed blocks for segments being constructed across all repos
ingest-kafka-timeoutsMeter  Number of times Kafka has timed out when we tried to write to the ingest queue
ingest-listener-tcp-availableGauge  TCP ingest listener free slots for lines to be processed (high when idle, zero when over-loaded)
ingest-reader-occupancyHistogram  Percentage of time spent in local ingest reader threads on work that is not plain waiting for more records from the ingest queue in Kafka
ingest-reader-polltimeTimer  Time blocked waiting for next message from Kafka from ingest queue
ingest-request-delayHistogram  Histogram of ingest request time spent being delayed due to exceeding limit on concurrent processing of ingest
ingest-writer-bulksizeHistogram  Histogram of size (bytes) of data for jobs that carry events. Some jobs are no-payload and are not included here
ingest-writer-compressed-bytesMeter  Number of bytes written to Kafka as compressed events into the ingest queue in total
ingest-writer-jobsMeter  Number of jobs pushed to in-memory job queue for digest writers
ingest-writer-queue-addMeter  Number of times an ingest queue consumer pushes to in-memory job queue for digest writers, including when the operation fails due to the queue being full
ingest-writer-queue-emptyMeter  Number of times an ingest queue consumer hit an empty queue while pushing to in-memory job queue for digest writers
ingest-writer-queue-fullMeter  Number of times an ingest queue consumer hit a full queue while pushing to in-memory job queue for digest writers
ingest-writer-uncompressed-bytesMeter  Number of bytes written to Kafka before compression for events into the ingest queue in total
internal-poll-rateMeter  Internal polls requests rate from this node
internal-throttled-poll-rateMeter  The rate of internal polls during query execution that were throttled due to rate limiting.
internal-throttled-poll-wait-timeTimer  The delay per poll round and request in polling query workers due to rate limiting.
jvm-NON-heap-max-usageGauge  Maximum java virtual machine NON heap memory usage
jvm-NON-heap-usageGauge  Java Virtual Machine NON heap memory usage
jvm-heap-usageGauge  Java Virtual Machine heap memory usage
jvm-heap-usage-percentGauge  Java Virtual Machine heap memory usage in percent
jvm-hiccup-latencyHistogram  Latency of timed events inside Humio jvm
kafka-chatter-bytesMeter  Number of bytes written to Kafka on the chatter topic
kafka-chatter-putTimer  Time waiting for getting ack from Kafka when publishing to the chatter topic
kafka-ingestqueue-putTimer  Time waiting for getting ack when adding ingest events to the ingest queue
kafka-request-bytesHistogram  Number of bytes written to Kafka as compressed events for the ingest queue
kafka-request-eventsHistogram  Number of events written to Kafka as compressed events for the ingest queue
kafka-request-queue-fill-percentageGauge  Percentage of queue used for tasks waiting to send to the Kafka ingest queue
live-dashboard-query-countGauge  Number of live queries on dashboards
livequeries-canceled-due-to-digest-delayMeter  Number of live queries that have been canceled due to excessive digest delay
livequeries-rateGauge  The rate of the cost of live queries, in cost/s
livequeries-rate-canceled-due-to-digest-delayGauge  The rate of the cost of live queries canceled due to excessive digest delay, in cost/s
livequery-countGauge  Number of live (real-time) queries active
load-segment-totalTimer  Time spent reading (waiting for) blocks from segment files
local-query-jobs-queueGauge  Count queries currently queued or active on node including exports
local-query-jobs-queue-exports-partGauge  Count queries currently queued or active on node for exports
local-query-jobs-waitHistogram  Histogram of time in milliseconds that each query waited between getting any work done including exports
local-query-segments-queueGauge  Count of elements in queue as number of segments currently queued for query including exports
local-query-segments-queue-exports-partGauge  Count of elements in queue as number of segments currently queued for query for exports
logplex-ingestion-request-errorsCounter  Number of ingest errors in the logplex endpoint since the node started
lookup-tables-loading-tasksGauge  Maximum number of pending tasks queued for loading lookup tables since latest sample for metrics
lookup-tables-number-of-filesGauge  Number of files registered in the uploaded file cache
lookup-tables-number-of-index-structuresGauge  Number of index-structures registered in the lookup table coordinator
lookup-tables-number-of-lookup-tablesGauge  Number of lookup tables registered in the lookup table coordinator
lookup-tables-number-of-queriesGauge  Number of queries registered in the lookup table coordinator
lookup-tables-size-of-cached-filesGauge  Size of files in the uploaded file cache
mapsegmentTimer  Time spent on 'map' phase while searching non-real time segment files
max-ingest-delayGauge  The maximum overall latency of ingest+digest pipeline across all hosts and partitions. Excludes time spent in parsers.
min-ingest-timestampHistogram  Lowest ingest timestamp seen by this node across partitions, relative to current time.
min-unacked-ingest-timestampHistogram  Lowest unacked-pending-ingest timestamp seen by this node across partitions, relative to current time.
mini-segment-createdMeter  Number of new mini-segment being created. The number gets incremented when the mini-segment gets closed and added to global
minisegment-blocksHistogram  Number of blocks in segments created by pre-merging mini-segments into larger mini-segments
minisegment-compressed-sizeHistogram  Size of the file for segments created by pre-merging mini-segments into larger mini-segments
minisegment-merge-cpu-timeTimer  CPU time spent pre-merging mini-segments
minisegment-uncompressed-sizeHistogram  Number of bytes uncompressed for segments created by pre-merging mini-segments into larger mini-segments
missing-cluster-nodesGauge  Number of nodes that are considered dead by this node, or nodes that have been manually stopped.
missing-cluster-nodes-statefulGauge  Number of stateful nodes that are considered dead by this node, or nodes that have been manually stopped.
month-day-year-last-timestamp-foundCounter  Total number of events for which the findTimestamp() found a timestamp in the month-day-year(last) format
month-day-year-timestamp-foundCounter  Total number of events for which the findTimestamp() found a timestamp in the month-day-year format
primary-disk-usageGauge  Percent used on the primary disk
proxied-query-pollsTimer  Timing of internal requests due to polling of queries not hitting the server coordinating the query
queriesCounter  Total number of queries started since this node started
queryTimer  Measure how long it takes for queries to complete
query-compiler-mapper-task-timeHistogram  Time in milliseconds spent by the query compiler compiling a targeted query to a mapper task
query-compiler-parse-timeHistogram  Time in nanoseconds spent by the query compiler parsing from input text to AST
query-compiler-preprocess-timeHistogram  Time in milliseconds spent by the query compiler in the preprocessing phase
query-coordinator-latencyHistogram  Latency for responses on query state refreshes from nodes within the cluster
query-delta-total-costHistogram  30s delta of total cost on queries for the entire cluster
query-delta-total-memory-allocationHistogram  30s delta of total memory allocation on queries for the entire cluster
query-deserialization-bytesGauge  Maximum total number of pending bytes in tasks queued for deserialization since the latest sample for metrics
query-deserialization-tasksGauge  Maximum number of pending tasks queued for deserialization since latest sample for metrics
query-fsm-snapshot-refresh-timeTimer  Measures the time between QueryFSM state snapshot updates
query-live-delta-cpu-usageHistogram  30s delta of CPU usage on live queries for the entire cluster
query-segments-countMeter  Segment being queried that hit local files. Includes those fetched from remote once they arrive
query-segments-count-from-remoteMeter  Segments being queried that missed local, triggering a fetch from remote
query-static-cost-cache-hitMeter  Part of static cost of queries coordinated by this host and completed in this time interval, which were based on results loaded from the query state cache
query-static-cost-cache-missMeter  Part of static cost of queries coordinated by this host and completed in this time interval, which were accumulated across the cluster refreshing the result
query-static-cost-totalMeter  Total static cost of queries coordinated by this host and completed in this time interval
query-static-delta-cpu-usageHistogram  30s delta of CPU usage on static queries for the entire cluster
query-thread-limitHistogram  Number of threads allowed to execute historical parts of queries. Gets turned down if digest is unable to keep up
query-time-spent-starved-waiting-for-bucket-storage-avgGauge  Average milliseconds a query has spent not running due to waiting for data from Bucket Storage. Resets when logged.
query-time-spent-starved-waiting-for-bucket-storage-maxGauge  Max milliseconds a query has spent not running due to waiting for data from Bucket Storage. Resets when logged.
querycache-disk-usageGauge  Sum of sizes of files in local query cache
querycache-max-ageGauge  Age of the oldest cache entry that has not been reused or deleted yet. As the cache drops the least recently used, this is the age of the next item to be dropped from the cache
read-compressed-bytesMeter  Number of bytes of read from compressed blocks in segment files
read-prefilter-bytesMeter  Number of bytes of read from pre-filter files
recompress-millisMeter  Number of milliseconds CPU time spent merging and re-compressing segment files
s3-archiving-bytes-per-secondMeter  Bytes archived in S3 per second
s3-archiving-errors-per-secondMeter  Errors per second archiving logs in S3
s3-archiving-latency-max-msGauge  The number of milliseconds that have elapsed since the oldest event that is not currently archived in Bucket Storage was ingested
s3-archiving-writes-per-secondMeter  Successful S3 archival writes per second
s3-storage-readMeter  Bytes fetched for raw segment files and aux files from S3 to local data store
s3-storage-writeMeter  Bytes stored for raw segment files and aux files using S3 as data store
schedulesegmentsTimer  Time spent scheduling segment files for the 'map' phase while searching non-real time segment files
secondary-disk-usageGauge  Percent used on the secondary disk. Only present if secondary disk is configured
segment-entity-compactGauge  Number of new segments using the compact format being added or updated in the in-memory cache
segment-entity-fullGauge  Number of new segments requiring the full format being added or updated in the in-memory cache
segment-merge-cpu-timeTimer  CPU time spent merging segments
segment-merge-latency-msHistogram  Histogram measuring the merge latency, which is defined as the latency between the last minisegment being written in a sequence with the same merge target
serialize-state-bytesHistogram  Number of bytes serialized for internal query states
serialize-state-timeTimer  Time spent serializing internal query states
shafka-ingest-bytesMeter  Number of bytes ingested from shafka
shafka-ingest-eventsMeter  Number of events ingested from shafka
table-cache-calls-to-getGauge  Number of table fetches from the cache
table-cache-calls-to-loadGauge  Number of tables loaded into the cache
table-cache-calls-to-putGauge  Number of tables put in the cache
table-cache-disk-usageGauge  Total size of tables cached on disk in bytes
table-cache-number-of-cached-tables-on-DiskGauge  Number of cached tables on disk
table-cache-reads-from-diskGauge  Number of tables loaded into the cache from disk
table-cache-writes-to-diskGauge  Number of cached tables written to disk
table-coordinator-calls-to-getTableFromClusterGauge  Number of calls to TableCoordinator.getTableFromCluster
table-coordinator-calls-to-getTableFromNodeGauge  Number of calls to TableCoordinator.getTableFromNode
table-coordinator-calls-to-hasTableOnClusterGauge  Number of calls to TableCoordinator.hasTableOnCluster
table-coordinator-calls-to-hasTableOnNodeGauge  Number of calls to TableCoordinator.hasTableOnNode
table-coordinator-calls-to-putTableOnClusterGauge  Number of calls to TableCoordinator.putTableOnCluster
table-coordinator-calls-to-putTableOnNodeGauge  Number of calls to TableCoordinator.putTableOnNode
table-coordinator-calls-to-putTableOnRemoteClusterGauge  Number of calls to TableCoordinator.putTableOnRemoteCluster
table-registry-number-of-queriesGauge  Number of queries in TableRegistry
target-segment-blocksHistogram  Number of blocks in segments created by merging mini-segments
target-segment-compressed-sizeHistogram  Size of the file for segments created by merging mini-segments
target-segment-createdMeter  Number of new segment targets being created. The number gets incremented when the target ID is chosen, before any of the mini-segments exist
target-segment-uncompressed-sizeHistogram  Number of bytes uncompressed for segments created by merging mini-segments
temp-disk-usage-bytesGauge  Bytes used on the temp disk
time-digestGauge  CPU time used on digest as a fraction of wall time
time-livequeryGauge  CPU time used on live queries as a fraction of wall time
time-only-timestamp-foundCounter  Total number of events for which the findTimestamp() function found a timestamp in the time only format
timestamp-parsing-failedCounter  Total number of timestamp strings that did not parse as a time stamp since start of the node
unix-epoch-timestamp-foundCounter  Total number of events for which the findTimestamp() found a timestamp in the unix epoch format
uploaded-files-cache-entriesGauge  Cached uploaded files. How many files are cached in memory
user-permissions-lookupCounter  Total number of user permission lookups
user-permissions-lookup-cache-missCounter  Number of user permission lookups with a cache miss
written-events-after-queueMeter  Number of events added to segment file blocks being constructed
year-month-day-timestamp-foundCounter  Total number of events for which the findTimestamp() found a timestamp in the year-month-day format