Query Metrics

The metrics command returns the metrics of pipelines for collecting data. The metrics include meters and gauges describing the flow of data through the Log Collector.

Note

  • The metrics returned by query metrics are local pipeline diagnostics intended for debugging a running collector instance. They are separate from the Fleet Management metrics that the collector sends to the humio-fleet repository when enrolled. For information on Fleet Management metrics, see Log Collector Metrics.

  • The Log Collector install location varies depending on the type of install (sensor managed, custom install, or full install). This means that your install location might be different from the ones shown below. For details on the different ways of installing see the Log Collector Installation page.

Linux:

shell
logscale-collector query metrics --cfg /etc/humio-log-collector/config.yaml

Windows:

shell
"C:\Program Files (x86)\CrowdStrike\Humio Log Collector\humio-log-collector.exe" query metrics --cfg "C:\Program Files (x86)\CrowdStrike\Humio Log Collector\config.yaml"

Output Format

The command returns a single JSON object organised into three top-level sections:

Field Description
queuesMemory queue statistics for each configured sink
sinksTransmission and error metrics for each configured sink
sourcesData ingestion metrics for each configured source

Metric Types

Metrics appear in one of three forms:

Meter - tracks a cumulative count and exponentially weighted moving average rates over three time windows:

json
"published_events": {
  "count": 25191,
  "1m.rate": 0.000088,
  "5m.rate": 0.001471,
  "15m.rate": 0.004099,
  "mean.rate": 0.111199
}

Field Description
countTotal cumulative count since the collector started
1m.rateExponentially weighted moving average rate over the last 1 minute
5m.rateExponentially weighted moving average rate over the last 5 minutes
15m.rateExponentially weighted moving average rate over the last 15 minutes
mean.rateMean rate since the collector started

Guage - a single current value:

json
"used_bytes": {
  "value": 0
}

Histogram - distribution statistics for a measured value:

json
"ingest_latency": {
  "count": 25191,
  "min": 1015,
  "max": 222792258,
  "mean": 201900474.34,
  "median": 222784559.5,
  "stddev": 64937681.43,
  "75%": 222788534.75,
  "95%": 222792123.1,
  "99%": 222792249.01,
  "99.9%": 222792258
}

Field Description
countNumber of observations
min / maxMinimum and maximum observed values
meanArithmetic mean of all observations
median50th percentile
stddevStandard deviation
75%, 95%, 99%, 99.9%Percentile thresholds

Note

All latency and timing values are in nanoseconds.

Metrics Reference

Metrics are grouped by sink name and queue type (e.g. queues.logscaleSink.memory).

Metric Type Description
published_bytesMeterNumber of events currently awaiting acknowledgement
published_eventsMeterEvents published to the queue
dropped_bytesMeterBytes dropped due to the queue being full. A non-zero count indicates data loss and warrants investigation
size_bytesGaugeConfigured maximum queue capacity in bytes
used_bytesGaugeCurrent queue utilisation in bytes

Sinks

Metrics are grouped by sink name and sink type (e.g. sinks.logscaleSink.humio).

Metric Type Description
successful_requestsMeterHTTP requests that completed successfully
sent_bytesMeterBytes successfully transmitted to the sink
bad_requestsMeterRequests rejected by the sink with an HTTP 4xx response. A non-zero count may indicate a configuration or authentication issue
http_client_errorsMeterHTTP client-side errors
http_server_errorsMeterRequests rejected by the sink with an HTTP 5xx response
network_errorsMeterConnection-level failures when attempting to reach the sink
requests_too_largeMeterRequests rejected because they exceeded the sink's maximum request size
compression_ratioGaugeCurrent compression ratio of data sent to the sink
workersGaugeNumber of active worker threads sending data to the sink
latest_attemptGaugeTimestamp of the most recent transmission attempt (Unix milliseconds)
latest_successful_requestGaugeTimestamp of the most recent successful transmission (Unix milliseconds)
ingest_latencyHistogramEnd-to-end latency of ingestion operations (nanoseconds)
request_sizeHistogramSize distribution of HTTP requests sent to the sink (bytes)
time_to_sendHistogram + MeterTime taken to complete each transmission (nanoseconds), also reported as a rate

Sources

Metrics are grouped by sink name, source name, and source type.

Metric Type Description
published_bytesMeterBytes read from the source and submitted to the pipeline
published_eventsMeterEvents read from the source and submitted to the pipeline

Sample Output

The following is a representative example of the full JSON output from a healthy collector with a single pipeline. Values shown are illustrative:

json
{
  "queues": {
    "logscaleSink": {
      "memory": {
        "dropped_bytes": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "published_bytes": {
          "15m.rate": 8.72,
          "1m.rate": 0.00019,
          "5m.rate": 3.10,
          "count": 84055241,
          "mean.rate": 371.04
        },
        "published_events": {
          "15m.rate": 0.0041,
          "1m.rate": 0.000000088,
          "5m.rate": 0.00147,
          "count": 25191,
          "mean.rate": 0.1112
        },
        "size_bytes": {
          "value": 1073741824
        },
        "used_bytes": {
          "value": 0
        }
      }
    }
  },
  "sinks": {
    "logscaleSink": {
      "humio": {
        "bad_requests": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "compression_ratio": {
          "value": 12.98
        },
        "http_client_errors": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "http_server_errors": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "ingest_latency": {
          "75%": 222788534.75,
          "95%": 222792123.1,
          "99%": 222792249.01,
          "99.9%": 222792258,
          "count": 25191,
          "max": 222792258,
          "mean": 201900474.34,
          "median": 222784559.5,
          "min": 1015,
          "stddev": 64937681.43
        },
        "latest_attempt": {
          "value": "<unix-milliseconds>"
        },
        "latest_successful_request": {
          "value": "<unix-milliseconds>"
        },
        "network_errors": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "request_size": {
          "75%": 2861611.5,
          "95%": 18406251.7,
          "99%": 18407224,
          "99.9%": 18407224,
          "count": 26,
          "max": 18407224,
          "mean": 3563717.65,
          "median": 63956,
          "min": 1953,
          "stddev": 6442259.93
        },
        "requests_too_large": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "sent_bytes": {
          "15m.rate": 1265.65,
          "1m.rate": 0.00019,
          "5m.rate": 3.82,
          "count": 84055241,
          "mean.rate": 371.04
        },
        "successful_requests": {
          "15m.rate": 0.0031,
          "1m.rate": 0.000000063,
          "5m.rate": 0.00108,
          "count": 26,
          "mean.rate": 0.000115
        },
        "time_to_send": {
          "15m.rate": 0.0031,
          "1m.rate": 0.000000063,
          "5m.rate": 0.00108,
          "75%": 140028475,
          "95%": 548330870,
          "99%": 550419600,
          "99.9%": 550419600,
          "count": 26,
          "max": 550419600,
          "mean": 132980076.92,
          "mean.rate": 0.000115,
          "median": 21320750,
          "min": 14934700,
          "stddev": 190518589.21
        },
        "workers": {
          "value": 4
        }
      }
    }
  },
  "sources": {
    "logscaleSink": {
      "windowsEvents": {
        "wineventlog": {
          "published_bytes": {
            "15m.rate": 8.72,
            "1m.rate": 0.00019,
            "5m.rate": 3.11,
            "count": 84055241,
            "mean.rate": 371.04
          },
          "published_events": {
            "15m.rate": 0.0041,
            "1m.rate": 0.000000089,
            "5m.rate": 0.00147,
            "count": 25191,
            "mean.rate": 0.1112
          }
        }
      }
    }
  }
}