Query Metrics

The metrics command returns the metrics of pipelines for collecting data. The metrics include meters and gauges describing the flow of data through the Log Collector.

Note

  • The metrics returned by query metrics are local pipeline diagnostics intended for debugging a running collector instance. They are separate from the Fleet Management metrics that the collector sends to the humio-fleet repository when enrolled. For information on Fleet Management metrics, see Log Collector Metrics.

  • The Log Collector install location varies depending on the type of install (sensor managed, custom install, or full install). This means that your install location might be different from the ones shown below. For details on the different ways of installing see the Log Collector Installation page.

Linux:

shell
logscale-collector query metrics --cfg /etc/humio-log-collector/config.yaml

Windows:

shell
"C:\Program Files (x86)\CrowdStrike\Humio Log Collector\humio-log-collector.exe" query metrics --cfg "C:\Program Files (x86)\CrowdStrike\Humio Log Collector\config.yaml"

Output Format

The command returns a single JSON object organised into three top-level sections:

Field Description
queues Memory queue statistics for each configured sink
sinks Transmission and error metrics for each configured sink
sources Data ingestion metrics for each configured source

Metric Types

Metrics appear in one of three forms:

Meter - tracks a cumulative count and exponentially weighted moving average rates over three time windows:

json
"published_events": {
  "count": 25191,
  "1m.rate": 0.000088,
  "5m.rate": 0.001471,
  "15m.rate": 0.004099,
  "mean.rate": 0.111199
}

Field Description
count Total cumulative count since the collector started
1m.rate Exponentially weighted moving average rate over the last 1 minute
5m.rate Exponentially weighted moving average rate over the last 5 minutes
15m.rate Exponentially weighted moving average rate over the last 15 minutes
mean.rate Mean rate since the collector started

Guage - a single current value:

json
"used_bytes": {
  "value": 0
}

Histogram - distribution statistics for a measured value:

json
"ingest_latency": {
  "count": 25191,
  "min": 1015,
  "max": 222792258,
  "mean": 201900474.34,
  "median": 222784559.5,
  "stddev": 64937681.43,
  "75%": 222788534.75,
  "95%": 222792123.1,
  "99%": 222792249.01,
  "99.9%": 222792258
}

Field Description
count Number of observations
min / max Minimum and maximum observed values
mean Arithmetic mean of all observations
median 50th percentile
stddev Standard deviation
75%, 95%, 99%, 99.9% Percentile thresholds

Note

All latency and timing values are in nanoseconds.

Metrics Reference

Metrics are grouped by sink name and queue type (e.g. queues.logscaleSink.memory).

Metric Type Description
published_bytes Meter Number of events currently awaiting acknowledgement
published_events Meter Events published to the queue
dropped_bytes Meter Bytes dropped due to the queue being full. A non-zero count indicates data loss and warrants investigation
size_bytes Gauge Configured maximum queue capacity in bytes
used_bytes Gauge Current queue utilisation in bytes

Sinks

Metrics are grouped by sink name and sink type (e.g. sinks.logscaleSink.humio).

Metric Type Description
successful_requests Meter HTTP requests that completed successfully
sent_bytes Meter Bytes successfully transmitted to the sink
bad_requests Meter Requests rejected by the sink with an HTTP 4xx response. A non-zero count may indicate a configuration or authentication issue
http_client_errors Meter HTTP client-side errors
http_server_errors Meter Requests rejected by the sink with an HTTP 5xx response
network_errors Meter Connection-level failures when attempting to reach the sink
requests_too_large Meter Requests rejected because they exceeded the sink's maximum request size
compression_ratio Gauge Current compression ratio of data sent to the sink
workers Gauge Number of active worker threads sending data to the sink
latest_attempt Gauge Timestamp of the most recent transmission attempt (Unix milliseconds)
latest_successful_request Gauge Timestamp of the most recent successful transmission (Unix milliseconds)
ingest_latency Histogram End-to-end latency of ingestion operations (nanoseconds)
request_size Histogram Size distribution of HTTP requests sent to the sink (bytes)
time_to_send Histogram + Meter Time taken to complete each transmission (nanoseconds), also reported as a rate

Sources

Metrics are grouped by sink name, source name, and source type.

Metric Type Description
published_bytes Meter Bytes read from the source and submitted to the pipeline
published_events Meter Events read from the source and submitted to the pipeline

Sample Output

The following is a representative example of the full JSON output from a healthy collector with a single pipeline. Values shown are illustrative:

json
{
  "queues": {
    "logscaleSink": {
      "memory": {
        "dropped_bytes": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "published_bytes": {
          "15m.rate": 8.72,
          "1m.rate": 0.00019,
          "5m.rate": 3.10,
          "count": 84055241,
          "mean.rate": 371.04
        },
        "published_events": {
          "15m.rate": 0.0041,
          "1m.rate": 0.000000088,
          "5m.rate": 0.00147,
          "count": 25191,
          "mean.rate": 0.1112
        },
        "size_bytes": {
          "value": 1073741824
        },
        "used_bytes": {
          "value": 0
        }
      }
    }
  },
  "sinks": {
    "logscaleSink": {
      "humio": {
        "bad_requests": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "compression_ratio": {
          "value": 12.98
        },
        "http_client_errors": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "http_server_errors": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "ingest_latency": {
          "75%": 222788534.75,
          "95%": 222792123.1,
          "99%": 222792249.01,
          "99.9%": 222792258,
          "count": 25191,
          "max": 222792258,
          "mean": 201900474.34,
          "median": 222784559.5,
          "min": 1015,
          "stddev": 64937681.43
        },
        "latest_attempt": {
          "value": "<unix-milliseconds>"
        },
        "latest_successful_request": {
          "value": "<unix-milliseconds>"
        },
        "network_errors": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "request_size": {
          "75%": 2861611.5,
          "95%": 18406251.7,
          "99%": 18407224,
          "99.9%": 18407224,
          "count": 26,
          "max": 18407224,
          "mean": 3563717.65,
          "median": 63956,
          "min": 1953,
          "stddev": 6442259.93
        },
        "requests_too_large": {
          "15m.rate": 0,
          "1m.rate": 0,
          "5m.rate": 0,
          "count": 0,
          "mean.rate": 0
        },
        "sent_bytes": {
          "15m.rate": 1265.65,
          "1m.rate": 0.00019,
          "5m.rate": 3.82,
          "count": 84055241,
          "mean.rate": 371.04
        },
        "successful_requests": {
          "15m.rate": 0.0031,
          "1m.rate": 0.000000063,
          "5m.rate": 0.00108,
          "count": 26,
          "mean.rate": 0.000115
        },
        "time_to_send": {
          "15m.rate": 0.0031,
          "1m.rate": 0.000000063,
          "5m.rate": 0.00108,
          "75%": 140028475,
          "95%": 548330870,
          "99%": 550419600,
          "99.9%": 550419600,
          "count": 26,
          "max": 550419600,
          "mean": 132980076.92,
          "mean.rate": 0.000115,
          "median": 21320750,
          "min": 14934700,
          "stddev": 190518589.21
        },
        "workers": {
          "value": 4
        }
      }
    }
  },
  "sources": {
    "logscaleSink": {
      "windowsEvents": {
        "wineventlog": {
          "published_bytes": {
            "15m.rate": 8.72,
            "1m.rate": 0.00019,
            "5m.rate": 3.11,
            "count": 84055241,
            "mean.rate": 371.04
          },
          "published_events": {
            "15m.rate": 0.0041,
            "1m.rate": 0.000000089,
            "5m.rate": 0.00147,
            "count": 25191,
            "mean.rate": 0.1112
          }
        }
      }
    }
  }
}