The humio-fleet Repository

The humio-fleet repository is a system repository that stores metadata from log shippers, used by the Fleet Management UI.

Log Collector Metrics

The Log Collector sends metrics to Fleet Management for enrolled instances of Log Collector. The Log Collector reports the following metrics, which could be used to create dashboards to monitor your fleet:

  • Process CPU Usage: The CPU usage (%) of the Log Collector process.

  • Process Memory Usage: The memory usage of the Log Collector process.

  • Host Disk Usage: The full percentage of the disk partition that the process is using as the data directory. The metrics are automatically ingested into the humio-fleet system repository.

Prerequisites

The following prerequisites must be met to use this feature:

  • Running Log Collector newer or equal to 1.5.1.

  • Running LogScale newer or equal to 1.100.

  • The Log Collector has been enrolled using the enrollment command.

Log Collector Metrics

The Log Collector sends the following structured JSON events to the humio-fleet repository. The events contain a discriminator field called kind which can be used to determine what kind of metric is being reported.

The kindfield is being converted to a LogScale tag, as such the field name is converted to #kind, but the key in the JSON event is called kind. This is relevant when querying the data in humio-fleet, since it must be filtered like: #kind=system.

The Log Collector also sends an event that doesn't contain the #kind tag. Those events are deprecated, but are currently being used to generate the Fleet Overview page. Use the following descriptions to understand the JSON events that are being stored in the humio-fleet repository.

Kind system

The following is an example of a system event: 

{
  "kind": "system",
  "id": "gmIhP974udWeeaMzRKPOrxa557OfGTMp",
  "ipAddress": "198.51.100.1",
  "timestamp": 1690840800000,
  "version": "1.5.0",
  "hostname": "logscale-collector-example",
  "system": "Ubuntu 20.04.6 LTS (Focal Fossa) (amd64)",
  "startTime": 1690840800000,
  "bootTime": 1690840800000,
  "mode": "full",
  "machineId": "c53aa5a4-2828-475f-836a-97c1a933b0b5"
}
Field Type Required Description
kind constant system yes The kind field determines the type of event.
id string yes The globally unique ID assigned to the enrolled Falcon LogScale Collector. Used to correlate the metrics events.
ipAddress string yes The source IP address of the Log Collector as seen by Fleet Management.
timestamp number yes The timestamp of the event in epoch milliseconds.
version string yes The version of the Log Collector.
hostname string yes The system hostname of the Log Collector.
system string yes A descriptive string of the operating system that is running the Log Collector.
startTime number yes The start time of the Log Collector process in epoch milliseconds.
bootTime number yes The operating system boot time in epoch milliseconds.
mode string yes The current mode of Fleet Management. The value is full when using remote configuration.
machineId string yes The machine ID generated by the Log Collector locally when it was first installed.

Kind collectorCpuUsage

The following is an example of a collectorCpuUsage event: 

{
  "kind": "collectorCpuUsage",
  "id": "gmIhP974udWeeaMzRKPOrxa557OfGTMp",
  "bucketStart": 1690840800000,
  "bucketEnd": 1690840860000,
  "avg": 4.12,
  "cores": 8
}
Field Type Required Description
kind constant collectorCpuUsage yes The kind field determines the type of event.
id string yes The globally unique ID assigned to the enrolled Falcon LogScale Collector. Used to correlate the metrics events.
bucketStart number yes The bucket start timestamp in epoch milliseconds. The value of the event covers the interval [bucketStart;bucketEnd].
bucketEnd number yes The bucket end timestamp in epoch milliseconds. The value of the event covers the interval [bucketStart;bucketEnd].
avg number yes The CPU usage of the Log Collector process averaged over the bucket duration. A value of 100 % corresponds to one vCPU utilized completely.
cores number yes The number of CPU cores in the system. Dividing avg by cores results in a value between 0 % and 100 %.

Kind collectorMemoryUsage

The following is an example of a collectorMemoryUsage event: 

{
  "kind": "collectorMemoryUsage",
  "id": "gmIhP974udWeeaMzRKPOrxa557OfGTMp",
  "bucketStart": 1690840800000,
  "bucketEnd": 1690840860000,
  "max": 34848768
}
Field Type Required Description
kind constant collectorMemoryUsage yes The kind field determines the type of event.
id string yes The globally unique ID assigned to the enrolled Falcon LogScale Collector. Used to correlate the metrics events.
bucketStart number yes The bucket start timestamp in epoch milliseconds. The value of the event covers the interval [bucketStart;bucketEnd].
bucketEnd number yes The bucket end timestamp in epoch milliseconds. The value of the event covers the interval [bucketStart;bucketEnd].
max number yes The memory usage of the Log Collector process in bytes. The value is the maximum value over the bucket duration.

Kind hostDiskUsage

The following is an example of a hostDiskUsage event: 

{
  "kind": "hostDiskUsage",
  "id": "gmIhP974udWeeaMzRKPOrxa557OfGTMp",
  "bucketStart": 1690840800000,
  "bucketEnd": 1690840860000,
  "path": "/var/lib/logscale-collector",
  "max": 8.62
}
Field Type Required Description
kind constant hostDiskUsage yes The kind field determines the type of event.
id string yes The globally unique ID assigned to the enrolled Falcon LogScale Collector. Used to correlate the metrics events.
bucketStart number yes The bucket start timestamp in epoch milliseconds. The value of the event covers the interval [bucketStart;bucketEnd].
bucketEnd number yes The bucket end timestamp in epoch milliseconds. The value of the event covers the interval [bucketStart;bucketEnd].
max number yes The disk usage in percent of the partition containing the path path.
path string yes The path that is being monitored. Currently specifies the data directory of the Log Collector.