Metricbeat
Metricbeat is a lightweight tool for collecting and shipping metrics.
Metricbeat collects a large set of valuable system metrics, including:
CPU usage statistics
Memory statistics
File and disk IO statistics
Per-process statistics
Network and socket statistics
On top of the system-level statistics, Metricbeat comes with modules that offer integrations to many well-known services like Docker, MongoDB, and MySQL. Check out the Modules page in the official Metricbeat documentation for more details on these integrations and how they work. You might also read their Getting Started Guide.
Warning
Beats 7.16 and later Log Shippers have compatibility issues with different versions of LogScale, reporting an Invalid version from Elasticsearch error.
Beats 8.0 and higher require a configuration change to enable them to work. See Troubleshooting: Beats and Logstash Log Shippers 7.13 and higher No Longer Work with LogScale for more information.
Beats/Logstash Version | Humio 1.36 and below | Humio 1.37-1.62/LogScale 1.63 and higher |
---|---|---|
Logstash 7.16 and up | Incompatible | Compatible |
Metricbeat 7 and below | Compatible | Compatible |
Metricbeat 8.0.0 | Compatible but requires setup.ilm.enabled: false | Compatible but requires setup.ilm.enabled: false |
Metricbeat 8.1.0 |
Compatible but requires setup.ilm.enabled: false and output.elasticsearch.allow_older_versions: true |
Compatible but requires setup.ilm.enabled: false and output.elasticsearch.allow_older_versions: true |
Installation
To download Metricbeat, visit the Metricbeat OSS downloads page.
You can find installation documentation for Metricbeat on the Installation page of the official Metricbeat website.
Note
This documentation is written for versions 6.x of Metricbeat. Either make sure to install from the 6.x branch (https://www.elastic.co/guide/en/beats/metricbeat/6.8/metricbeat-installation.html) or make sure to read https://www.elastic.co/guide/en/beats/libbeat/7.6/breaking-changes-7.0.html to know what fields are available in the 7.x metrics.
Configuration
Because LogScale supports parts of the ElasticSearch insertion API, you can send data from Metricbeat to LogScale by configuring Metricbeat to use the built-in ElasticSearch output.
You can find configuration documentation for Metricbeat at the Metricbeat configuration page.
Editing the Configuration
You must make the following changes to the example configuration, see Configuration Example.
Open
metricbeat.yml
file which you can find in/etc/metricbeat/metricbeat.yml
.Specify the
Module
and the list ofMetricsets
, see Metricbeat Modules documentation for more information.Set the value of
enabled
totrue
and specify how often to execute the microsets inperiod
.Insert the URL of your LogScale installation followed by /api/v1/ingest/elastic-bulk in
hosts
.Set the username to a value as required.
Generate and insert the Ingest Tokens from the repository in the
password
.Run Metricbeat as a service or if it is already running restart the service using the following commands:
systemctl enable metricbeat
systemctl restart metricbeat
Configuration Example
The following example shows a simple Metricbeat configuration collecting host metrics and sending them to LogScale:
metricbeat.modules:
- module: system
enabled: true
period: 10s
metricsets:
- cpu
- load
- filesystem
- fsstat
- memory
- network
- socket # linux only
output.elasticsearch:
hosts: [""https://cloud.humio.com:8080/api/v1/ingest/elastic-bulk"]
username: my-organization
password: 750y0940-ec68-4889-9e3a-e7e78d5536er
. _`metricbeat-config-objects`:
Configuration Objects
The section only aims to document the set of keys and value required to ship data to LogScale and therefore not all of the configuration options which are available in Filebeat are listed.
module
The metric collecting module in metricbeats which all contain one or more metricsets.
enabled
Specify if the module is enabled or not, set to
true
to enable the module. If not specified the module is enabled by default.period
Specify the frequency with which the microsets are executed.
metricsets
Specify the list of microset to execute.
output.elasticsearch
hosts
The url of your LogScale account and port. Using the standard LogScale API (preferred)
$YOUR_LOGSCALE_URL:8080/api/v1/ingest/elastic-bulk
or using the elasticsearch port$YOUR_LOGSCALE_URL:9200
.username
This value is not used by LogScale but will be logged by the proxy.
password
Specify the ingest token of your LogScale repository.
Running Metricbeat
Run Metricbeat as a service on Linux with the following commands
systemctl enable metricbeat
systemctl restart metricbeat
Adding Fields
You can add fields with static values using the fields section. These fields will be added to each event.
Metricbeat automatically sends the host name of the system along with the data. LogScale adds the host name in the @host field to each event. It uses this field name to try not to collide with other fields in the event.
Host Metrics Example Queries
Once you have data from Metricbeat in LogScale, you can run some interesting queries, such as
Show CPU load for each host
#type=beat
| timechart(series=@host, function=max(system.load.1, as=load))
Show memory usage for each host
#type=beat
| timechart(series=@host, function=max(system.memory.actual.used.bytes))
Show disk free space (in gigabytes)
#type=beat @host=host1 system.filesystem.mount_point="/"
| timechart(function=min(system.filesystem.free, as=free))
| eval(free=free/(1024*1024*1024))
Disk IO — show bytes read for each disk
#type=beat @host=host1
| system.diskio.read.bytes=*
| timechart(
series=system.diskio.name,
function=counterAsRate(system.diskio.read.bytes), span=1m
)
Network traffic — Show bytes sent on the eth0 interface
#type=beat @host=host1 system.network.name=eth0
| timechart(function=count(system.network.out.bytes), span=1m)
Show the top 10 processes using the most CPU resources
#type=beat
| system.process.name=*
| groupBy(system.process.name, function=avg(system.process.cpu.total.pct, as=cpu))
| sort(cpu, limit=10)