Metricbeat

Metricbeat is a lightweight tool for collecting and shipping metrics.

Metricbeat collects a large set of valuable system metrics, including:

  • CPU usage statistics

  • Memory statistics

  • File and disk IO statistics

  • Per-process statistics

  • Network and socket statistics

On top of the system-level statistics, Metricbeat comes with modules that offer integrations to many well-known services like Docker, MongoDB, and MySQL. Check out the Modules page in the official Metricbeat documentation for more details on these integrations and how they work. You might also read their Getting Started Guide.

Warning

Beats 7.16 and later Log Shippers have compatibility issues with different versions of LogScale, reporting an Invalid version from Elasticsearch error.

Beats 8.0 and higher require a configuration change to enable them to work. See Troubleshooting: Beats and Logstash Log Shippers 7.13 and higher No Longer Work with LogScale for more information.

Beats/Logstash Version Humio 1.36 and below Humio 1.37-1.62/LogScale 1.63 and higher
Logstash 7.16 and up Incompatible Compatible
Metricbeat 7 and below Compatible Compatible
Metricbeat 8.0.0 Compatible but requires setup.ilm.enabled: false Compatible but requires setup.ilm.enabled: false

Metricbeat 8.1.0

Compatible but requires setup.ilm.enabled: false and output.elasticsearch.allow_older_versions: true

Compatible but requires setup.ilm.enabled: false and output.elasticsearch.allow_older_versions: true

Installation

To download Metricbeat, visit the Metricbeat OSS downloads page.

You can find installation documentation for Metricbeat on the Installation page of the official Metricbeat website.

Note

This documentation is written for versions 6.x of Metricbeat. Either make sure to install from the 6.x branch (https://www.elastic.co/guide/en/beats/metricbeat/6.8/metricbeat-installation.html) or make sure to read https://www.elastic.co/guide/en/beats/libbeat/7.6/breaking-changes-7.0.html to know what fields are available in the 7.x metrics.

Configuration

Because LogScale supports parts of the ElasticSearch insertion API, you can send data from Metricbeat to LogScale by configuring Metricbeat to use the built-in ElasticSearch output.

You can find configuration documentation for Metricbeat at the Metricbeat configuration page.

Editing the Configuration

You must make the following changes to the example configuration, see Configuration Example.

  1. Open metricbeat.yml file which you can find in /etc/metricbeat/metricbeat.yml.

  2. Specify the Module and the list of Metricsets, see Metricbeat Modules documentation for more information.

  3. Set the value of enabled to true and specify how often to execute the microsets in period.

  4. Insert the URL of your LogScale installation followed by /api/v1/ingest/elastic-bulk in hosts.

  5. Set the username to a value as required.

  6. Generate and insert the Ingest Tokens from the repository in the password.

  7. Run Metricbeat as a service or if it is already running restart the service using the following commands:

logscale
systemctl enable metricbeat
systemctl restart metricbeat
Configuration Example

The following example shows a simple Metricbeat configuration collecting host metrics and sending them to LogScale:

yaml
metricbeat.modules:
  - module: system
    enabled: true
    period: 10s
    metricsets:
      - cpu
      - load
      - filesystem
      - fsstat
      - memory
      - network
      - socket # linux only

output.elasticsearch:
  hosts: [""https://cloud.humio.com:8080/api/v1/ingest/elastic-bulk"]
  username: my-organization
  password: 750y0940-ec68-4889-9e3a-e7e78d5536er

. _`metricbeat-config-objects`:

Configuration Objects

The section only aims to document the set of keys and value required to ship data to LogScale and therefore not all of the configuration options which are available in Filebeat are listed.

module

The metric collecting module in metricbeats which all contain one or more metricsets.

  • enabled

    Specify if the module is enabled or not, set to true to enable the module. If not specified the module is enabled by default.

  • period

    Specify the frequency with which the microsets are executed.

  • metricsets

    Specify the list of microset to execute.

output.elasticsearch

  • hosts

    The url of your LogScale account and port. Using the standard LogScale API (preferred) $YOUR_LOGSCALE_URL:8080/api/v1/ingest/elastic-bulk or using the elasticsearch port $YOUR_LOGSCALE_URL:9200.

  • username

    This value is not used by LogScale but will be logged by the proxy.

  • password

    Specify the ingest token of your LogScale repository.

Running Metricbeat

Run Metricbeat as a service on Linux with the following commands

logscale
systemctl enable metricbeat
systemctl restart metricbeat

Adding Fields

You can add fields with static values using the fields section. These fields will be added to each event.

Metricbeat automatically sends the host name of the system along with the data. LogScale adds the host name in the @host field to each event. It uses this field name to try not to collide with other fields in the event.

Host Metrics Example Queries

Once you have data from Metricbeat in LogScale, you can run some interesting queries, such as

Show CPU load for each host

logscale
#type=beat 
| timechart(series=@host, function=max(system.load.1, as=load))

Show memory usage for each host

logscale
#type=beat 
| timechart(series=@host, function=max(system.memory.actual.used.bytes))

Show disk free space (in gigabytes)

logscale
#type=beat @host=host1  system.filesystem.mount_point="/"
| timechart(function=min(system.filesystem.free, as=free))
| eval(free=free/(1024*1024*1024))

Disk IO — show bytes read for each disk

logscale
#type=beat @host=host1
| system.diskio.read.bytes=*
| timechart(
   series=system.diskio.name,
   function=counterAsRate(system.diskio.read.bytes), span=1m
  )

Network traffic — Show bytes sent on the eth0 interface

logscale
#type=beat @host=host1 system.network.name=eth0
| timechart(function=count(system.network.out.bytes), span=1m)

Show the top 10 processes using the most CPU resources

logscale
#type=beat 
| system.process.name=*
| groupBy(system.process.name, function=avg(system.process.cpu.total.pct, as=cpu))
| sort(cpu, limit=10)