Comparing Filebeat, Vector, and Fluentd

Each of these log shippers is extremely capable and have their pros and cons delineated below (however this is not an exhaustive list):

Feature Filebeat Vector Fluentd
Platform Support Installers for Debian, RPM, MacOS, Brew, Linux (tar.gz), Windows Amazon Linux, CentOS, Debian, MacOS, NixOS, Raspbian, RHEL, Ubuntu, Windows Installers for RPM, Debian, MacOS, Windows, platforms running Ruby
License Commercial and OSS; Note: Only the OSS licensed (Apache 2.0) package will work with Humio OSS - Mozilla Public License 2.0 OSS - Apache 2.0
Centralized Management No - Under development No - Under development No
Deployment Models Agent Agent (daemon, sidecar), aggregator Agent, Aggregator
Supported Inputs/Sources AWS CloudWatch, AWS S3, Azure Event Hub, Cloud Foundry, Container, Docker, GCP Pub/Sub, HTTP Endpoint, HTTP JSON, Kafka, Log, MQTT, NetFlow, Office 365 Management Activity API, Redis, Stdin, Syslog, TCP, UDP; Important: Bolded inputs require X-Pack which is not a part of the open source package and requires a paid license. Apache metrics, AWS Kinesis Firehose, AWS ECS Metrics, AWS S3, Datadog Logs, Docker, File, Generator, Heroku Logplex, Host Metrics, Internal Metrics, Journald, Kafka, Kubernetes Logs, MongoDB Metrics, Nginx Metrics, PostgresSQL Metrics, Prometheus Remote Write, Prometheus Scrape, Socket, Splunk HTTP Event Collector (HEC), StatsD, STDIN, Syslog, Vector Inputs supported via Input Plugins: in_tail, in_forward, in_udp, in_tcp, In_unix, in_http, in_syslog, in_exec, in_sample, in_windows_eventlog. Additional input formats (793 input/output plugins) can be found in the full list here: https://docs.fluentd.org/input
Supported Outputs/Sinks Elasticsearch Service, Elasticsearch, Logstash, Kafka, Redis, File, Console. Important: Events are typically sent to Humio from Filebeat using the Elasticsearch output method. AWS Cloudwatch Logs, AWS Cloudwatch Metrics, AWS Kinesis Firehose, AWS Kinesis Data Streams, AWS S3, Amazon Simple Queue Service (SQS), Azure Monitor Logs, Blackhole, Clickhouse, Console, Datadog Logs, Datadog Metrics, Elasticsearch, File, GCP Cloud Storage (GCS), GCS PubSub, GCP Operations (Stackdriver) Logs, GCP Cloud Monitoring (Stackdrive) Metrics, Honeycomb, HTTP, Humio Logs, Humio Metrics, InfluxDB Logs, InfluxDB Metrics, Kafka, LogDNA, Loki, NATS, New Relic Logs, Papertrail, Prometheus Exporter, Prometheus Remote Write, Apache Pulsar, Sematext Logs, Sematex Metrics, Socket, Splunk HEC, StatsD, Vector Output is supported vial plugins: out_copy, out_null, out_roundrobin, out_stdout, out_exec_filter, out_forward, out_mongo / out_mongo_replset, out_exec, out_file, out_s3, out_webhdfs. Additional output formats (793 input/output plugins) can be found in the full list here: https://www.fluentd.org/plugins
ETL Functionality Yes, limited, however organizations typically pair Filebeat with LogStash for more complicated transformations Yes - Vector has its own language designed for transforming log events/metrics Yes - Fluentd has the ability to filter and transform events using its Filter functionality: https://docs.fluentd.org/filter
Humio End Point Utilized Elastic HEC Elastic / HEC (preferred)

While all three log shippers have extensive feature sets Vector and Fluentd offer:

  • Fewer license based restrictions than Filebeat

  • The ability to deploy as both an agent and aggregator

  • More input and output options

  • More complete transformation options

  • The ability to utilize Humio's HEC endpoint which provides for significant improvements in performance over the Elastic endpoint

Neither Filebeat, Vector or Fluentd cover all possible log sources and there are use cases that will require organizations to implement additional log shippers. Windows Event Logs for example are a common source that neither Filebeat or Vector currently handle — Winlogbeat is a great choice.

Windows Event Logs

Vector or Filebeat do not have access to the Windows APIs that are required to read the Windows Event Log. Organizations that require sending Windows events to Humio will need to implement a log shipper for this specific use case.

The most commonly used log shipper for Windows Event Logs is Winlogbeat. Organizations can choose to put Winlogbeat on each server individually however at scale this becomes difficult to manage in terms of deploying log shippers to every Windows machine. The solution is to use the Windows Event Forwarding to forward events from each Windows machine to a central aggregation server that will then forward the events on to Humio. While Winlogbeat is often used in this role, there are other log shippers like Fluentd that forward the events on to Humio.

Kafka

For organizations using Kafka as part of their data pipeline there are multiple solutions for moving data from Kafka to Humio. Vector and Fluentd for example can both read from Kafka and then ship that data to Humio. Alternatively Humio has a Kafka Connect Log Format that is also capable of sending data from Kafka topics to Humio.

One of the primary advantages to using Kafka in front of Humio as part of your log shipping pipeline is that Kafka can be used as a queue to help buffer events under various failure conditions (network outages, log shipper failures, etc.)