Fluentd

Fluentd is an open-source data collector that unifies log collection and consumption, enabling you to aggregate logs from multiple sources, transform them, and route them to various destinations including Falcon LogScale Collector.

As a Cloud Native Computing Foundation graduated project, Fluentd is widely adopted in containerized and cloud-native environments. It uses a plugin-based architecture with over 500 plugins for different data sources, transformations, and outputs.

Data Flow

The following diagram shows how Fluentd collects and routes logs through Log Collector's ingestion pipeline:

flowchart LR A[Source Systems] --> B[Fluentd Collector] B --> C[LogScale HTTP Endpoint] C --> D[Parser Pipeline] subgraph "Parser Types" D --> D1[Structured Data Parser] D --> D2[JSON Parser] D --> D3[Regex Parser] D --> D4[Multi-line Parser] end D1 & D2 & D3 & D4 --> E[Compression] E --> F[Repository Storage]
flowchart LR A[Source Systems] --> B[Fluentd Collector] B --> C[LogScale HTTP Endpoint] C --> D[Parser Pipeline] subgraph "Parser Types" D --> D1[Structured Data Parser] D --> D2[JSON Parser] D --> D3[Regex Parser] D --> D4[Multi-line Parser] end D1 & D2 & D3 & D4 --> E[Compression] E --> F[Repository Storage]

Fluentd's unified logging layer approach and JSON-based structured logging make it particularly well-suited for modern microservices architectures where logs from many services need to be aggregated and standardized.