Kafka
| Typically Used By | Organizations with existing Kafka infrastructure; High-throughput environments |
| Description | A distributed streaming platform integration that enables high-throughput, fault-tolerant ingestion of real-time data streams. |
| Official Vendor Documentation | https://kafka.apache.org/documentation/ |
| Setup Difficulty | 5 (Advanced). This method also requires Kafka expertise to maintain properly |
| Useful for | DevOps , SecOps |
| Primary Use Case | High-volume streaming ingest |
| Scenarios not Recommended | Small deployments without existing Kafka infrastructure; Simple logging needs |
| Data Volume Handling | Very High |
| Authentication Method | SASL/SSL |
| Fault Tolerance | Very High |
| Additional Tools Required | Kafka broker |
Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant handling of real-time data feeds. It enables organizations to collect, process, store, and analyze data streams at scale with low latency, serving as a central messaging backbone between various systems and applications. Kafka's durable storage and publish-subscribe model make it ideal for building reliable data pipelines for streaming analytics solutions such as Falcon LogScale Collector.
Kafka ingest flowThe following diagram shows how Kafka log data flows through Log Collector's ingestion pipeline, highlighting specific parser types applied to the log data before data is compressed, stored in the repository, and indexed for searching: