Kafka

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant handling of real-time data feeds. It enables organizations to collect, process, store, and analyze data streams at scale.

Kafka acts as a buffer between data producers and consumers, providing durability through data replication and the ability to replay events. This makes it ideal for high-volume logging environments where temporary downstream unavailability should not result in data loss.

Data Flow

The following diagram shows how Kafka streams flow through Log Collector's ingestion pipeline:

flowchart LR A[Data Producers] --> B[Kafka Topics] B --> C[LogScale Kafka Consumer] C --> D[Parser Pipeline] subgraph "Parser Types" D --> D1[JSON Parser] D --> D2[Avro Parser] D --> D3[Custom Parser] D --> D4[Multi-line Parser] end D1 & D2 & D3 & D4 --> E[Compression] E --> F[Repository Storage]
flowchart LR A[Data Producers] --> B[Kafka Topics] B --> C[LogScale Kafka Consumer] C --> D[Parser Pipeline] subgraph "Parser Types" D --> D1[JSON Parser] D --> D2[Avro Parser] D --> D3[Custom Parser] D --> D4[Multi-line Parser] end D1 & D2 & D3 & D4 --> E[Compression] E --> F[Repository Storage]

LogScale can consume data from Kafka topics, allowing organizations to leverage existing Kafka infrastructure for log aggregation. The integration supports multiple consumer groups and partitions for scalable ingestion.