Logstash

Logstash is a free and open-source data processing pipeline from Elastic that ingests data from multiple sources simultaneously, transforms it, and sends it to various destinations. As part of the Elastic Stack, Logstash provides a rich ecosystem of input, filter, and output plugins.

Logstash excels at complex data transformation scenarios where logs need to be enriched, filtered, or restructured before ingestion. It supports hundreds of plugins for connecting to diverse data sources and destinations.

Data Flow

The following diagram shows how Logstash processes and forwards logs through Log Collector's ingestion pipeline:

flowchart LR A[Multiple Sources] --> B[Logstash Pipeline] B --> C[LogScale HTTP Endpoint] C --> D[Parser Pipeline] subgraph "Parser Types" D --> D1[JSON Parser] D --> D2[Pre-processed Data Parser] D --> D3[Enriched Log Parser] D --> D4[Grok Parser] end D1 & D2 & D3 & D4 --> E[Compression] E --> F[Repository Storage]
flowchart LR A[Multiple Sources] --> B[Logstash Pipeline] B --> C[LogScale HTTP Endpoint] C --> D[Parser Pipeline] subgraph "Parser Types" D --> D1[JSON Parser] D --> D2[Pre-processed Data Parser] D --> D3[Enriched Log Parser] D --> D4[Grok Parser] end D1 & D2 & D3 & D4 --> E[Compression] E --> F[Repository Storage]

While more resource-intensive than lightweight shippers like Filebeat, Logstash's transformation capabilities make it valuable for environments requiring advanced log processing before data reaches LogScale.