Amazon S3 Bucket

Amazon Simple Storage Service (S3) is a highly scalable, durable object storage service designed for storing and retrieving any amount of data from anywhere on the web. S3 provides robust availability, security, and performance, making it ideal for storing log files, application data, and backup archives.

By integrating S3 with Falcon LogScale Collector, organizations can efficiently ingest historical and real-time log data stored in S3 buckets for centralized analysis and monitoring. The integration uses S3 event notifications delivered through Amazon Simple Queue Service (SQS) to automatically detect and process new log files as they arrive.

Data Flow

The following diagram shows how S3 Bucket log data flows through Log Collector's ingestion pipeline, highlighting specific parser types applied to the log data before data is compressed, stored in the repository, and indexed for searching:

flowchart LR A[AWS S3 Bucket] --> B[S3 Event Notification] B --> C[LogScale S3 Connector] C --> D[Parser Pipeline] subgraph "Parser Types" D --> D1[JSON Parser] D --> D2[CSV Parser] D --> D3[CloudTrail Parser] D --> D4[AWS Load Balancer Parser] D --> D5[S3 Access Logs Parser] end D1 & D2 & D3 & D4 & D5 --> E[Compression] E --> F[Repository Storage]
flowchart LR A[AWS S3 Bucket] --> B[S3 Event Notification] B --> C[LogScale S3 Connector] C --> D[Parser Pipeline] subgraph "Parser Types" D --> D1[JSON Parser] D --> D2[CSV Parser] D --> D3[CloudTrail Parser] D --> D4[AWS Load Balancer Parser] D --> D5[S3 Access Logs Parser] end D1 & D2 & D3 & D4 & D5 --> E[Compression] E --> F[Repository Storage]

LogScale S3 ingestion supports multiple AWS log formats including CloudTrail, VPC Flow Logs, AWS Load Balancer logs, and S3 Access logs. The system automatically handles decompression and applies appropriate parsers based on the log format.