Amazon S3 Bucket
| Typically Used By | AWS users; Organizations with existing S3 storage; Batch processing needs |
| Description | A method for ingesting logs stored in AWS S3 buckets, ideal for batch processing of historical data or AWS service logs. |
| Official Vendor Documentation | https://docs.aws.amazon.com/s3/ |
| Setup Difficulty | 3 (Moderate) |
| Useful for | DevOps , SecOps |
| Primary Use Case | S3‑stored logs |
| Scenarios not Recommended | Real-time analysis requirements; Non-AWS environments; Time-sensitive monitoring |
| Data Volume Handling | Very High |
| Authentication Method | IAM/Token |
| Fault Tolerance | Very High |
| Additional Tools Required | AWS S3 bucket |
Amazon Simple Storage Service (S3) is a highly scalable, durable object storage service designed for storing and retrieving any amount of data from anywhere on the web. S3 provides robust availability, security, and performance, making it ideal for storing log files, application data, and backup archives. By integrating S3 with Falcon LogScale Collector, organizations can efficiently ingest historical and real-time log data stored in S3 buckets for centralized analysis and monitoring.
A typical workflow uses Amazon SQS notifications to identify newly created S3 objects. Log Collector assumes an AWS IAM role, retrieves the object from S3, processes it with the selected parser, and ingests the resulting events.
Amazon S3 Bucket ingest flowThe following diagram shows how S3 Bucket log data flows through Log Collector's ingestion pipeline, highlighting specific parser types applied to the log data before data is compressed, stored in the repository, and indexed for searching:
Before you begin
Identify the Log Collector repository that will receive the data.
Ensure the source logs are being written to an Amazon S3 bucket.
Identify or create the parser that Log Collector should apply to the incoming data.
Ensure that you have permissions in AWS to create or modify SQS queues, S3 event notifications, and IAM roles and policies.
Ensure AWS Security Token Service (STS) is activated for the AWS account.