Ingest Data from AWS S3
Security Requirements and Controls
Change ingest feeds
permission
CrowdStrike Falcon LogScale has the ability to ingest logs from AWS S3 buckets, which can then be managed in LogScale and leveraged using queries, alerts and alarms. In the following we will run through the configuration process of ingesting this data.
Amazon Web Services log data is an extremely valuable data source that comes in a variety of flavors depending on the services you are looking to learn more about. Some of the most common data sources include AWS VPC flow™, CloudTrail™ and CloudWatch™. These logs can be directed to S3 buckets where they are often ingested by LogScale.
Ingesting data using this method operates through the use of the Amazon Simple Queue Service (SQS) to provide the information about which S3 buckets to read for data, as shown in the following diagram:
During process:
LogScale reads a message from the SQS queue containing the information about the S3 bucket where the data is located.
LogScale downloads the raw data from S3, and parses and ingestes the files in the bucket.
LogScale removes the message from the queue to preview re-processing the material.
For more details on these logs, see here.
Prerequisites for Ingesting AWS Data
To follow these steps, you will need:
Access to AWS and basic knowledge of AWS architecture.
To configure your source in AWS to log to an S3 bucket refer to documentation. This can be to a separate bucket or a directory within a bucket. These log files will then be pulled into Falcon LogScale for analysis and visualization, the format of the data can be line-delimited or AWS JSON events. AWS example events are referenced here.
Access to a Falcon LogScale environment, with a repository where you want to ingest the data.
Change ingest feed
permissionPerformed the S3 Ingest Self-hosted Preparation before using ingest feeds in self hosted scenarios.
Once these requirements are met, you are ready to follow Set up a New Ingest Feed and configure your ingest feeds.
Monitor Ingest Feeds
Errors during ingest will be shown within the Ingest
feeds
page.
To display the Ingest feeds
page and
monitor ingest feeds:
![]() |
The Ingest feeds
page shows the
individual feeds, their configuration and current status:
Name
Name of the ingest feed given during configuration.
Preprocessing
A description of any preprocessing performed on the data before it is provided to the parser.
Parser
The configured parser for this ingest feed.
Last activity
The time of the last ingest activity for this feed.
Status
The current status of the feed, including whether there is an error or if the feed has been disabled.
Transient errors during ingest should automatically be cleared. You can click on the status message to get more information.
To edit, delete or enable/disable an existing feed, click the ⋮ icon next to each feed configuration. To create a new feed, click the button.