Ingesting Data from AWS S3

Security Requirements and Controls

CrowdStrike Falcon LogScale has the ability to ingest logs from AWS S3 buckets, which can then be managed in LogScale and leveraged using queries, alerts and alarms. In the following we will run through the configuration process of ingesting this data.

Amazon Web Services log data is an extremely valuable data source that comes in a variety of flavors depending on the services you are looking to learn more about. Some of the most common data sources include AWS VPC flow™, Cloudtrail™ and CloudWatch™. These logs can be directed to S3 buckets where they are often ingested by LogScale.

For more details on these logs, see here.

Prerequisites for Ingesting AWS Data

To follow these steps, you will need:

  • Access to AWS and basic knowledge of AWS architecture. You will also need to configure your source in AWS to log to an S3 bucket refer to documentation. This can be to a separate bucket or a directory within a bucket. These log files will then be pulled into Falcon LogScale for analysis and visualization, the format of the data can be line-delimited or AWS JSON events. AWS example events are referenced here.

  • Access to a Falcon LogScale environment, with a repository where you want to ingest the data.

  • Change ingest feed permission

  • Performed the S3 Ingest Self-hosted Preparation before using ingest feeds in self hosted scenarios.

Once this these requirements have been satisfied, you are ready to follow Setting up a New Ingest Feed and configure your ingest feeds.

Monitoring Ingest Feeds

Errors during ingest will be shown within the Ingest feeds page.

The page shows the individual feeds, their configuration and current status:

  • Name

    Name of the ingest feed given during configuration.

  • Preprocessing

    A description of any preprocessing performed on the data before it is provided to the parser.

  • Parser

    The configured parser for this ingest feed.

  • Last activity

    The time of the last ingest activity for this feed.

  • Status

    The current status of the feed, including whether there is an error or if the feed has been disabled.

    Transient errors during ingest should automatically be cleared. You can click on the status message to get more information.

To edit an existing feed, use the ⋮ next to each feed configuration. To create a new feed, blick the + New Ingest feed button.