Ingest Data from AWS S3

Security Requirements and Controls

CrowdStrike Falcon LogScale has the ability to ingest logs from AWS S3 buckets, which can then be managed in LogScale and leveraged using queries, alerts and alarms. In the following we will run through the configuration process of ingesting this data.

Amazon Web Services log data is an extremely valuable data source that comes in a variety of flavors depending on the services you are looking to learn more about. Some of the most common data sources include AWS VPC flow™, CloudTrail™ and CloudWatch™. These logs can be directed to S3 buckets where they are often ingested by LogScale.

Ingesting data using this method operates through the use of the Amazon Simple Queue Service (SQS) to provide the information about which S3 buckets to read for data, as shown in the following diagram:

graph LR %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% s3[("Amazon S3")] Q[SQS Topic] QS[SQS Subscriber] LS[LogScale] s3 <--> LS Q --Message--> QS QS --> LS

During process:

  1. LogScale reads a message from the SQS queue containing the information about the S3 bucket where the data is located.

  2. LogScale downloads the raw data from S3, and parses and ingests the files in the bucket.

  3. LogScale removes the message from the queue to preview re-processing the material.

For more details on these logs, see here.

Prerequisites for Ingesting AWS Data

To follow these steps, you will need:

  • Access to AWS and basic knowledge of AWS architecture.

  • To configure your source in AWS to log to an S3 bucket refer to documentation. This can be to a separate bucket or a directory within a bucket. These log files will then be pulled into Falcon LogScale for analysis and visualization, the format of the data can be line-delimited or AWS JSON events. AWS example events are referenced here.

  • Access to a Falcon LogScale environment, with a repository where you want to ingest the data.

  • Change ingest feed permission

  • Performed the S3 Ingest Self-hosted Preparation before using ingest feeds in self hosted scenarios.

Once these requirements are met, you are ready to follow Set up a New Ingest Feed and configure your ingest feeds.

Monitor Ingest Feeds

Errors during ingest will be shown within the Ingest feeds page.

To display the Ingest feeds page and monitor ingest feeds:

  1. Go to Repositories and views page and select a relevant repository in which you want to ingest the data.

  2. Click Settings, under Ingest on the side menu click Ingest feeds.

The Ingest feeds page shows the individual feeds, their configuration and current status:

  • Name

    Name of the ingest feed given during configuration.

  • Preprocessing

    A description of any preprocessing performed on the data before it is provided to the parser.

  • Parser

    The configured parser for this ingest feed.

  • Last activity

    The time of the last ingest activity for this feed.

  • Status

    The current status of the feed, including whether there is an error or if the feed has been disabled.

    Transient errors during ingest should automatically be cleared. You can click on the status message to get more information.

To edit, delete or enable/disable an existing feed, click the ⋮ icon next to each feed configuration. To create a new feed, click the + New Ingest feed button.