Ingesting FDR Data into a Repository

To ingest any data from FDR, you must have at least one FDR feed established on a Falcon instance with data flowing into it.

Important

Non-FDR data should not be ingested into an FDR repository.

To set up an FDR feed on your Falcon instance, please follow the instructions in the official Falcon documentation.

Note

Falcon documentation is only available to CrowdStrike customers; you must have a Falcon instance in order to read it.

Once you have an FDR feed set up in Falcon, you can make LogScale ingest data from it into a particular repository by creating an FDR feed component.

When an FDR feed component has been created, it will spawn a polling process on all designated LogScale nodes, which will then poll data from the FDR feed and ingest it into the LogScale repository.

In Falcon, an FDR Feed consists of two AWS resources, an SQS queue and an S3 bucket.

As log data is generated within Falcon, it will be stored in the S3 bucket as files, and the SQS queue will have messages added to it, which point to those files.

The basic flow of the polling process spawned within LogScale is:

  • Pull a message off the SQS queue.

  • Identify S3 file identifiers contained within the message.

  • Download the files from the S3 bucket.

  • Ingest the file contents into the repository.

  • Delete the message from the SQS queue, so it is not read again.

To create an FDR feed component so that polling can commence, decide which repository to ingest into and then:

  1. Go to Repositories and views page and select a relevant repository.

  2. Click Settings, under Ingest in the side menu click CrowdStrike FDR.

  3. On the CrowdStrike FDR page, click New FDR Feed.

  4. In the New FDR feed dialog box, fill out the fields prompted.

    See Parser, Client ID & Client Secret, S3 Identifier, and SQS URL for more details.

  5. Click Save.

Alternatively, you can use the GraphQL API to view, create, update and delete FDR feed components using the associated queries and mutations.

In the following, we cover some of the fields that must be set on an FDR feed component during creation.

Note

The contents of most of these fields can be copied directly from the Falcon UI and relate to the SQS queue and S3 bucket.

You can find them under the API Clients and Keys page for your Falcon instance relating to your particular FDR feed.

Parser

Specifies which LogScale parser should be used to parse the FDR data upon ingest.

We recommend that you use the official FDR parser for this, which you can get access to by installing the crowdstrike/fdr package into your repository.

To install it in a repository go to Settings, under Packages in the side menu click Marketplace, select the crowdstrike/fdr application and click Install package.

For more details, read about Packages.

Client ID & Client Secret

The client ID and client secret are AWS credentials used for accessing the AWS resources.

Important

You are only shown the client secret once in the Falcon UI, so you must make sure to store it in a safe location for later use.

Likewise, LogScale treats the client secret as a confidential field.

This means that it cannot be read. After you have saved your FDR component, it can only be overwritten.

S3 Identifier

This URI is a unique identifier for the AWS S3 bucket, which is a part of the FDR Feed.

logscale
s3://<BUCKET_NAME>/data

Where BUCKET_NAME is an id unique to your bucket.

The identifier is used to validate that messages pulled from the SQS queue point to the expected S3 bucket before any ingest takes place.

SQS URL

This URL is an identifier for the AWS SQS queue, which is a part of the FDR Feed.

logscale
"https://sqs.us-west-2.amazonaws.com/"