Amazon Web Services CloudTrail (Preview)

This package provides a preview parser for Amazon Web Services (AWS) CloudTrail events in JSON format.

The parser normalizes data to a common schema based on an OpenTelemetry standard. This schema allows you to search the data without knowing Amazon's data specifically, and just knowing the common schema instead. It also allows you to combine the data more easily with other data sources which conform to the same schema.

Preview Status

Note that this package is considered a PREVIEW. This means we are seeking feedback on the package, and may make breaking changes to the parser in the future. There is also the existing aws/cloudtrail package if you would like to use a package which is considered more stable.

In time, this in-preview package may replace the existing one, but the existing one will remain supported for the foreseeable future.

Configurations and Sending The Logs to LogScale

  1. First you need to follow Amazon's instructions to configure CloudTrail to send logs into an S3 bucket of your choice.

  2. Then you need to configure LogScale to collect data from AWS S3 buckets using the cloudtrail parser. Ingesting Data from AWS S3 or self-hosted Ingesting Data from AWS S3 LogScale deployments to send logs directly from S3 bucket into LogScale repository.

Installing the Amazon CloudTrail Package in LogScale

Find the repository where you want to send the CloudTrail logs, or create a new one.

  1. Navigate to your repository in the LogScale interface, click Settings and then Packages on the left.

  2. Click Marketplace and install the LogScale package for CloudTrail (i.e. amazon/cloudtrail).

Verify Data is Arriving in LogScale

Once you have completed the above steps the CloudTrail data should be arriving in your LogScale repository.

You can verify this by doing a simple search for #Vendor = "amazon" | Product = "cloudtrail" to see the CloudTrail events.

Package Contents Explained

This package is only for parsing incoming data, and normalizing the data as part of that parsing. The parser normalizes the data to a subset of this schema based on OpenTelemetry standards, while still preserving the original data.

If you want to search using the original field names and values, you can access those in the fields whose names are prefixed with the word "Vendor". Fields which are not prefixed with "Vendor" are standard fields which are either based on the schema (e.g. source.ip) or on LogScale conventions (e.g. @rawstring).

The fields which the parser currently maps the data to, are chosen based on what seems the most relevant, and will potentially be expanded in the future. But the parser won't necessarily normalize every field that has potential to be normalized.

Event Categorisation

As part of the schema, events are categorized by four different fields:

  • event.kind

  • event.category

  • event.type

  • #event.outcome

(#event.outcome is a tag, hence the "#")

event.kind and #event.outcome can be searched as normal fields, but event.category and event.type are arrays, so need to be searched like so:

array:contains("event.category[]", value="api")

This will find events where some event.category[n] field contains the value api, regardless of what n is. Note that not all events will be categorized to this level of detail.

Normalized Fields

Here are some of the normalized fields which are being set by this parser:

  • cloud.* (e.g. cloud.region, cloud.account.id)

  • user.* (e.g. user.name)

  • source.* (e.g. source.address)