Amazon Web Services CloudTrail

This package provides a parser for Amazon Web Services (AWS) CloudTrail events in JSON format.

Breaking Changes

This update includes parser changes, which means that data ingested after upgrade will not be backwards compatible with logs ingested with the previous version.

Updating to version 1.0.0 or newer will therefore result in issues with existing queries in for example dashboards or alerts created prior to this version.

See CrowdStrike Parsing Standard (CPS) for more details on the new parser schema.

Follow the CPS Migration to update your queries to use the fields and tags that are available in data parsed with version 1.0.0.

Configurations and Sending The Logs to LogScale

  1. First you need to follow Amazon's instructions to configure CloudTrail to send logs into an S3 bucket of your choice.

  2. Then you need to configure LogScale to collect data from AWS S3 buckets using the cloudtrail parser. Ingesting Data from AWS S3 or self-hosted Ingesting Data from AWS S3 LogScale deployments to send logs directly from S3 bucket into LogScale repository.

Installing the Amazon CloudTrail Package in LogScale

Find the repository where you want to send the CloudTrail logs, or create a new one.

  1. Navigate to your repository in the LogScale interface, click Settings and then Packages on the left.

  2. Click Marketplace and install the LogScale package for CloudTrail (i.e. amazon/cloudtrail).

Verify Data is Arriving in LogScale

Once you have completed the above steps the CloudTrail data should be arriving in your LogScale repository.

You can verify this by doing a simple search for #Vendor = "amazon" | Product = "cloudtrail" to see the CloudTrail events.

Package Contents Explained

This package is only for parsing incoming data, and normalizing the data as part of that parsing. The parser normalizes the data to a subset of this schema based on OpenTelemetry standards, while still preserving the original data.

If you want to search using the original field names and values, you can access those in the fields whose names are prefixed with the word "Vendor". Fields which are not prefixed with "Vendor" are standard fields which are either based on the schema (e.g. source.ip) or on LogScale conventions (e.g. @rawstring).

The fields which the parser currently maps the data to, are chosen based on what seems the most relevant, and will potentially be expanded in the future. But the parser won't necessarily normalize every field that has potential to be normalized.

Event Categorisation

As part of the schema, events are categorized by four different fields:

  • event.kind

  • event.category

  • event.type

  • #event.outcome

(#event.outcome is a tag, hence the "#")

event.kind and #event.outcome can be searched as normal fields, but event.category and event.type are arrays, so need to be searched like so: 

array:contains("event.category[]", value="api")

This will find events where some event.category[n] field contains the value api, regardless of what n is. Note that not all events will be categorized to this level of detail.

Normalized Fields

Here are some of the normalized fields which are being set by this parser:

  • user.* (e.g. user.name,user.id)

  • user_agent.* (e.g. user_agent.original)

  • source.* (e.g. source.address, source.ip)

  • error.* (e.g. error.messageerror.code)

  • event.* (e.g. event.kindevent.type, event.module, event.action, event.id, event.outcome, event.category, event.provider)

  • file.* (e.g. file.path, file.has.sha)

  • ecs* (e.g. ecs.version )

  • hash* (e.g. hash.sha )