Amazon S3 Server Access (Preview)

Server access logging provides detailed records for the requests that are made to a bucket. This package parses this data in LogScale for efficient security and access auditing.

The parser normalizes data to a common schema based on an OpenTelemetry standard. This schema allows you to search the data without knowing the data specifically, and just knowing the common schema instead. It also allows you to combine the data more easily with other data sources which conform to the same schema.

Preview Status

Note that this package is considered a PREVIEW. This means we are seeking feedback on the package, and may make breaking changes to the parser in the future. It means also it may not include all documentation which usually we publish for packages.

Given the preview status of this package there are some limitations to what the parser is able to do. Please check out the package on Marketplace to see the most recent list of limitations.

Installing the Package in LogScale

Find the repository where you want to send the Amazon S3 Server Access (Preview) events, or Creating a Repository or View.

  1. Navigate to your repository in the LogScale interface, click Settings and then Packages on the left.

  2. Click Marketplace and install the LogScale package for Amazon S3 Server Access (Preview) (i.e. aws/s3access).

Configurations and Sending the Logs to LogScale

  1. To get logs from AWS S3 Server Access into LogScale, configure the S3 Server Access logging to send logs to an S3 bucket.

  2. Then you need to configure LogScale to collect data from AWS S3 buckets using the s3access-space-delimited. See the documentation for cloud: Ingesting Data from AWS S3 and self-hosted Ingesting Data from AWS S3 deployments to send logs directly from S3 bucket into LogScale repository.

Verify Data is Arriving in LogScale

Once you have completed the above steps the Amazon S3 Server Access (Preview) data should be arriving in your LogScale repository.

You can verify this by doing a simple search for the events:

logscale
#Vendor = "amazon" 
| Product="s3access"

Package Contents Explained

This package parses incoming data, and normalizing the data as part of that parsing. The parser normalizes the data to a subset of this schema based on OpenTelemetry standards, while still preserving the original data.

If you want to search using the original field names and values, you can access those in the fields whose names are prefixed with the word Vendor. Fields which are not prefixed with Vendor are standard fields which are either based on the schema (e.g. source.ip) or on LogScale conventions (e.g. @rawstring).

The fields which the parser currently maps the data to, are chosen based on what seems the most relevant, and will potentially be expanded in the future. But the parser won't necessarily normalize every field that has potential to be normalized.

Event Categorisation

As part of the schema, events are categorized by fields:

  • event.category

  • event.kind

  • event.outcome

  • event.type

event.category is an array so needs to be searched using the following syntax:

logscale
array:contains("event.category[]", value="web")

For example, the following will find events where some event.type[n] field contains the value network, regardless of what n is.

Note that not all events will be categorized to this level of detail.

Normalized Fields

Here are some of the normalized fields which are being set by this parser:

  • event.* (e.g. event.type, event.kind, event.category )

  • client.* (e.g. client.ip)

  • http.* (e.g. http.request.referrer.id, http.version )

  • tls.* (e.g. tls.version_protocol, tls.version )