LogScale supports multiple ways of configuring access to the bucket, allowing you to use any of the 5 listed in Using the Default Credential Provider Chain on top of the following, which takes precedence if set.

If you run your LogScale installation outside AWS, you need an IAM user with write access to the buckets used for storage. That user must have programmatic access to S3, so when adding a new user through the AWS console make sure programmatic access is set.

Figure 85. Self-Hosting with S3

Later in the process, you can retrieve the access key and secret key:

Figure 86. Self-Hosting with S3

To configure S3 storage, the access key and secret key from the S3 bucket configuration must be set using the corresponding S3_STORAGE_ACCESSKEY and S3_STORAGE_SECRETKEY configuration variables.

For example:

# These two take precedence over all other AWS access methods.
S3_STORAGE_ACCESSKEY=$ACCESS_KEY
S3_STORAGE_SECRETKEY=$SECRET_KEY
S3_STORAGE_ENCRYPTION_KEY=off

Alternative variables are also supported:

# Also supported:
AWS_ACCESS_KEY_ID=$ACCESS_KEY
AWS_SECRET_ACCESS_KEY=$SECRET_KEY

The keys are used for authenticating the user against the S3 service. For more guidance on how to retrieve S3 access keys, see AWS access keys. More details on creating a new user in IAM.

Configuring the user to have write access to a bucket can be done by attaching the right access policy to the user.

Amazon S3 enforces server-side encryption of stored data since January 2023. Explicit configuration of an encryption key for storage of bucket data must be configured. Without explicit declaration of an encryption key, S3 bucket key encryption in LogScale is required due to the checksum reported by S3 being different to the checksum calculated by LogScale.

Best practice is to use the key managed by Amazon Service Side Encryption (SSE) with KMS support turned on, in place of the key configured and stored on disk in plain text.

KMS support in LogScale is controlled via the S3_STORAGE_KMS_KEY_ARN configuration.

To switch off additional LogScale encryption in LogScale:

S3_STORAGE_ENCRYPTION_KEY=off

This will allow data stored in S3 to be encrypted using the the Amazon S3 configured key (see Protecting data with server-side encryption for more information).

Using a custom key is supported, and can be configured to encrypt the bucket data by setting the encryption key explicitly:

S3_STORAGE_ENCRYPTION_KEY=customencryptionkey

This key will be used in addition to the SSE key used.

When using this option, LogScale automatically disables checking of the encryption tag information once AWS Key Management Service is configured (through the S3_STORAGE_KMS_KEY_ARN environment variable).

The following JSON is an example policy configuration:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::BUCKET_NAME"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
      "Resource": ["arn:aws:s3:::BUCKET_NAME/*"]
    }
  ]
}

The policy can be used as an inline policy attached directly to the user through the AWS console:

Figure 87. IAM User Example Policy

You must also tell which bucket to use. These settings must be identical on all nodes across the entire cluster.

S3_STORAGE_BUCKET=$BUCKET_NAME
S3_STORAGE_REGION=$BUCKET_REGION
S3_STORAGE_ENCRYPTION_KEY=$ENCRYPTION_SECRET
S3_STORAGE_OBJECT_KEY_PREFIX=/basefolder
USING_EPHEMERAL_DISKS=true

The first option here is to set the name of the bucket to use. The encryption key given with S3_STORAGE_ENCRYPTION_KEY can be any UTF-8 string. The suggested value is 64 or more random ASCII characters.

The S3_STORAGE_OBJECT_KEY_PREFIX is used to set the optional prefix for all object keys allows multiple LogScale clusters to use the same bucket. The prefix is unset by default.