Amazon Bucket Storage
LogScale supports writing a copy of the ingested logs to Amazon S3 and Google Cloud Storage using the native file format of LogScale, allowing LogScale to fetch those files and search them efficiently if the local copies are lost or deleted. This page will explain how to set up bucket storage with Amazon S3. For more details on this topic in general, see the Bucket Storage page.
If you use S3 for bucket storage, two options for encryption are supported (you can apply both):
LogScale-implemented and enforced, not relying on any support from the bucket provider. It is controlled by the configuration variable
S3_STORAGE_ENCRYPTION_KEY, which is applied whenever its value is not set to
off(because this value would disable the use of an encryption key).
AWS-implemented. Access to objects and decryption is managed by AWS as the object storage provider. To ensure better security using AWS supplied encryption, you may want to turn on KMS key for encrypting objects in S3 via the
S3_STORAGE_KMS_KEY_ARNconfiguration — for more information on how Key Management Service works on the AWS side, see Amazon S3 documentation.
Configuring an Amazon S3 Bucket
LogScale supports multiple ways of configuring access to the bucket, allowing you to use any of the 5 listed in Using the Default Credential Provider Chain on top of the following, which takes precedence if set.
If you run your LogScale installation outside AWS, you need an
IAM user with write
access to the buckets used for storage. That user must have
programmatic access to S3, so when adding a new user through the AWS
console make sure
access is set.
Figure 87. Self-Hosting with S3
Later in the process, you can retrieve the access key and secret key:
Figure 88. Self-Hosting with S3
Enabling S3 Storage in LogScale
See Encryption of S3 Bucket Data for more information on the LogScale encryption key used for storing bucket data. In the example below, use of a LogScale encryption key has been disabled.
# These two take precedence over all other AWS access methods.
Alternative variables are also supported:
# Also supported:
Configuring the user to have write access to a bucket can be done by attaching the right access policy to the user.
Encryption of S3 Bucket Data
Amazon S3 enforces server-side encryption of stored data since January 2023. Explicit configuration of an encryption key for storage of bucket data must be configured. Without explicit declaration of an encryption key, S3 bucket key encryption in LogScale is required due to the checksum reported by S3 being different to the checksum calculated by LogScale.
Best practice is to use the key managed by Amazon Service Side Encryption (SSE) with KMS support turned on, in place of the key configured and stored on disk in plain text.
KMS support in LogScale is controlled via the
To switch off additional LogScale encryption in LogScale:
This will allow data stored in S3 to be encrypted using the the Amazon S3 configured key (see Protecting data with server-side encryption for more information).
Using a custom key is supported, and can be configured to encrypt the bucket data by setting the encryption key explicitly:
This key will be used in addition to the SSE key used.
When using this option, LogScale automatically disables checking of
the encryption tag information once AWS Key Management Service is
configured (through the
IAM User Example Policy
The following JSON is an example policy configuration:
"Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
The policy can be used as an inline policy attached directly to the user through the AWS console:
Figure 89. IAM User Example Policy
You must also tell which bucket to use. These settings must be identical on all nodes across the entire cluster.
The first option here is to set the name of the bucket to use. The
encryption key given with
can be any UTF-8 string. The suggested value is 64 or more random
S3_STORAGE_OBJECT_KEY_PREFIX is used to set the
optional prefix for all object keys allows multiple LogScale
clusters to use the same bucket. The prefix is unset by default.
There is a performance penalty when using a non-empty prefix. We
recommend an unset prefix. If there are any ephemeral disks in the
cluster, you must set the last option here to
Switching to a Fresh Bucket
You can change the settings using the
S3_STORAGE_REGION to point to a fresh bucket at any
point in time. From that point, LogScale will write new files to
that bucket while still reading from any previously-configured
buckets. Existing files already written to any previous bucket
will not get written to the new bucket. LogScale will continue to
delete files from the old buckets that match the file names that
LogScale would put there.