Other options

Deprecated

DEPRECATED: Because storage vendors have chosen to implement s3-compatibility in wildly different ways with variable levels of completeness, we cannot support generic s3-compatible storage at this time. We are currently evaluating several storage options that are s3-compatible to help us better understand what level of compatibility is fully required to meet LogScale requirements. These instructions are preserved for existing customers who have adopted this form of storage in the past. However, use of "s3-compatible" products is purely at your own risk and neither supported nor recommended by CrowdStrike at this time.

Configure for use with non-default endpoints

You can point to your own hosting endpoint for S3 to use for bucket storage if you host an S3-compatible service.

There are two styles of S3 base URL LogScale can use, depending on which URLs your service supports.

Virtual host style (default)

LogScale will construct virtual host-style URLs such as: 

https://my-bucket.my-own-s3:8080/path/inside/bucket/file.txt

For this style of access, set your base URL, so it contains a placeholder for the bucket name.

ini
S3_STORAGE_ENDPOINT_BASE=http://{bucket}.my-own-s3:8080

LogScale will replace the placeholder {bucket} with the relevant bucket name at runtime.

Path-style

Some services do not support virtual host style access, and require path-style access.

Such URLs have the format:

uri
https://my-own-s3:8080/my-bucket/path/inside/bucket/file.txt

If you are using such a service, your endpoint base URL should not contain a bucket placeholder.

ini
S3_STORAGE_ENDPOINT_BASE=http://my-own-s3:8080

Additionally, you must set the S3_STORAGE_PATH_STYLE_ACCESS configuration variable to true.

HTTP Proxy

If LogScale is set up to use an HTTP proxy, it will be used for communicating with S3 by default. To disable it, set:

ini
# Use the globally configured HTTP proxy for communicating with S3.
# Default is true.
S3_STORAGE_USE_HTTP_PROXY=false
Amazon Bucket Storage Parameters

The following configurations allow tuning for performance. There may be a cost associated with increasing these variables as S3 is billed based on the number of operations executed.

ini
# How many parallel chunks to split each file into when uploading and downloading.
# Defaults to 4, the maximum is 16.
S3_STORAGE_CHUNK_COUNT=4

# Maximum number of files that LogScale will run concurrent downloads for at a time.
# Default is the number of hyperthreads / 2
S3_STORAGE_DOWNLOAD_CONCURRENCY=8

# Maximum number of files that LogScale will run concurrent uploads for at a time.
# Default is the number of hyperthreads / 2
# S3_STORAGE_UPLOAD_CONCURRENCY=8

# Chunk size for uploads and download ranges. Max 8 MB, which is the default.
# Minimum value is 5 MB.
S3_STORAGE_CHUNK_SIZE=8388608

# Prefer to fetch data files from the bucket when possible even if another
# node in the LogScale cluster also has a copy of the file.
# In some environments, it may be less expensive to do the transfers this way.
# The transfer from the bucket may be billed at a lower cost than a transfer from
# a node in another region or in another data center.  This preference does not
# guarantee that the bucket copy will be used, as the cluster can
# still make internal replicas directly in case the file is not yet in
# a bucket.
# Default is false.
S3_STORAGE_PREFERRED_COPY_SOURCE=false