Other options

Deprecated:DEPRECATED: Generic s3-compatible storage

Because storage vendors have chosen to implement s3-compatibility in wildly different ways with variable levels of completeness, cannot support generic s3-compatible storage at this time. is currently evaluating several storage options that are s3-compatible to help us better understand what level of compatibility is fully required to meet LogScale requirements. These instructions are preserved for existing customers who have adopted this form of storage in the past. However, use of "s3-compatible" products is purely at your own risk and neither supported nor recommended by CrowdStrike at this time.

Configure for use with non-default endpoints

You can point to your own hosting endpoint for S3 to use for bucket storage if you host an S3-compatible service.

There are two styles of S3 base URL can use, depending on which URLs your service supports.

Virtual host style (default)

will construct virtual host-style URLs such as: 

https://my-bucket.my-own-s3:8080/path/inside/bucket/file.txt

For this style of access, set your base URL, so it contains a placeholder for the bucket name.

ini
S3_STORAGE_ENDPOINT_BASE=http://{bucket}.my-own-s3:8080

will replace the placeholder {bucket} with the relevant bucket name at runtime.

Path-style

Some services do not support virtual host style access, and require path-style access.

Such URLs have the format:

uri
https://my-own-s3:8080/my-bucket/path/inside/bucket/file.txt

If you are using such a service, your endpoint base URL should not contain a bucket placeholder.

ini
S3_STORAGE_ENDPOINT_BASE=http://my-own-s3:8080

Additionally, you must set the S3_STORAGE_PATH_STYLE_ACCESS configuration variable to true.

HTTP Proxy

If is set up to use an HTTP proxy, it will be used for communicating with S3 by default. To disable it, set:

ini
# Use the globally configured HTTP proxy for communicating with S3.
# Default is true.
S3_STORAGE_USE_HTTP_PROXY=false
Amazon Bucket Storage Parameters

The following configurations allow tuning for performance. There may be a cost associated with increasing these variables as S3 is billed based on the number of operations executed.

ini
# How many parallel chunks to split each file into when uploading and downloading.
# Defaults to 4, the maximum is 16.
S3_STORAGE_CHUNK_COUNT=4

# Maximum number of files that  will run concurrent downloads for at a time.
# Default is the number of hyperthreads / 2
S3_STORAGE_DOWNLOAD_CONCURRENCY=8

# Maximum number of files that  will run concurrent uploads for at a time.
# Default is the number of hyperthreads / 2
# S3_STORAGE_UPLOAD_CONCURRENCY=8

# Chunk size for uploads and download ranges. Max 8 MB, which is the default.
# Minimum value is 5 MB.
S3_STORAGE_CHUNK_SIZE=8388608

# Prefer to fetch data files from the bucket when possible even if another
# node in the  cluster also has a copy of the file.
# In some environments, it may be less expensive to do the transfers this way.
# The transfer from the bucket may be billed at a lower cost than a transfer from
# a node in another region or in another data center.  This preference does not
# guarantee that the bucket copy will be used, as the cluster can
# still make internal replicas directly in case the file is not yet in
# a bucket.
# Default is false.
S3_STORAGE_PREFERRED_COPY_SOURCE=false

By default, once uploaded to S3, files are not overwritten. uses an If-None-Match header, so any attempt at additional upload to a key that already exists in a bucket will fail prior to any file data bytes actually being sent. A handleFailedMultipartUpload message in logs means that the file upload failed. You might also see s3.statusCode() == 412. Status code 412 means that the file already exists in the bucket and the If-None-Match header is in place, preventing overwrites of files.

To override this behavior in cases where some other storage provider (an S3 clone) is used that fails to support this setting, change S3_STORAGE_DISABLE_UPLOAD_DUPLICATE_CHECK to true.