Other Options
Configuring for Use With Non-Default Endpoints

You can point to your own hosting endpoint for the S3 to use for bucket storage if you host an S3-compatible service.

There are two styles of S3 base URL LogScale can use, depending on which URLs your service supports.

Virtual host style (default)

LogScale will construct virtual host-style URLs like: 

https://my-bucket.my-own-s3:8080/path/inside/bucket/file.txt

For this style of access, you need to set your base URL, so it contains a placeholder for the bucket name.

ini
S3_STORAGE_ENDPOINT_BASE=http://{bucket}.my-own-s3:8080

LogScale will replace the placeholder, {bucket} with the relevant bucket name at runtime.

Path-style

Some services do not support virtual host style access, and require path-style access.

Such URLs have the format:

uri
https://my-own-s3:8080/my-bucket/path/inside/bucket/file.txt

If you are using such a service, your endpoint base URL should not contain a bucket placeholder.

ini
S3_STORAGE_ENDPOINT_BASE=http://my-own-s3:8080

Additionally, you must set S3_STORAGE_PATH_STYLE_ACCESS to true

Hitachi Content Platform compatibility

Bucket storage can be used with the Hitachi Content Platform by setting S3_STORAGE_HCP_COMPAT to true.

HTTP Proxy

If LogScale is set up to use an HTTP proxy, it will per default be used for communicating with S3. It can be disabled using the following:

ini
# Use the globally configured HTTP proxy for communicating with S3.
# Default is true.
S3_STORAGE_USE_HTTP_PROXY=false
Amazon Bucket Storage Parameters

The following allows tuning for performance. There may be a costs associated with increasing these variables as S3 is billed also based on the number of operations executed.

ini
# How many parallel chunks to split each file into when uploading and downloading.
# Defaults to 4, the maximum is 16.
S3_STORAGE_CHUNK_COUNT=4

# Maximum number of files that LogScale will run concurrent downloads for at a time.
# Default is the number of hyperthreads / 2
S3_STORAGE_DOWNLOAD_CONCURRENCY=8

# Maximum number of files that LogScale will run concurrent uploads for at a time.
# Default is the number of hyperthreads / 2
# S3_STORAGE_UPLOAD_CONCURRENCY=8

# Chunk size for uploads and download ranges. Max 8 MB, which is the default.
# Minimum value is 5 MB.
S3_STORAGE_CHUNK_SIZE=8388608

# Prefer to fetch data files from the bucket when possible even if another
# node in the LogScale cluster also has a copy of the file.
# In some environments, it may be less expensive to do the transfers this way.
# The transfer from the bucket may be billed at a lower cost than a transfer from
# a node in another region or in another data center.  This preference does not
# guarantee that the bucket copy will be used, as the cluster can
# still make internal replicas directly in case the file is not yet in
# a bucket.
# Default is false.
S3_STORAGE_PREFERRED_COPY_SOURCE=false