Bucket storage allows you to run LogScale without any persistent disks. In fact, doing so is encouraged when setting up the system this way, as persistent disks in a cloud environment require network traffic and that bandwidth is better put to use for bucket storage and for networking internally in the cluster.

Using network attached disks in combination with bucket storage is discouraged in cloud environments. Often you end up using much more network bandwidth this way.

Compression type should be set at "high" at level 9 (as is the default in 1.7.0 and later) to ensure that the files transferred to/from the bucket get as much compression applied as possible. This reduces the storage requirement both locally and in the bucket, reduces network traffic and improves search performance when data is not present locally.

When using ephemeral disks the choice of node ID needs selecting in a controlled manner, and the strategy of storing it in the file system no longer works. See the option ZOOKEEPER_PREFIX_FOR_NODE_UUID in the configuration page for a method of assigning unique node IDs to nodes using ZooKeeper.

Combining the feature of "backup to a remote file system" and Bucket Storage is discouraged. Bucket storage replaces all functionality in the previous backup system.

LogScale can manage which segment files to keep on the local file system based on the amount of disk space used, and delete local files that also exist in bucket storage. This allows you to keep more data files than the local disk has room for, essentially allowing for infinite storage of events, at the cost of paying for S3 storage and (potentially) transfer costs when the files required for a search are not present on any node.

A copy in a bucket counts as a safe replica that allows the cluster to delete all local copies of the file and still report it as "perfectly replicated" from a cluster management perspective.

LogScale keeps track of which segment files are read by searches. When the disk is filled more than the desired percentage, LogScale will delete the least recently used segment files to free up disk space. Only files that have been copied to a bucket are ever removed.

If a file is not present locally and a search needs to read it, the search runs on all the parts that are present locally and schedules a download of the missing segment files from the bucket in the background. Once the required files arrive, they get searched as if they had been there from the start.

The files get downloaded by (one of) the hosts that is assigned to the storage partition for the file at the time of the search starts.

Files downloaded from the bucket to satisfy a search are kept in the secondary data directory if configured, or the primary data dir if no secondary is set. The downloaded segment files are treated like the recently created segment files, and are subject to the same rules regarding deleting them again when they become the least recently accessed file at some point.

Since LogScale can search the data at a much faster pace than the network and bucket storage provider can deliver them, the performance of searches that need files from the bucket provider will be orders of magnitude slower than searches that only access local files (on fast drives.) The good use case for over-committing the file system like this is to lower the cost of hosting LogScale trading in performance of searches in old events that happen to not be on the local disk.

The local file system needs to have room for at least accommodate all the mini segments and the most recent merged segments. Providing more local disk space allows LogScale to keep more files in the cache, resulting in better search performance.

Hash filter files are also stored in the bucket and also get downloaded when a search requires the segment file.

# Percentage of disk full that LogScale aims to keep the disk at.
# Segment files will be deleted in a least recently used order, in order to hit the configured fill target.

LOCAL_STORAGE_PERCENTAGE=80

# Minimum number of days to keep a fresh segment file before allowing
# it to get deleted locally after upload to a bucket.
# Setting such a lower bound can help ensure that recent files are kept on disk,
# even if they would otherwise be evicted due to queries on older data.
# Mini segment files are kept in any case until their merge result also exists.
# (The age is determined using the timestamp of the most recent event in the file)
# Make sure to leave most of the free space as room for the system to
# manage as a mix of recent and old files.
# Note! Min age takes precedence over the fill percentage, so increasing this
# implies the risk of overflowing the local file system!

LOCAL_STORAGE_MIN_AGE_DAYS=0

LogScale supports a disaster recovery method based on bucket storage. The objects stored in the target bucket of a cluster can form the basis for bootstrapping a fully independent cluster. The new cluster needs to have a bucket of its own to store objects in, and will consider the "source" bucket as read only. It relies on the source bucket being immutable from the point the disaster recovery process starts - don't point to the bucket of another running LogScale cluster.

All example configs use S3_ as prefix, use GCP_ if applicable.

Common use cases for disaster recovery using this method include:

In all cases these are the steps that need to happen:

The LogScale nodes will start fetching segment files from the "old" bucket and upload them to the new target, which will over time make the cluster have an independent copy of the full contents of the old bucket. Once this replication process completes, the new cluster no longer depends on any objects in the source bucket it was bootstrapped from, nor from other buckets that were referred in the snapshot fetched from that bucket.

In order for the recovery procedure to be lossless, the target cluster needs to get a global snapshot from the source cluster that does not reference segment files missing from the bucket. To create such a snapshot, the source cluster needs to be shut down gracefully before initiating the recovery procedure. This is typically not the case for an actual disaster recovery scenario, in which some data loss may be unavoidable, but for the migrate or clone cases it is doable. If the source cluster did not shut down gracefully, then there may be reports of missing segment files in the new target cluster. These can be resolved using the Missing Segments.

LogScale provides a utility function to decrypt a file outside of LogScale that has been uploaded to the bucket by LogScale. This may come in handy as part of an audit process or as part of other processes that involve fetching files directly from the bucket. Usage:

java -cp humio.jar com.humio.main.DecryptAESBucketStorageFile <secret string> <encrypted file> <decrypted file>

Provide the path / filename of the input file to decrypt as argument. The last part of the file name is included in the decryption process, so file names must be preserved identical to those in the bucket for the path after the last /. The "secret string" argument is the value of S3_STORAGE_ENCRYPTION_KEY / GCP_STORAGE_ENCRYPTION_KEY that was in the LogScale config at the time the file was uploaded by LogScale.

To re-ingest log data that has been written to an a bucket through archiving can be achieved by using LogScale Collector and the native JSON parsing within LogScale.

This process has the following requirements:

To re-ingest logs this way:

The Log Collector will read the file that has been copied, send it to LogScale, where the JSON event data will be parsed and recreated.