Skip to content
LogScale Self-Hosted Deployment
Content was updated: Oct 24, 2025
This is an archived version of the documentation downloaded from LogScale Documentation Archives. Up to date documentation, and documentation for later versions, is available on LogScale Documentation

## Bucket Storage

LogScale supports writing a copy of the ingested logs to Amazon S3 , Azure bucket storage , and Google Cloud Storage using the native file format of LogScale, allowing LogScale to fetch those files and search them efficiently if the local copies are lost or deleted.

### Important

LogScale only supports AWS, Azure, GCP, or MinIO bucket storage; it does not support use of any other bucket storage technologies.

The files are encrypted by LogScale, as described under Security. For guidance on configuring encryption on Amazon S3, see Encryption of S3 Bucket Data.

### Note

Segments are uploaded to bucket storage as soon as possible once they are closed for writing.

### Why bucket storage

Bucket storage provides several significant features in LogScale. First, it makes cloud deployment of LogScale cheaper, faster, and easier to run using ephemeral disks as primary storage. It also enables recovery from the loss of multiple LogScale nodes without any data loss. New nodes can join the cluster by using just configuration, Kafka, and bucket storage. Plus, bucket storage enables you to keep more events than the local disks can hold, enabling unlimited retention.

The copies written to the bucket are considered "safe" compared to a local copy as the bucket storage provided by S3 and GCP keep multiple copies in multiple locations. A bucket has unlimited storage capacity and can hold an unlimited number of files. This enables the LogScale cluster to fetch the file from the provider in case of loss of node(s) that had the local copies. It also enables configuring LogScale for infinite retention by letting LogScale delete local files that are also in the bucket and fetch the data from the bucket once there is a request to search them.

LogScale manages the set of files in the bucket using the same retention settings configured for each repository. LogScale deletes segment files in the bucket storage at the same time as the segment file has been deleted in LogScale, taking into account the configured backup interval as specified in the `DELETE_BACKUP_AFTER_MILLIS` variable.

As an alternative, you may want to read the S3 Archiving page. It's about how to enable LogScale to export ingested events into S3 buckets in an open format readable by other applications, but not searchable in LogScale.

### Amazon Bucket storage

LogScale supports writing a copy of the ingested logs to Amazon S3 using the native file format of LogScale, enabling LogScale to fetch those files and search them efficiently if the local copies are lost or deleted. For more details on this topic in general, see the Bucket Storage page.

If you use S3 for bucket storage, two options for encryption are supported (you can apply both):

- LogScale-implemented and enforced, not relying on any support from the bucket provider.

- AWS-implemented. Access to objects and decryption is managed by AWS as the object storage provider.

### Bucket Storage with a Cluster in a Cloud Hosting Deployment

Bucket storage enables you to run LogScale without any persistent disks. In fact, doing so is encouraged when setting up the system in the cloud, as persistent disks in a cloud environment require network traffic and that bandwidth is better used for bucket storage and for networking internally in the cluster.

### Note

The Kafka cluster needs to be persistent and independent from the LogScale cluster. It's also important to tell LogScale that the disks are not to be relied upon as permanent storage by setting `USING_EPHEMERAL_DISKS` to `true` so that LogScale won't trust replicas supposedly present on other nodes in the cluster, as they may vanish in this kind of environment. However, you can start out without bucket storage and later migrate to using bucket storage. If you do this set `USING_EPHEMERAL_DISKS` to `false`, and then when the bucket storage has been verified as working correctly, you can set `USING_EPHEMERAL_DISKS` to `true`.

Using network attached disks in combination with bucket storage is discouraged in cloud environments. Often you end up using much more network bandwidth this way.

Set the compression type to "high" at level 9 to ensure that the files transferred to and from the bucket get as much compression applied as possible. This reduces the storage requirement both locally and in the bucket, reduces network traffic, and improves search performance when data is not present locally.

When using ephemeral disks the choice of node ID needs selecting in a controlled manner, and the strategy of storing it in the file system no longer works. See the option `ZOOKEEPER_PREFIX_FOR_NODE_UUID` in the configuration page for a method of assigning unique node IDs to nodes using ZooKeeper.

Combining the feature of "backup to a remote file system" and Bucket Storage is discouraged. Bucket storage replaces all functionality in the previous backup system.

### Target Bucket

LogScale has the concept of a Target bucket, to which new files are uploaded. A node has a single target at any point in time. All buckets that have received a file at some point will get registered in "global" in the LogScale cluster. This enables any node in the cluster to fetch files from all buckets that have ever been a target for any node in the cluster.

Bucket storage should be configured for the entire cluster to ensure that all segment files get uploaded. There are no settings for individual repositories, and all data gets written to a single bucket. It is okay to switch to a fresh bucket at any point in time, also on a subset of the nodes, or in a rolling update fashion. A node uploads to the configured target bucket. Please note that the nodes will need to be able to fetch files from the previous target buckets for as long as there are files in them that they need to fetch. Do not delete or empty the previous target buckets. Make sure not to revoke any authentication tokens required for the reading of these existing files either. When a fresh target is configured, existing files do not get re-uploaded into that. The existing copy in the previous targets is the one it trusts.

### High-Availability

It's possible to achieve high-availability of ingested events using this feature in combination with a Kafka installation that is not on the same file systems as the LogScale segment files. With LogScale set up to have multiple digest nodes for each partition, ingested events are either still in the ingest queue in Kafka, or on one of the LogScale nodes designated as digest node, or in the bucket storage. LogScale can resume from a combination of these after sudden and complete loss of a single LogScale node.

The local data storage directory holds the segment files as they get constructed. Once a segment file is complete, it gets uploaded to the bucket. Files copied to bucket storage also stay on the local disk in most cases. See Managing local disk space on how to allow LogScale to delete local files that also exist in the bucket.

### Note

LogScale also uploads the `global-snapshot.json` files to the bucket every five minutes. This enables starting a node on an empty local data directory if there is a recent snapshot in the bucket configured as target for the node. The node then fetches the snapshot from the bucket and then gets up to date on the current state by reading global events from Kafka that have the changes since the snapshot was written.

For more information on the concept of segments files and datasources, see segments files and Datasources.

### One Node for Each Partition

Partition assignments are still being applied to segment files when configuring bucket storage. While it is possible to run with just one node assigned on both digest and storage partition rules, it's a choice with consequences for availability in case of node failure.

The reason is that only "complete" files get put in the bucket, not the "in-progress work". The partition scheme is applied to decide which hosts are applicable as storage for any particular segment file. LogScale does not let every node download any file from the bucket; only the ones listed on the partitions that apply to the segment.

This means that if you set up storage rules to only a single node on each partition, then a single-node failure would make most queries respond with a subset of the events while the node is down. No data would be lost, but the results would be partial until the node returns to the cluster.

If you set up digest rules to have just one node on any partition, then the failure of such a node will result in incoming events on that partition not being processed by LogScale but just remaining queued in Kafka. This may eventually lead to data loss as Kafka is not set up for infinite retention by default — and doing so would likely overflow Kafka at some point.

Another consideration is that having 2 replicas in both storage and digest partition rules also makes the system resilient to cases where the bucket storage upload is unable to keep up or even fails to accept uploads at all for a while. Such a scenario is a problem if any partition has only a single node assigned, as that node may fail and there are then no other copies of the data.

For storage partition rules, two replicas are recommended to allow queries to complete without responding with a "partial results" warning in case of a single node failure.

The recommendation thus is to run with two digest replicas and two storage replicas on all partitions, for better reliability of the system.

A future version of LogScale will improve this by adding the ability to handle queries without partial results in case of a single node failure, and also in the case of storage partition replicas being set to just one. We will have any node in the cluster able to fetch the segment for query in case of failure of the primary node for the segment. Digest partitions will continue to need at least two for ingest to flow, in case of a single node failure.

### Security

The copies in the bucket are optionally encrypted by LogScale using AES-256 while being uploaded to ensure that even if read access to the bucket is accidentally allowed, an attacker is not able to read any events or other information from the data in the files. The encryption key is based on the seed key string set in the configuration, and each file gets encrypted using a unique key derived from that seed.

Where the bucket service supports or enforces server-side encryption of bucket data with a service defined encryption key, secondary encryption by LogScale can be disabled.

The seed key is stored in global along with all the other information required to read and decrypt the files in that bucket. LogScale handles changes to the encryption seed key by keeping track of which files were encrypted using what secret.

LogScale computes a checksum of the local file (after encryption) while uploading and this checksum must match that computed by the receiver for the upload to complete successfully. This ensures that the bucket does not hold invalid or incomplete copies of the segment files.

To learn more about integrating services like Amazon AWS and Google Cloud into bucket storage with LogScale, see the documentation page on Amazon Bucket Storage or Google Bucket

### S3 Bucket Storage Format and Layout

When uploading a segment file during bucket storage, LogScale creates the S3 object key based on the dataspace ID, datasource ID and segment ID.

LogScale uses the following pattern:

shell

    PREFIX/org_ORGID/dataspace_DATASPACEID/datasource_DATASOURCEID/aes-humiodata.SEGMENTID

Where:

- `PREFIX`

  The prefix, as configured by the `S4_STORAGE_OBJECT_KEY_PREFIX`.

- `DATASPACEID`

  The dataspace ID of the source data.

- `DATASOURCEID`

  The ID of the datasource.

- `SEGMENTID`

  The ID of the individual segment.

An example of this layout can be seen in the file list below:

shell

    $ s3cmd ls -r s3://logscale3/
    2025-07-02 11:41      5852968  s3://logscale3/dataspace_dMeOY0W0O60WDuW6WlF3Cbbw/datasource_RrkB9295WTEIeeLZ7Sj81h9t/aes-humiodata.ylfnPwWiXArqtI4DCJGpBUmg
    2025-07-02 11:40     14728224  s3://logscale3/dataspace_dMeOY0W0O60WDuW6WlF3Cbbw/datasource_dKCnTYSxwifAZfWbPcr8awVE/aes-hash5h3.KK7LCu63EJYapqqSBdUMq4tj
    2025-07-02 11:40     14916732  s3://logscale3/dataspace_dMeOY0W0O60WDuW6WlF3Cbbw/datasource_dKCnTYSxwifAZfWbPcr8awVE/aes-hash5h3.aNAyT5A7oEKgtJPWpii8Kqeb

### Over-Committing Local Disk Space

LogScale can manage which segment files to keep on the local file system based on the amount of disk space used, and delete local files that also exist in bucket storage. This enables you to keep more data files than the local disk has room for, essentially allowing for infinite storage of events, at the cost of paying for S3 storage and (potentially) transfer costs when the files required for a search are not present on any node.

A copy in a bucket counts as a safe replica that enables the cluster to delete all local copies of the file and still report it as "perfectly replicated" from a cluster management perspective.

LogScale keeps track of which segment files are read by searches. When the disk is filled more than the desired percentage, LogScale will delete the least recently used segment files to free up disk space. Only files that have been copied to a bucket are ever removed.

If a file is not present locally and a search needs to read it, the search runs on all the parts that are present locally and schedules a download of the missing segment files from the bucket in the background. Once the required files arrive, they get searched as if they had been there from the start.

The files get downloaded by (one of) the hosts that is assigned to the storage partition for the file at the time of the search starts.

Files downloaded from the bucket to satisfy a search are kept in the primary data directory if configured. The downloaded segment files are treated like the recently created segment files, and are subject to the same rules regarding deleting them again when they become the least recently accessed file at some point.

Since LogScale can search the data much faster than the network and bucket storage provider can deliver them, the performance of searches that need files from the bucket provider will be orders of magnitude slower than searches that only access local files (on fast drives.) Ideally, primary storage would store all data, but the cost for vast quantities of data would make this prohibitive. The use case for using bucket storage is to lower the cost of hosting LogScale by trading in the performance of searches of old events that are not on the local storage.

The local file system needs to have the capacity to store at least all the mini segments and the most recent merged segments. Providing more local disk space enables LogScale to keep more files in the cache, resulting in better search performance.

Hash filter files are also stored in the bucket and also get downloaded when a search requires the segment file.

ini

    # Percentage of disk full that LogScale aims to keep the disk at.
    # Segment files will be deleted in a least recently used order, in order to hit the configured fill target.

    LOCAL_STORAGE_PERCENTAGE=80

    # Minimum number of days to keep a fresh segment file before allowing
    # it to get deleted locally after upload to a bucket.
    # Setting such a lower bound can help ensure that recent files are kept on disk,
    # even if they would otherwise be evicted due to queries on older data.
    # Mini segment files are kept in any case until their merge result also exists.
    # (The age is determined using the timestamp of the most recent event in the file)
    # Make sure to leave most of the free space as room for the system to
    # manage as a mix of recent and old files.
    # Note! Min age takes precedence over the fill percentage, so increasing this
    # implies the risk of overflowing the local file system!

    LOCAL_STORAGE_MIN_AGE_DAYS=0

### Start a new LogScale cluster based on another with buckets

### Note

This is a feature for advanced users, requiring multiple independent Kafka clusters and buckets with careful management of these in order to not mix up targets.

LogScale supports a disaster recovery method based on bucket storage. The objects stored in the target bucket of a cluster can form the basis for bootstrapping a fully independent cluster. The new cluster needs to have a bucket of its own to store objects in, and will consider the "source" bucket as read only. It relies on the source bucket being immutable from the point the disaster recovery process starts - don't point to the bucket of another running LogScale cluster.

### Note

All example configurations that follow use `S3_` as the prefix for AWS S3 storage. If you are using Google Cloud Platform (GCP) for bucket storage, specify `GCP_` instead of `S3_`.

Common use cases for disaster recovery using this method include:

> - Recover a cluster to a working state after cluster failure.

> - Start a staging cluster for testing, based on data from a production cluster. This requires cloning the production cluster's bucket before starting, since the method requires the source bucket to be immutable during the recovery process.

> - Moving a cluster from one set of nodes to a fresh set. This requires the source cluster to run in ephemeral disk storage mode.

In all cases these are the steps that need to happen:

1. A single node in the target cluster must be started with an empty data directory.

   - It must use a Kafka cluster where none of the LogScale topics exist, or those topics must be empty if you are manually managing topics. Use a fresh Kafka cluster or set a fresh prefix for the topic names.

   - It must not have access to a `global-data-snapshot.json`. The recovery procedure only happens when there is no other snapshot to start from.

   - It must have a target bucket for bucket storage that is empty.

   - It must have the configs `S3_RECOVER_FROM_*` set for all the bucket storage target parameters otherwise named `S3_STORAGE_*`` so that those refer to the bucket that it should start based on.

   - It may have the configuration variables `S3_RECOVER_FROM_REPLACE_REGION` and `S3_RECOVER_FROM_REPLACE_BUCKET` set to strings that are on the form `substring/substitution`. These enable you to make LogScale edit certain bucket settings in the global snapshot it loads from the source bucket, before it starts fetching other data. This way you can, for example, load a snapshot originally configured for region "us-west" in the source cluster, but have the target cluster read from a bucket copy in region "us-east".

   - The new node may specify a new number of ingest partitions using `INGEST_QUEUE_INITIAL_PARTITIONS` which will re-hash all datasources internally to the new number of partitions. This enables you to have different Kafka ingest partition counts in the source and target clusters.

   - The new node should have whatever other configuration is required for the cluster (as a single node cluster for the moment) as the source cluster's configuration cannot be loaded from the source bucket.

2. The single new node starts by fetching the latest global snapshot file from the `S3_RECOVER_FROM_BUCKET` and then rewrites that taking the following steps:

   - Drop all hosts from the cluster, make the new cluster consist of just this single host that will be vhost node ID 1.

   - Rewrite both storage and digest partition tables to match a cluster of just this node.

   - Rewrite all segment entities to forget previous Kafka offsets and replace owner and current hosts, as those are not relevant anymore.

   - Create the required topics in Kafka (if allowed by configurations).

   - Upload the resulting rewritten global snapshot to the new (so far empty) bucket storage target.

3. At this point you can access and inspect the new cluster using the single node if you want. Check the health of the cluster and if it is working correctly, then remove the `S3_RECOVER_FROM_*` settings from the node. You can continue to ingest data while the rest of the bucket migration takes place.

4. Extend the cluster to the number of nodes you desire just as if this was any other LogScale cluster. The new nodes should not use the special recovery configuration.

The LogScale nodes will start fetching segment files from the "old" bucket and upload them to the new target, which will over time make the cluster have an independent copy of the full contents of the old bucket. Once this replication process completes, the new cluster no longer depends on any objects in the source bucket it was bootstrapped from, nor from other buckets that were referred in the snapshot fetched from that bucket.

In order for the recovery procedure to be lossless, the target cluster needs to get a global snapshot from the source cluster that does not reference segment files missing from the bucket. To create such a snapshot, the source cluster needs to be shut down gracefully before initiating the recovery procedure. This is typically not the case for an actual disaster recovery scenario, in which some data loss may be unavoidable, but for the migrate or clone cases it is doable. If the source cluster did not shut down gracefully, then there may be reports of missing segment files in the new target cluster. These can be resolved using the Missing Segments.

### Decrypt Bucket Files Externally

LogScale provides a utility function to decrypt a file outside of LogScale that has been uploaded to the bucket by LogScale. This may come in handy as part of an audit process or as part of other processes that involve fetching files directly from the bucket. Usage:

shell

    java -cp humio.jar com.humio.main.DecryptAESBucketStorageFile <secret string> <encrypted file> <decrypted file>

Provide the path / filename of the input file to decrypt as argument. The last part of the file name is included in the decryption process, so file names must be preserved identical to those in the bucket for the path after the last `/`. The "secret string" argument is the value of `S3_STORAGE_ENCRYPTION_KEY` / `GCP_STORAGE_ENCRYPTION_KEY` / `AZURE_STORAGE_ENCRYPTION_KEY` that was in the LogScale config at the time the file was uploaded by LogScale.

### Important

When downloading the files on a macOS-based system, be aware of the following conditions:

- The filename when downloaded will include a period to separate the bucket and unique identifier for the file. This can be interpreted by macOS as a file extension. To prevent this, configure the Finder to use display full filenames. To do this, open Finder and and select Finder → Preferences → Advanced → Show all filename extensions before downloading the files using the AWS console.

- Files downloaded use a case-sensitive filename and may cause issues on an APFS volume that is not case sensitive.

### Re-ingest log data archived to bucket

To re-ingest log data that has been written to an a bucket through archiving can be achieved by using LogScale Collector and the native JSON parsing within LogScale.

This process has the following requirements:

- The files will need to be downloaded from the bucket to the machine running the LogScale Collector. The files cannot be accessed natively by the log collector.

- The ingested events will be ingested into the repository that is created for the purpose of receiving the data.

To re-ingest logs this way:

1. Create a repo in LogScale where the ingested data will be stored. See Creating a Repository or View.

2. Create an ingest token, and choose the JSON parser. See Assigning Parsers to Ingest Tokens.

3. Install the Falcon LogScale Collector to read from a file using the `.gz` extension as the file match. For example, using a configuration similar to this:

   yaml

       #dataDirectory is only required in the case of local configurations and must not be used for remote configurations files.
       dataDirectory: data
       sources:
       bucketdata:
         type: file
         # Glob patterns
         include:
         - /bucketdata/*.gz
         sink: my_humio_instance
         parser: json
         ...

   For more information, see Configuration Examples.

4. Copy the log file from the bucket into the configured directory (`/bucketdata`) in the above example.

The Log Collector reads the file that has been copied, sends it to LogScale, where the JSON event data will be parsed and recreated.

#### Enter search term