Skip to content
LogScale Self-Hosted Deployment
Content was updated:Jul 7, 2025
This is an archived version of the documentation downloaded from LogScale Documentation Archives. Up to date documentation, and documentation for later versions, is available on LogScale Documentation

# Storage Architecture

The following sections describe the main concepts in the LogScale storage architecture. It is assumed you are already familiar with the LogScale Internal Architecture.

### Note

Primary and secondary storage are collectively referred to as local storage.

The three main types of storage of the LogScale storage architecture are summarized here:

- Primary storage (hot storage) - Data that is ingested into a repository is stored initially on primary local storage. This is the main drive, usually mounted by the container (if used) or the install directory. It can also be configured using the `DIRECTORY` configuration variable. Typically on a bare metal install it will be based on fast NVMe drives or other SSDs. If LogScale is running in a cloud-based deployment such as AWS, then instance stores using NVMe are recommended for maximum performance. However, in this case, as instance stores are considered ephemeral, bucket storage needs to be configured in order to persist data on a longer time frame.

  Read more about primary storage here.

- Secondary storage (warm storage) - To enable LogScale to have fast access to more data than the capacity of the primary storage, you can optionally configure secondary storage. This is not available by default. You can configure secondary storage using the variable `SECONDARY_DATA_DIRECTORY`. Secondary storage is usually SSDs or a spinning disk array. Whereas primary storage is based on faster (but more expensive) options such as NVMe-based drives or instance stores, secondary storage can be based on cheaper but higher capacity options such as SSDs and spinning disk arrays.

  When secondary storage is configured, segment files are moved from primary to secondary when the percentage threshold on primary is exceeded. This threshold is set using the variable `PRIMARY_STORAGE_PERCENTAGE`. Secondary storage can also be configured with `SECONDARY_STORAGE_MAX_FILL_PERCENTAGE`, which sets the maximum capacity of the storage to use.

  Note that if `PRIMARY_STORAGE_MAX_FILL_PERCENTAGE` and `SECONDARY_STORAGE_MAX_FILL_PERCENTAGE` are exceeded, the processing of logs stops until more disk space is made available.

  Read more about secondary storage here.

- Bucket storage (cold storage) - As storage capacity of primary and secondary storage is of a finite size often dictated by price-performance characteristics, capacity can be exceeded for high ingest volumes.

  Further, data is typically controlled by data retention policies, so data might not be stored on local storage for the longer term, and this is where bucket storage comes into play. Bucket storage is usually a much cheaper option than using local storage such as NVMe and SSD, as it typically uses cloud-based options such as AWS S3. Also, the capacity of bucket storage is for practical purposes unlimited (although costs and data rention policies may put an upper limit of what is actually retained).

  When data on local storage (primary plus secondary) exceeds the capacity threshold set in the variable `LOCAL_STORAGE_PERCENTAGE` (85% by default), data is copied to bucket storage for persistence. So, for example, if the local storage is more than 85% full, data is now moved to bucket storage, based on the @timestamp of the data (older data is moved first).

  When the data stored on primary and secondary storage exceeds the set thresholds, then it is automatically deleted - this is why it is important to have a bucket storage strategy, so that all data is persisted for the duration required by the use case.

  Note that if older data (contained in segment files) is queried, that is not currently on local storage, then the relevant segment files containing that data are copied from bucket storage to local storage.

  While replication can obviate short-term node failure, bucket storage can provide longer term backup capabilities in the event that a node containing local storage fails.

  Note that bucket storage is optional, but highly recommended. It is also worth noting that data copied to bucket storage is always encrypted by LogScale.

  Read more about bucket storage here.

In addition to the main storage types, there are some more concepts that will frequently arise as you configure your storage architecture. These are summarized here, with links to more information in the guidance panel at the bottom of this page.

- Intelligent tiering - You learned in the previous sections on the storage tiers that LogScale moves data between the different storage tiers, taking into account the configuration variables mentioned such as `LOCAL_STORAGE_PERCENTAGE`. This automatic copying of data between the different tiers is known as intelligent tiering. Intelligent tiering has been the default in LogScale since version 1.85.

- Segment files - LogScale reads events from the Kafka ingest queue and processes them into segment files. LogScale's model for avoiding data loss is that newly written segments are considered "untrusted" and can be replayed from the ingest queue if needed. Once a segment is considered "trusted", the data can be removed from Kafka. From that point on, LogScale ensures that the generated segment is not lost. The point at which a segment is considered "trusted" depends on the configuration of LogScale.

  LogScale can optionally be configured to use bucket storage. If bucket storage is enabled, the bucket is trusted unconditionally. Segment files that make it into bucket storage are expected to stay there. In some modes, LogScale will be unable to tolerate the bucket storage system becoming unavailable.

  LogScale offers a small number of configuration options enabling customers to adjust how it handles replication of segment files. These options can have a large impact on the cost of a cluster, as movement of these files often accounts for a large portion of the total transferred bytes in a cluster.

- Segment replication factor - This setting controls how many complete copies of a segment file LogScale should store locally.

  LogScale attempts to always ensure the replication factor is hit, moving segment file copies around if nodes go offline or leave the cluster. The exception is when a segment is stored in bucket storage. In that case, LogScale may choose to leave a segment with fewer replicas than configured, since segments in bucket storage are treated as always available.

  The default setting for `DEFAULT_SEGMENT_REPLICATION_FACTOR` is 1, but every production environment is likely to customize this. A more reasonable factor for production environments is 3, unless the cluster is ephemeral.

- Ephemeral storage - When using LogScale in a cloud environment such as AWS, as opposed to installation directly on physical (bare metal) machines, it is possible to use NVMe-based instance stores as primary storage, and this is the recommended approach for best performance. It's important to note however, that because of how these instance stores are provisioned in cloud environments, that data on these stores (but not data in bucket storage or replicated segment files) is lost in the event of one of the following:

  - The instance stops or terminates

  - The underlying host fails

  For this reason, these instance stores are referred to as ephemeral storage, as the data can potentially be lost in certain circumstances. It's therefore important to have a bucket storage strategy in place.

  An important setting is `USING_EPHEMERAL_DISKS`, which controls when segment files are considered "trusted". When this setting is disabled (`false`, the default), a segment is "trusted" once LogScale reaches the configured segment replication factor, or the segment enters bucket storage. When this setting is enabled (`true`), a segment is "trusted" once it is in bucket storage. It is not possible to enable this setting if bucket storage is not configured for the cluster.

  If LogScale is initially set up without bucket storage, it is important to set `USING_EPHEMERAL_DISKS` to `false`, and later when bucket storage is implemented and verified, set `USING_EPHEMERAL_DISKS` to `true`, to denote that primary storage is ephemeral, and should not be relied upon for persistence.

  If data on ephemeral disks is lost for some reason, then LogScale would need to copy the relevant segments from bucket storage, or replicated segment files, to primary storage, incurring a performance penalty.

  While it seems that this performance penalty might be something to be avoided, there are some factors that should be considered. First, you can increase the segment replication factor (`DEFAULT_SEGMENT_REPLICATION_FACTOR`), which is 1 by default. This means if an instance store fails, data can be refreshed more quickly from the replicated segments, rather than bucket storage. You would need to allow for the increased cost of local storage for the replicated segments, and more CPU and memory to help create them. Another factor to consider is that LogScale is optimized for maximum price-performance. In normal use NVMe instance stores provide excellent performance, and recovery from bucket storage is low cost. As NVMe are unlikely to lose data frequently, using them provides you with the best price-performance solution for most use cases.

- Copying segments - The `S3_STORAGE_PREFERRED_COPY_SOURCE` and `AZURE_STORAGE_PREFERRED_COPY_SOURCE` options control how LogScale fetches copies of segments from nodes for replication. When this setting is disabled (the default), LogScale attempts to fetch segment copies from other local nodes if possible. Segments will only be fetched from bucket storage (AWS S3 or Azure) if no local copies are available.

  When this setting is enabled, LogScale will always fetch from bucket storage.

  This does not affect fetching from bucket storage for queries. Such fetching always happens from bucket storage, because if segments need to be fetched for queries at all, it must be because they aren't present on any cluster nodes, as otherwise, the query would be sent to the node that had the segments.

  The reason this setting exists is because some environments pay more for bucket fetching versus local fetching, and in other environments, it is the opposite.

- Data retention - Data retention policies relate to how long you keep data. For example, you may have legal requirements to retain personal data for a certain length of time before deleting it. There can also be legal requirements not to hold personal data beyond a certain period. You might also want to reduce storage consumption by deleting data no longer required. Data retention policies control how data is handled using the following parameters:

  - Ingest limit in GB (uncompressed)

  - Storage size limit in GB (compressed)

  - Time limit in days

  When data retention thresholds are exceeded, the oldest data is deleted first.

  ### Note

  Data retention policies are set on a per-repository basis.

  Note though that if data is deleted from primary or secondary storage through data retention policies, its corresponding data in bucket storage is also deleted.

  Read more about data retention here.

- Compression - LogScale uses data compression extensively. This has a number of benefits such as data consumes less storage space, data transfers require less network bandwidth, and cost is reduced. When raw data is initially compressed into mini-segments, LZ4 is used, as it is extremely fast, while offering good compression ratios. Full segments are compressed using Zstd, which provides greater compression ratios while still being nearly as fast as LZ4. Your effective overall compression ratio is shown in your cluster statistics.

  Understanding compression is important, because you need to take it into account when sizing your primary and optionally secondary storage requirements.

  To calculate your storage requirements you need to take into account the following: compression ratio (you can assume a value of 10 initially, and later refine this as required), replication (the number of copies of the stored data across all nodes in the LS cluster), the ingest rate, and data retention duration. For example, if you had an ingest per day of 50TB, a data retention of 30 days, and the default replication factor of 3, then your requirement would be 50*30*3/10 = 450TB. For practical purposes this would be split across the primary, secondary, and bucket storage of the cluster.

  Read more about storage sizing calculations.

- Monitoring - It is important to monitor your storage usage to avoid running out of disk space, and having LogScale stop processing inbound data.

  To monitor the data storage:

  - Data storage across individual nodes can be monitoring using the Cluster Nodes page

  - To monitor the amount of data stored across the cluster and the effects of compression, see Cluster statistics

  - For more detailed and historic information, use the humio/insights dashboard.

For further information see the following:

> Primary Storage

> This section describes primary storage. This is the key storage to consider for ensuring fast and real-time queries.

> Secondary Storage

> Active data is stored on local disks within each node of the Falcon LogScale cluster. Primary disks should be high performance SSD. For additional local storage, secondary storage, for example, a lower performance SSD can be used. Falcon LogScale will automatically move segment files to secondary storage once the primary disk reaches a configured limit.

> Bucket Storage

> To store larger volumes of data, bucket storage can be used. Similar to secondary storage, Falcon LogScale will move segments to solutions such as Amazon Bucket Storage or Google Bucket. Bucket storage also allows for deployment of nodes, expansion of an existing cluster, and to maintain back-ups in case a node or a cluster crashes.

> S3 Archiving

> Ingested log data can be archived to Amazon S3. Archiving stores a copy of the ingested data logs, but the the archived data is not searchable by Falcon LogScale as it is when stored on bucket storage. Archived storage can optionally be re-ingested or read by other software.

> Data Retention

> To avoid servers reaching their maximum storage capabilities, Falcon LogScale can be configured to expire (delete) data when approaching a given threshold, such as the compressed file sizes, uncompressed file sizes, or the age of data.

> Storage Capacity

> Provides some background information on storage capacity planning.

> Instance Sizing

> Provides information on sizing AWS instances, with suggestions for sizing primary storage.

#### Enter search term