FAQ: Using LOCAL_STORAGE_PERCENTAGE Disk Fills Past Configured Limit

The LOCAL_STORAGE_PERCENTAGE configuration setting limits the disk space used by LogScale by monitoring the disk space on the configure file system and deleting segment files once the configured threshold has been reached. Occasionally the disk space used will exceed the configured setting.

Deletion of segment files occurs as follows:

  • Once the disk space passed the LOCAL_STORAGE_PERCENTAGE setting has been passed, older segment files are marked to be deleted

  • Segment files are only able to be deleted when they are not being used within an actively executing query

  • Two separate processes, which run every 30 seconds, and every 5 minutes, perform the deletion of free and marked files

Disk space may therefore exceed the configured setting while data is still being ingested due to:

  • Segment files are still in use

  • Data files marked for deletion but the deletion process not being triggered

The increase in the use of disk space will only be temporary, but care should be taken to monitor the combination of disk space usage, segment expiry, and data ingestion rates.

If the disk space usage is causing issues, you may want to:

  • Increase the disk space available to LogScale

  • Reduce the LOCAL_STORAGE_PERCENTAGE to provide more of a buffer before the storage space is used