Instance Sizing

With LogScale, the limiting factor is typically query speed, not ingest capacity. Query speed depends upon the number of CPUs, available RAM, and disk speed. The following numbers are all general recommendations; your exact circumstances may be different.

A running LogScale instance will use approximately 10-20% of available RAM — the rest of available memory is used for OS-level file system cache. For example, If you're, for example, using an AWS i3.8xlarge (244 GB RAM, 32 vCPUs, 4 x 1,900 NVMe SSD), then you would typically see approximately 32 GB used for running LogScale, and the rest is available for file system cache. i3.8xlarge has fast NVMe disks, so data can be read fast enough from disk to keep the CPUs busy. In the case of slower disks, available memory is important for the file system cache.

Data is typically compressed by a factor in the range 10-20x, depending on the nature of the data. Short log lines (HTTP access logs, syslog) compress better than long JSON logs (such as those coming from Metricbeat).

For data available on fast NVMe disks or as compressed data in the OS-level cache, LogScale generally provides query speed at 0.5 GB/s/vCPU, or 0.5 GB/s/hyperthread. So, on a i3.8xlarge instance with 32 vCPUs, you observe ~16GB/s queries. This is the search speed when all data needs to be scanned. In many cases LogScale's hash indexes improve the search speed significantly.

In cloud environments, like Google and Amazon, CrowdStrike recommends using fast ephemeral NVMe disks for good search performance. CrowdStrike does not recommend EBS storage as it is slow and expensive (IOPS). Set up LogScale with Bucket storage to keep data durable. Bucket storage stores data in Amazon S3 or Google Cloud Storage. ZooKeeper and Kafka should not be run on ephemeral disks. It is important to keep that data durable. You can read more about Storage in Storage Architecture.

Searches requiring beyond what fits in fast disks or the OS-level file system caches can be significantly slower, as they depend on disk I/O performance. If you have sufficiently fast NVMe drives, or similar, that can deliver the compressed data as fast as the CPU cores can decompress them, then there is little penalty for doing searches that extend beyond the size of the page cache. LogScale was built to run on local SSDs, so it is not optimized to run on high-latency EBS storage or slow spinning disks. But it will work, especially if most searches are "live" or within the page cache. LogScale reads files using "read ahead" instructions to the Linux kernel, which enables file systems on spinning disks to read continuous ranges of blocks rather than random blocks.

There are some basics to consider when sizing an instance. First, one vCPU per hyperthread can ingest 250 GB/day. Search speed is 0.5 GB per vCPU per hyperthread — for data on fast disks or in RAM. Note, that a vCPU can perform only one of the above at any point in time. Search speed is thus influenced by time spent on ingest.

Another factor to consider is the compression ratio multiplied by the amount of RAM is how much data can be kept in memory, using OS-level file system cache. However, fast SSDs can achieve as good search speeds as when data is in RAM. For better performance, the disk subsystem should be able to read data at a minimum of 150 MB/s/core when not cached. For a very general rule of thumb, one i3.8xlarge per TB/day data is recommended. This depends on the scenario: how many users are using the system and how much data is being searched and so forth.

For example, assume that data compresses 10 times. Test your setup to be sure precisely; better compression means better performance. You want enough hyper threads/vCPUs, each giving you 0.5 GB/s search. You need this to be able to search 24 hours of data in less than 10 seconds. You will need disk space to hold your compressed data. Never fill your disk more than 80%.

Based on the above, your machine has 64 GB of RAM, 16 hyper threads (8 cores), and 256GB NVMe storage. In this case that means that 10 seconds worth of query time will run through 80 GB of data. So this machine fits an 80 GB/day ingest, with 1 months of data available for fast querying. You can store 2 TB of data before your disk is 80% full.

Running many live queries/dashboards is less of an issue with LogScale than with similar products, because these are kept in memory as an in-memory view. The time spent on updating them is part of the ingest flow and thus having many live queries increases the CPU usage for the ingest process. When initializing such queries, it does need to run a historic query to fill in past data, and that can take some time in particular if it extends beyond what fits within the compressed data in memory horizon.

FIO is a great tool for testing the IO performance of your system. You may have to install it on your server, though. Run fio either with all options on the command-line or through a jobfile.

$ fio --filename=/data/fio-test.tmp --filesize=1Gi  \
    --bs=256K -rw=read --time_based --runtime=5s  \
    --name=read_bandwidth_test --numjobs=8 --thread --direct=1

Here are a sample contents for a jobfile that somewhat mimics how LogScale reads from the file system. Make sure to modify the file path to where your humio-data would be located, once installed.

[global]
thread
rw=read
bs=256Ki
directory=/data/fio-tmp-dir
direct=1

[read8]
stonewall
size=1Gi
numjobs=8

Once that's done, run the fio command from the command-line like so:

$ fio --bandwidth-log ./humio-read-test.fio
$ rm /data/fio-tmp-dir/read8.?.?

The second line here will remove the temporary files generated by fio. When executing the first line, look for the lines at the bottom of the very detailed report similar to this:

Run status group 0 (all jobs):
   READ: bw=3095MiB/s (3245MB/s), 387MiB/s-431MiB/s
   (406MB/s-452MB/s), io=8192MiB (8590MB), run=2375-2647msec

This example is an NVME providing ~3 GB per second read performance. This allows for searching up to (9*3) GB per second of uncompressed data if the system has sufficient number of CPU cores according as suggested earlier. This NVME is well matched to a CPU with 32 hyper-threads (16 hardware cores.)

Each reference architecture has its own suggested sizing requirements. See the following for more details: