Capacity Planning

Capacity planning for Falcon LogScale relies on understanding the different systems within LogScale and how they work together to support the overall functionality.

The aim of this guide is to provide an overview of the different areas of the system which impact and affect capacity planning, and guidance on how to measure and change the corresponding capacity value. Answering specific questions will be dependent on multiple factors.

Typically when considering capacity planning there are five main dimensions to consider:

  1. Ingest performance — how quickly LogScale can accept and parse incoming data. This is explained in more detail in Ingestion Capacity.

  2. Digest performance — how quickly LogScale can organise the data in preparation for storage. This is explained in more detail in Digest Capacity.

  3. Storage performance — how quickly LogScale can store the data in preparation for querying. This is explained in more detail in Storage Capacity.

  4. Query performance — how fast queries are, either for recent data or old data. This is explained in more detail in Query Capacity.

  5. UI Performance — how responsive the web interface is to users. This is explained in more detail in UI Performance.

The performance of LogScale, and therefore the impact of the capacity planning, will depend on a variety of different factors, including:

  • Volume and rate of ingested data

  • Volume and rate of queries

  • Duration and timespan of queries

  • Number of physical nodes

  • Number of CPU cores and RAM in each physical node

  • Number of nodes in the LogScale cluster

  • Number of nodes within the Kafka cluster

  • Whether the LogScale deployment uses dedicated node roles

  • Use of tags and datasources when ingesting (and therefore searching) data

  • Use of block storage (such as S3)

  • Quantity and speed of local storage

  • Replication settings

  • Number of repositories being used for storing incoming data

When monitoring and/or changing a value, changes to one area may affect another. For example, increasing the local storage availability may change the behavior of digest and bucket storage, because more data can be stored locally, and may also increase query performance by keeping more data local.

Before proceeding with any changes or planning, ensure the architecture of LogScale, the processes, and how the different components of the system work together are understood by reading:

When capacity planning, it's best to plan to achieve the maximum performance in that dimension. For example, aim to address the highest required rate for ingest, query, or digest performance and plan against the maximum value, rather than planning against the medium of average values. This is because using the average rates or values may degrade performance in other parts of the system. By planning against the maximum possible, it allows for some spare capacity during brief periods of higher load.