Instance Sizing Recommendations
The following sections detail sizing guidelines for the following platforms:
- Bare-Metal/VM/Kubernetes Workers
- AWS: EC2/EKS Workers
- Google Cloud Platform (GCP)/Google Kubernetes Engine (GKE)
Important
It should be noted that these guidelines are for budgetary purposes only, and actual sizing requirements will depend on your specific use case.
Bare-Metal/VM/Kubernetes Workers
Assumptions:
30 Day Retention on NVME (gen4)
20% Overhead left on NVME (gen4)
10x Compression
Supported Object Storage provider
Kafka 5x Compression - 24 Hour Storage
LogScale requires a kafka cluster most commonly provided by the Strimzi Kubernetes operator or Amazon MSK (required up to LogScale 1.107.0).
Kafka clusters are separate from LogScale clusters to avoid resource contention and allow independent management.
Important
Any deployed LogScale instance should not be sharing resources with any other services and it should be a dedicated node to avoid resource contention. LogScale assumes that all spare memory is available for use by LogScale.
| Software | Instances | vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 3 | 16 | 64 GB | NVME 2 TB | 6 TB |
| Kafka | 3 | 4 | 8 GB | SSD 500 GB | 1.5 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8 GB | SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16 GB | SSD 64 GB | 150 GB |
AWS: EC2/EKS Workers
Assumptions:
Retention on NVME Varies due to fixed size, but is > 30 days
20% Overhead left on NVME
10x Compression
S3 Bucket storage used for longer retention
AWS Managed Kafka Service (MKS) for Kafka
LogScale requires a kafka cluster most commonly provided by the Strimzi Kubernetes operator or Amazon MSK
Kafka clusters are separate from LogScale clusters to avoid resource contention and allow independent management.
AWS EKS
 |
Figure 3. Recommended AWS EKS
AWS Reference Architecture
 |
Figure 4. AWS Reference Architecture
X-Small - 1 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 3 | i3.2xlarge / 8 | 61 GB | NVME 1.9 TB (direct attached) | 5.7 TB |
| Kafka | 3 | kafka.m5.xlarge/ 4 | 16 GB | EBS 500 GB | 1.5 TB |
| ZooKeeper (up to LogScale 1.107) | Strimzi | Strimzi | Strimzi | Strimzi | Strimzi |
| KRaft controller (after LogScale 1.107) | Strimzi | Strimzi | Strimzi | Strimzi | Strimzi |
Small - 3 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 3 | i3.4xlarge / 16 | 122 GB | NVME 3.8 TB (direct attached) | 11.4 TB |
| Kafka | 3 | kafka.m5.xlarge/ 4 | 16 GB | EBS 500 GB | 1.5 TB |
| ZooKeeper (up to LogScale 1.107) | Strimzi | Strimzi | Strimzi | Strimzi | Strimzi |
| KRaft controller (after LogScale 1.107) | Strimzi | Strimzi | Strimzi | Strimzi | Strimzi |
Medium - 5 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 6 | i3.8xlarge / 32 | 244 GB | NVME 7.6 TB (direct attached) | 45.6 TB |
| Kafka | 3 | kafka.m5.2xlarge/ 8 | 16 GB | EBS 1.5 TB | 4.5 TB |
| ZooKeeper (up to LogScale 1.107) | Strimzi | Strimzi | Strimzi | Strimzi | Strimzi |
| KRaft controller (after LogScale 1.107) | Strimzi | Strimzi | Strimzi | Strimzi | Strimzi |
Large - 10 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 12 | i3.8xlarge / 32 | 244 GB | NVME 7.6 TB (direct attached) | 91.2 TB |
| Kafka | 6 | kafka.m5.2xlarge/ 8 | 16 GB | EBS 1.5 TB | 9 TB |
| ZooKeeper (up to LogScale 1.107) | Strimzi | Strimzi | Strimzi | Strimzi | Strimzi |
| KRaft controller (after LogScale 1.107) | Strimzi | Strimzi | Strimzi | Strimzi | Strimzi |
X-Large - 30 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 30 | i3.8xlarge / 32 | 244 GB | NVME 7.6 TB (direct attached) | 228 TB |
| Kafka | 9 | kafka.m5.2xlarge/ 8 | 16 GB | EBS 2 TB | 18 TB |
| ZooKeeper (up to LogScale 1.107) | Strimzi | Strimzi | Strimzi | Strimzi | Strimzi |
Google Cloud Platform (GCP)/Google Kubernetes Engine (GKE)
Assumptions:
30 Day Retention NVME
20% Overhead left on NVME
10x Compression
GCS Bucket storage used for longer retention
LogScale does not provide a self-hosted Kubernetes solution for Kafka
Kafka clusters are separate from LogScale clusters to avoid resource contention and allow independent management.
X-Small - 1 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 3 | n-standard-16 / 16 | 122 GB | NVME 3 TB | 9 TB |
| Kafka | 3 | n-standard-8 / 8 | 32 GB | PD-SSD 500 GB | 1.5 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |
Small - 3 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 3 | n2-highmem-16 / 16 | 128 GB | NVME 6 TB | 18 TB |
| Kafka | 3 | n-standard-8 / 8 | 32 GB | PD-SSD 500 GB | 1.5 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |
Medium - 5 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 6 | n-standard-32 / 32 | 128 GB | NVME 6 TB (16x375GB) | 36 TB |
| Kafka | 6 | n-standard-8 / 8 | 32 GB | PD-SSD 1 TB | 6 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |
Large - 10 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 12 | n-standard-32 / 32 | 128 GB | NVME 6 TB (16x375GB) | 72 TB |
| Kafka | 6 | n-standard-8 / 8 | 32 GB | PD-SSD 1 TB | 6 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |
X-Large - 30 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 30 | n-standard-64 / 64 | 256 GB | NVME 7.5 TB (16x375GB) | 225 TB |
| Kafka | 9 | n-standard-8 / 8 | 32 GB | PD-SSD 1.5 TB | 13.5 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |