Instance Sizing
Assumptions:
30 Day Retention NVME
20% Overhead left on NVME
10x Compression
GCS Bucket storage used for longer retention
LogScale does not provide a self-hosted Kubernetes solution for Kafka
Kafka clusters are separate from LogScale clusters to avoid resource contention and allow independent management.
X-Small - 1 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 3 | n-standard-16 / 16 | 122 GB | NVME 3 TB | 9 TB |
| Kafka | 3 | n-standard-8 / 8 | 32 GB | PD-SSD 500 GB | 1.5 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |
Small - 3 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 3 | n2-highmem-16 / 16 | 128 GB | NVME 6 TB | 18 TB |
| Kafka | 3 | n-standard-8 / 8 | 32 GB | PD-SSD 500 GB | 1.5 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |
Medium - 5 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 6 | n-standard-32 / 32 | 128 GB | NVME 6 TB (16x375GB) | 36 TB |
| Kafka | 6 | n-standard-8 / 8 | 32 GB | PD-SSD 1 TB | 6 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |
Large - 10 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 12 | n-standard-32 / 32 | 128 GB | NVME 6 TB (16x375GB) | 72 TB |
| Kafka | 6 | n-standard-8 / 8 | 32 GB | PD-SSD 1 TB | 6 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |
X-Large - 30 TB/Day Ingestion
| Software | Instances | EC2 Instance Type/vCPU | Memory | Storage | Total Storage |
|---|---|---|---|---|---|
| LogScale | 30 | n-standard-64 / 64 | 256 GB | NVME 7.5 TB (16x375GB) | 225 TB |
| Kafka | 9 | n-standard-8 / 8 | 32 GB | PD-SSD 1.5 TB | 13.5 TB |
| ZooKeeper (up to LogScale 1.107) | 3 | 4 | 8GB | PD-SSD 50 GB | 150 GB |
| KRaft controller (after LogScale 1.107) | 3 | 4 | 16GB | PD-SSD 64 GB | 150 GB |