Instance Sizing Recommendations

Bare-Metal/VM/Kubernetes Workers

Assumptions:

  • 30 Day Retention on NVME (gen4)

  • 20% Overhead left on NVME (gen4)

  • 10x Compression

  • Supported Object Storage provider

  • Kafka 5x Compression - 24 Hour Storage

  • LogScale requires a kafka cluster most commonly provided by the Strimzi Kubernetes operator or Amazon MSK (required up to LogScale 1.107.0).

Kafka clusters are separate from LogScale clusters to avoid resource contention and allow independent management.

Important

Any deployed LogScale instance should not be sharing resources with any other services and it should be a dedicated node to avoid resource contention. LogScale assumes that all spare memory is available for use by LogScale.

X-Small - 1 TB/Day Ingestion
Software Instances vCPU Memory Storage Total Storage
LogScale 3 16 64 GB NVME 2 TB 6 TB
Kafka 3 4 8 GB SSD 500 GB 1.5 TB
ZooKeeper (up to LogScale 1.107) 3 4 8 GB SSD 50 GB 150 GB
Small - 3 TB/Day Ingestion
Software Instances vCPU Memory Storage Total Storage
LogScale 3 32 128 GB NVME 6 TB 18 TB
Kafka 3 4 8 GB SSD 1 TB 3 TB
ZooKeeper (up to LogScale 1.107) 3 4 8 GB SSD 50 GB 150 GB
Medium - 5 TB/Day Ingestion
Software Instances vCPU Memory Storage Total Storage
LogScale 6 32 128 GB NVME 6 TB 36 TB
Kafka 3 8 16 GB SSD 1 TB 3 TB
ZooKeeper (up to LogScale 1.107) 3 4 8 GB SSD 50 GB 150 GB
Large - 10 TB/Day Ingestion
Software Instances vCPU Memory Storage Total Storage
LogScale 12 32 128 GB NVME 6 TB 72 TB
Kafka 6 8 16 GB SSD 1 TB 6 TB
ZooKeeper (up to LogScale 1.107) 3 4 8 GB SSD 50 GB 150 GB
X-Large - 30 TB/Day Ingestion
Software Instances vCPU Memory Storage Total Storage
LogScale 30 32 128 GB NVME 7 TB 210 TB
Kafka 6 8 16 GB SSD 1.5 TB 13.5 TB
ZooKeeper (up to LogScale 1.107) 3 4 8GB SSD 50 GB 150 GB
AWS: EC2/EKS Workers

Assumptions:

  • Retention on NVME Varies due to fixed size, but is > 30 days

  • 20% Overhead left on NVME

  • 10x Compression

  • S3 Bucket storage used for longer retention

  • AWS Managed Kafka Service (MKS) for Kafka

  • LogScale requires a kafka cluster most commonly provided by the Strimzi Kubernetes operator or Amazon MSK

Kafka clusters are separate from LogScale clusters to avoid resource contention and allow independent management.

AWS EKS
Recommended AWS EKS

Figure 12. Recommended AWS EKS


AWS Reference Architecture
AWS Reference Architecture

Figure 13. AWS Reference Architecture


X-Small - 1 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 3 i3.2xlarge / 8 61 GB NVME 1.9 TB (direct attached) 5.7 TB
Kafka 3 kafka.m5.xlarge/ 4 16 GB EBS 500 GB 1.5 TB
ZooKeeper (up to LogScale 1.107) MSK MSK MSK MSK MSK
Small - 3 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 3 i3.4xlarge / 16 122 GB NVME 3.8 TB (direct attached) 11.4 TB
Kafka 3 kafka.m5.xlarge/ 4 16 GB EBS 500 GB 1.5 TB
ZooKeeper (up to LogScale 1.107) MSK MSK MSK MSK MSK
Medium - 5 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 6 i3.8xlarge / 32 244 GB NVME 7.6 TB (direct attached) 45.6 TB
Kafka 3 kafka.m5.2xlarge/ 8 16 GB EBS 1.5 TB 4.5 TB
ZooKeeper (up to LogScale 1.107) MSK MSK MSK MSK MSK
Large - 10 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 12 i3.8xlarge / 32 244 GB NVME 7.6 TB (direct attached) 91.2 TB
Kafka 6 kafka.m5.2xlarge/ 8 16 GB EBS 1.5 TB 9 TB
ZooKeeper (up to LogScale 1.107) MSK MSK MSK MSK MSK
X-Large - 30 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 30 i3.8xlarge / 32 244 GB NVME 7.6 TB (direct attached) 228 TB
Kafka 9 kafka.m5.2xlarge/ 8 16 GB EBS 2 TB 18 TB
ZooKeeper (up to LogScale 1.107) MSK MSK MSK MSK MSK
Google Cloud Platform (GCP)/Google Kubernetes Engine (GKE)

Assumptions:

  • 30 Day Retention NVME

  • 20% Overhead left on NVME

  • 10x Compression

  • GCS Bucket storage used for longer retention

  • LogScale does not provide a self-hosted Kubernetes solution for Kafka

Kafka clusters are separate from LogScale clusters to avoid resource contention and allow independent management.

X-Small - 1 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 3 n-standard-16 / 16 122 GB NVME 3 TB 9 TB
Kafka 3 n-standard-8 / 8 32 GB PD-SSD 500 GB 1.5 TB
ZooKeeper (up to LogScale 1.107) 3 Shared with Kafka Shared with Kafka PD-SSD 50 GB 150 GB
Small - 3 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 3 n2-highmem-16 / 16 128 GB NVME 6 TB 18 TB
Kafka 3 n-standard-8 / 8 32 GB PD-SSD 500 GB 1.5 TB
ZooKeeper (up to LogScale 1.107) 3 Shared with Kafka Shared with Kafka PD-SSD 50 GB 150 GB
Medium - 5 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 6 n-standard-32 / 32 128 GB NVME 6 TB (16x375GB) 36 TB
Kafka 6 n-standard-8 / 8 32 GB PD-SSD 1 TB 6 TB
ZooKeeper (up to LogScale 1.107) 3 Shared with Kafka Shared with Kafka PD-SSD 50 GB 150 GB
Large - 10 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 12 n-standard-32 / 32 128 GB NVME 6 TB (16x375GB) 72 TB
Kafka 6 n-standard-8 / 8 32 GB PD-SSD 1 TB 6 TB
ZooKeeper (up to LogScale 1.107) 3 Shared with Kafka Shared with Kafka PD-SSD 50 GB 150 GB
X-Large - 30 TB/Day Ingestion
Software Instances EC2 Instance Type/vCPU Memory Storage Total Storage
LogScale 30 n-standard-64 / 64 256 GB NVME 7.5 TB (16x375GB) 225 TB
Kafka 9 n-standard-8 / 8 32 GB PD-SSD 1.5 TB 13.5 TB
ZooKeeper (up to LogScale 1.107) 3 Shared with Kafka Shared with Kafka PD-SSD 50 GB 150 GB