Operational monitoring for self-hosted LogScale.

This documentation describes the official package available for improved monitoring of LogScale self-hosted clusters. The package is called crowdstrike/operations.

Attention

This package for monitoring self-hosted LogScale is currently Beta only.

Two independent observability layers sit side by side:

  • A LogScale package including dashboards, alerts, actions, and scheduled searches installed inside the cluster using humio-operator, or the general package installation system.

  • An external Prometheus/Grafana stack that scrapes the cluster's metrics endpoint.

The two packages are complementary - the in-cluster package reasons over structured logs and activity data; Prometheus reasons over time-series metrics and can alert even when LogScale itself is degraded.

Architecture

The in-cluster package is intentionally broader than LogScale's built-in humio/insights and humio/activity content: it targets self-hosted failure domains (Kafka ingest lag, digest partition stability, bucket-storage durability, JVM/OOM precursors, tag-cardinality heap pressure) that the shipped views do not cover.

The Prometheus layer provides metric-based alerting that remains available when the LogScale query path itself is impaired.

%%{init: {'theme': 'default', 'themeVariables': {'fontSize': '12px'}}}%% graph TD subgraph ls["LogScale Cluster"] metrics["Metrics endpoint\n(PROMETHEUS_METRICS_PORT)"] end op["humio-operator"] pkg["LogScale Package\n(dashboards / alerts / actions /\nscheduled searches)"] prom["Prometheus"] graf["Grafana"] rules["Prometheus rules"] am["Alertmanager"] notif(["Notifications\n(email / Slack / PagerDuty / webhook)"]) op -->|installs into| pkg pkg -->|runs inside| ls prom -->|scrapes| metrics prom -->|datasource| graf prom -->|evaluates| rules rules -->|fires| am am -->|routes| notif pkg -->|actions route| notif
%%{init: {'theme': 'default', 'themeVariables': {'fontSize': '12px'}}}%% graph TD subgraph ls["LogScale Cluster"] metrics["Metrics endpoint\n(PROMETHEUS_METRICS_PORT)"] end op["humio-operator"] pkg["LogScale Package\n(dashboards / alerts / actions /\nscheduled searches)"] prom["Prometheus"] graf["Grafana"] rules["Prometheus rules"] am["Alertmanager"] notif(["Notifications\n(email / Slack / PagerDuty / webhook)"]) op -->|installs into| pkg pkg -->|runs inside| ls prom -->|scrapes| metrics prom -->|datasource| graf prom -->|evaluates| rules rules -->|fires| am am -->|routes| notif pkg -->|actions route| notif

LogScale Package

Content installed inside the cluster by humio-operator: dashboards, alerts, actions, and scheduled searches. It processes structured logs ( humio-activity ) and the internal humio log.

Note

Run books are located in the docs directory in the package source.

LogScale Dashboards

Each dashboard targets a self-hosted failure domain. All per-node series key on #vhost so the same dashboard works on Kubernetes and bare metal.

Dashboard Purpose Runbook
Alert Status Overview Alert-system health from structured activity logs: firing frequency, alert query cost, errors and silent alerts, and action-delivery failures. Any problems > 0 needs investigation. alert-execution-health
Cluster Errors & Health Error observability reads top-down: a health-at-a-glance overview (node presence and fault-tolerance headroom, ingest, digest, Kafka, storage, queries), then detail sections for node stability, application errors (ERROR/WARN volume and the exception classes behind them), API request errors (5xx/4xx), and thread-dump exploration for potentially stuck threads. Start at the health tiles. node-health-detection
Cluster Sizing & Growth Capacity planning: ingest load and tail latency per node, local-disk headroom against the eviction watermark, CPU utilisation, three memory signals (JVM heap, system RSS, page cache), node count over time, and a per-resource scaling verdict. Answers "do I need more nodes / bigger disks / rebalancing?" - not "am I about to crash". disk-fill-prediction, jvm-heap-pressure, high-cpu
Cluster Upgrade & Regression Detection Post-upgrade sanity: deploy a new version and watch for 30 minutes. Covers error rate, HTTP 5xx/4xx, node restarts, Kafka/ingest partition events, digest deviation, query starvation, segment download loop, bucket storage, and OOM. version-mismatch
Errors โ€” Selected & Grouped Cluster-wide error and warning observability: groups errors by class, by alert / action category and by repository, plus query function exceptions, query restart reasons, per-class trends and version drift. A rising error rate is the primary signal; a rising warning rate ahead of it is the leading indicator. N/A
External Feeds โ€” SQS / FDR FDR (Falcon Data Replicator) feed health: feed status, SQS message throughput and delay, a per-message trace (time on the queue vs consumer vs ingest), and failure signals such as the same SQS message being ingested more than once. feed-pressure-high
Kafka Consumer Lag & Broker Health Kafka consumer lag, broker health, the LogScale-to-Kafka produce path, the digest-consumer path back from Kafka, and the global-events coordination topic. The consumer-group lag section shows, for each Kafka consumer group with committed offsets on the ingest topic, how far that group is behind the latest offset. kafka-consumer-lag
Digest โ€” Overview Digest-side health: partition ownership and reassignment stability, digest pipeline latency, writer-queue saturation and writer-thread health, segment merge, the ingest-feed handoff to digest, and the ingest reader's Kafka lag. Central question - are writers STUCK (blocked, no progress) or just OVERLOADED (busy, still progressing)? ingest-imbalance, ingest-writer-queue-full
Ingest - Overview Ingest-side health: overall and per-partition event latency, ingest-queue wait, feed pressure and governor throttling, HTTP ingest API health, the parser stage, per-feed data flow, and feed errors including SQS messages processed more than once. high-ingest-latency
Cluster JVM & Node Stability JVM heap, GC, off-heap memory, and node availability. Surfaces the usual precursors of an out-of-memory kill (heap climbing, GC time rising, hiccup pauses, off-heap and resident-memory growth), plus node availability signals (missing nodes, failed inter-node HTTP checks, shutdowns). jvm-heap-pressure
Kubernetes Operator โ€” Reconcile Health & CRD Lifecycle Health of the humio-operator, which manages HumioCluster, HumioPackage and the other Humio custom resources: whether the operator is logging at all, reconcile rate per controller, errors and stuck resources, HumioCluster state changes, package install failures, node-pool scaling and operator version. operator-not-reconciling
Query Performance & Starvation Query cost attribution, starvation detection, search-queue health, search latency and the search-success SLO. Central question - is the search tier STARVED (no forward progress: CPU exhausted or segments unavailable from bucket storage) or merely EXPENSIVE (progressing, a few queries burning the budget)? Also detects the watchdog anomaly: queries cancelled for starvation while they were still progressing and CPU, bucket wait and heap were all healthy. query-cost-noisy-neighbor, query-cancellation-low-cpu
Retention & Data Lifecycle Health Is retention running and deleting old segments, is local disk below the eviction watermark, are segments merging, and are uploads keeping up. Local disk is a cache: LogScale evicts local segments at LOCAL_STORAGE_PERCENTAGE (default 85%) to stay below PRIMARY_STORAGE_MAX_FILL_PERCENTAGE (default 95%), and only evicts segments already stored in bucket storage. retention-not-running, disk-fill-prediction, no-bucket-uploads
Scheduled Search & Workflow Health Scheduled-search execution health from humio-activity, and alert job-loop health from the filter, aggregate and legacy alert jobs' heartbeats. These are fire-and-forget background jobs; absence is the key signal - a gap in "job run finished" heartbeats means the alert poll loop is stuck. Action delivery is on Alert Status Overview. alerts-execution-health
Segment Prefetch & Download Health Detects segment download loops (prefetch storm): nodes repeatedly prefetch and then delete the same segments because a segment's recorded owners differ from the owners the cluster would assign now, so the extra copy is deleted as over-replicated. Healthy: downloads โ‰ซ deletes. Unhealthy: downloads โ‰ˆ deletes. segment-download-loop
Storage Health Object-storage (S3/GCS/Azure Blob) health for the bucket-storage tier: upload count and latency, pending-upload backlog (including the part with only one known copy in the cluster), query fetch timeouts, and erroneous global-snapshot files (epoch newer than the cluster's current Kafka epoch). No errors is the expected healthy state. bucket-issues, no-bucket-uploads, erroneous-files
Tag Cardinality & Datasource Health Tag cardinality, where a datasource = one unique combination of tag field values. Too many datasources can exhaust ingest-node heap. Shows active datasources per node, tag cardinality per field and high-cardinality warnings per repository. Requires a field configured as a tag in Settings โ†’ Tag Grouping. high-cardinality-tag-oom, datasource-overflow

LogScale Alerts

Log-driven alerts. Every alert routes to the same three actions - Email, Slack, and PagerDuty - so notification fan-out is uniform. Each alert's description names its companion dashboard and runbook.

Alert Purpose Dashboard Runbook
Alerts not polling Alert polling has stopped - alerts will not fire. Alert Status Overview alert-execution-health
Bucket storage timeout errors Queries returning partial results due to bucket-storage fetch timeouts. Storage Health bucket-issues
Bucket storage upload pending over 90 minutes Bucket-storage uploads pending > 90 minutes. Storage Health bucket-issues
Connection timeout errors Multiple connection timeouts indicating inter-node or external connectivity issues. Cluster Errors & Health http-unreachable
CPU fully saturated A node is sustaining 99%+ CPU utilisation - no spare capacity, so ingest, queries and background jobs will all degrade. Cluster JVM & Node Stability high-cpu
Datasource Overflow A repository has hit its max-datasources limit and new events are routed to the shared overflow bucket. Overflow data does not carry the original repository tags, so queries that filter on those tags silently miss events. Usually caused by one high-cardinality tag field. Tag Cardinality & Datasource Health datasource-overflow
Dead node reported in cluster A node reported dead for 90+ minutes. Cluster Errors & Health dead-node
Erroneous snapshot files in bucket storage Snapshot files with invalid epoch values found in bucket storage. Storage Health erroneous-files
Event ingest latency critical (p99 > 3s) p99 event-to-searchable latency exceeds 3000ms - pipeline backed up. Check parser queue depth, writer lag, Kafka offsets. Ingest - Overview high-ingest-latency
Ingest feed pressure high A single feed applies sustained pressure on the PollCoordinator, starving others; may lose data for low-priority feeds. Rate-limit or isolate the feed. Ingest - Overview feed-pressure-high
Kafka ingest queue sustained expiry errors The Kafka ingest-queue leader thread has been failing with record-expiry errors for multiple consecutive hours - consistent with Kafka under-replicated partitions building up. Ingest loss may be occurring silently. Kafka Consumer Lag & Broker Health kafka-ingest-queue-error
Local disk full - bucket storage downloads aborted LogScale is aborting segment downloads because the local disk is reported full. Queries that need segments not on local disk return partial results until retention/eviction frees space. Retention & Data Lifecycle Health local-disk-full
Node restarting frequently A node restarted more than twice in the last hour. Cluster JVM & Node Stability frequent-restarts
Global Snapshot Size Growing Rapidly Global snapshot is consuming a high percentage of JVM heap - sustained growth from tag cardinality or excessive configuration. Cluster Errors & Health high-cardinality-tag-oom
Global snapshot size early warning Global snapshot is approaching a level that may cause heap pressure. Monitor for further growth and check tag cardinality if it continues. Cluster Errors & Health high-cardinality-tag-oom
Global snapshot size exceeds 5 GB Global snapshot is consuming a critical percentage of JVM heap - ingest-pod OOM is likely imminent. Cluster Errors & Health global-snapshot-size
Node has CPU over 95% A node's average CPU usage above 95%. Cluster JVM & Node Stability high-cpu
High GC overhead (>30%) JVM spending > 30% of wall-clock in GC - imminent heap exhaustion/OOM risk. Distinct from the 25% warning in 'Excessive garbage collection'. Cluster JVM & Node Stability jvm-heap-pressure
High ingest event latency Average p99 event latency exceeds 3 seconds - ingest pipeline delays. Ingest - Overview high-ingest-latency
High parser processing time One or more parsers consuming excessive CPU, which delays ingest and degrades performance. Ingest - Overview high-ingest-latency
High tag cardinality on ingest pod A digest node has an abnormally high active datasource count - a proxy for sustained tag-cardinality growth; unchecked growth leads to OOMKill. Tag Cardinality & Datasource Health ingest-pod-oom
HTTP reachability check failures Inter-node HTTP reachability checks failing - network or node issues. Cluster Errors & Health http-unreachable
Ingest Error Rate Elevated Elevated ingest error rates across nodes - usually ingest-pod memory pressure (OOM cycles), Kafka backpressure, or parser failures dropping events. Ingest - Overview high-cardinality-tag-oom
Repository ingest rate doubled One or more repos saw ingest rate double vs the previous baseline - misconfigured sources or an incident. Ingest - Overview ingest-spike
Ingest writer lag Ingest-queue latency (parser completion to digest) past threshold - writers saturated, cannot keep pace; precedes search degradation. Ingest Writer Queue high-ingest-latency
Ingest writer queue overflow Writer in-memory job queue consistently rejecting work - writers blocked (I/O, GC, locks) or volume exceeds capacity. Sustained overflow causes Kafka lag and latency growth. Ingest Writer Queue ingest-writer-queue-full
Inter-node HTTP latency elevated Inter-node HTTP latency past threshold - degraded cluster comms; spikes precede connection timeouts. Complements the count-based connect-timeout-errors alert. Cluster Errors & Health http-unreachable
Java OutOfMemoryError detected A node threw java.lang.OutOfMemoryError. Cluster JVM & Node Stability java-oom
Kafka ingest queue reader errors The Kafka ingest-queue reader is erroring - may cause data loss or ingest delays. Kafka Consumer Lag & Broker Health kafka-ingest-queue-error
Kafka out of sync replicas detected Kafka reports out-of-sync replicas - potential data replication issues. Kafka Consumer Lag & Broker Health kafka-out-of-sync
Kafka ingest partition bytes imbalanced One or more ingest partitions receiving far more bytes than others (hot partition) - uneven lag, parser buildup on specific nodes, potential OOM. Check producer key distribution and partition assignment. Kafka Consumer Lag & Broker Health kafka-out-of-sync
Kafka publish failures detected Kafka publish operations failing with timeouts - cluster communication issues. Kafka Consumer Lag & Broker Health kafka-publish-failed
Low data retention on local disk Local-disk eviction has narrowed its deletion threshold to under 3 days while disk fill is within 10% of the target - retention is being shortened to keep up with disk pressure. Distinct from the disk-full alert. Retention & Data Lifecycle Health disk-fill-prediction
No bucket storage uploads No successful uploads to bucket storage - data not being persisted to object storage. Storage Health no-bucket-uploads
Parser queue depth elevated Ingest parsing time (p99) past threshold - parsers cannot keep up; sustained high parse times create backpressure and precede ingest lag. Ingest - Overview high-ingest-latency
Preflight check failure An org-transfer preflight compatibility check found errors or fatal incompatibilities - blocks organization transfers. Cluster Errors & Health preflight-error
Retention not running - deleteRecordsBefore absent No deleteRecordsBefore entries in the past 25 hours - retention silently stopped, disk will fill without warning. Check DataSyncJob, segment lifecycle logs, repo retention config. Retention & Data Lifecycle Health retention-not-running
Search segment queue degradation Search segment queue exceeded 500,000 entries - queries timeout or return incomplete results. Query Performance & Starvation search-degradation
Sustained bucket storage download failures Bucket-storage segment downloads are failing at an elevated rate on one or more nodes. Segment Prefetch & Download Health segment-download-loop
Shard factor pod count mismatch Running digest pod count does not match the configured shard factor - shards unserved or ingest over-replicated. Check LOGSCALE_SHARD_FACTOR or Helm values against actual pod count. Digest Partition Health shard-factor-mismatch
Too many streaming queries Excessive streaming queries - can exhaust cluster resources. Query Performance & Starvation query-cost-noisy-neighbor
Excessive garbage collection JVM spending > 25% of wall-clock in GC - degrades all operations. Cluster JVM & Node Stability jvm-heap-pressure
Version mismatch across cluster nodes Nodes running different LogScale versions - can cause inconsistent behavior. Cluster Errors & Health version-mismatch

Actions

Actions are the notification channels alerts route to. All 42 alerts fan out to Email, Slack, and PagerDuty. Placeholder recipients and endpoints ship with the package and are configured per environment in LogScale โ†’ Settings โ†’ Actions.

Action Type What it does
LogScale Email Alert email Sends a templated email with alert name, repository, timestamp, description, event count, and a LogScale link. Ships with a placeholder recipient; set real recipients in Settings โ†’ Actions.
LogScale Slack Webhook webhook POSTs a Slack message attachment (alert name as title/link, description, repository, and event count fields) to an incoming-webhook URL. Ships with a placeholder endpoint.
LogScale PagerDuty pager-duty Triggers a PagerDuty incident at critical severity via a routing key. Ships with a placeholder routing key.
LogScale Generic Webhook webhook POSTs a JSON payload (alert name, description, repository, event count, URL, triggered-at) to an arbitrary endpoint for custom integrations. Not wired to alerts by default; ships with a placeholder endpoint.

Scheduled Searches

Recurring background queries that produce periodic reports. Schedules are cron expressions in UTC.

Scheduled searches Schedule (UTC) Search window Purpose
Daily Capacity Check 10 1 * * * (01:10 daily) 5 days Daily snapshot of cluster capacity: total ingest volume, host count, and per-host ingest rate. Results can be forwarded to a tracking repository or external system for trend analysis.
Storage growth report 30 2 * * * (02:30 daily) 1 day Daily report tracking disk-usage trends and bucket-storage growth rate, projecting days until local disk is full.
Weekly error summary 0 8 * * 1 (Mon 08:00) 7 days Aggregates ERROR-level logs by class over the past 7 days, producing a ranked summary of the most frequent error types.

Prometheus and Grafana

An external observability stack that scrapes the LogScale cluster's metrics endpoint: time-series dashboards (Grafana), metric-based alerting (Prometheus rules routed through Alertmanager), and the scrape wiring that connects them. Because it operates on metrics rather than the LogScale query path, it keeps alerting when LogScale itself is degraded.

Grafana Dashboards

Grafana dashboards backed by the Prometheus datasource, all in the LogScale folder. They chart the cluster's humio_* metrics - the metric-side complement to the in-cluster log dashboards. Several are derived from CrowdStrike operational experience (harrier issues), noted where applicable.

Dashboard Folder Purpose
logscale-all-alerts LogScale Every LogScale-related Prometheus alert in every state - firing, pending, no-data, error and inactive - on a single panel for at-a-glance triage and alert-catalog review.
logscale-bucket-transfers LogScale Bucket transfer backlog and data at risk: under-replicated pending uploads, oldest upload age, backlog burn-down, retry/failure rates and transfer concurrency.
logscale-cluster-overview LogScale Data integrity and cluster health: under-replication, missing nodes, segment health, disk usage, HTTP connectivity, storage trends, consumer lag and datasource cardinality.
logscale-digest-pipeline LogScale Digest-pipeline latency, deserialization time, throughput rate, and active datasources. Digest is the critical path from ingest queue to searchable data.
logscale-disk-prediction LogScale Disk-usage trends and fill-time predictions for LogScale data volumes, from kubelet volume metrics and the primary disk usage metric.
logscale-dispatcher-starvation LogScale Pekko dispatcher thread starvation and duration timers. Starvation >500ms p99 means the default dispatcher is saturated - tasks wait behind long-running operations, causing cascading latency.
logscale-global-state LogScale Global snapshot size, cluster coordination wait times, operation latency by function, and clock skew - the health of distributed state management.
logscale-infrastructure LogScale Cluster stability: node health, JVM heap pressure, GC pauses, thread-pool saturation. Derived from operational experience (node-health-detection, jvm-heap-pressure, thread-pool-saturation).
logscale-ingest-pod-memory LogScale Ingest-pod memory: cluster-aggregate stats, top-N pod heatmaps, JVM heap pressure, an OOMKill timeline and alert correlation.
logscale-ingest-queue-detail LogScale Ingest-queue latency per partition, consumer counts, bytes and events per repo. Drills below the aggregate ingest view.
logscale-internode-network LogScale Inter-node HTTP request rate, external vs internal rate, latency and reachability failures, chatter throughput/reader/serialization, peer-to-peer download capacity, and outbound Kafka producer backpressure.
logscale-jvm-internals LogScale Pekko dispatcher task duration and starvation, executor thread-pool occupancy, queue depths and named worker thread counts for LogScale nodes.
logscale-kafka-health LogScale Ingest-pipeline queue depth, consumer counts, and partition latency. Replaces Kafka consumer lag with queue-side metrics from LogScale's own ingest pipeline.
logscale-live-queries LogScale Live and streaming query health: active live-query counts, total live events, query wait times, and thread limits.
logscale-memory-detail LogScale JVM heap, GC pressure and memory detail for all LogScale pods. Thresholds: JVM heap red at 90%, yellow at 75%; GC hiccup red at 500 ms, yellow at 250 ms.
logscale-object-storage LogScale Object-storage API performance: connection-acquire latency, retry rates, connection-pool utilization, task-handler latency, per-backend latency, multipart upload activity and non-GET/PUT operation concurrency.
logscale-parser-performance LogScale Parser performance: cluster aggregates, top-N parsers by latency, per-parser table stats and ingest health alerts.
logscale-performance LogScale Query and ingest SLO: cluster-wide golden signals. Drill-downs: logscale-query-execution, logscale-dispatcher-starvation, logscale-live-queries, logscale-digest-pipeline.
logscale-query-cache LogScale Query and table cache utilization, hit/miss ratios, disk usage. Cache misses force expensive segment reads - size caches accordingly.
logscale-query-execution LogScale Query cost and repository load: per-repository query cost (top expensive repositories), static-query cache reuse, cross-query builder cache effectiveness, query starvation, and the live vs streaming query mix.
logscale-read-efficiency LogScale Query I/O efficiency: bytes scanned vs bytes relevant, segment read/decompress/map timings. High scan-to-result ratio indicates expensive, poorly selective queries.
logscale-bucket-storage-archiving LogScale Bucket-storage read/write throughput, concurrency-pool health, and the archiving pipeline. High pending_concurrency_acquires means requests to the object-storage backend are queuing.
logscale-segment-ops LogScale Segment creation rate, merge activity, upload latency, and write-pipeline health.
logscale-segment-write-merge LogScale Segment write rate, compression ratios, merge CPU, merger/validator thread counts, and write-pipeline queues.

Prometheus Alert Rules

Metric-based alert rules evaluated by Prometheus against the cluster's humio_*, node_*, and kube_* metrics, defined as two PrometheusRule resources across nine groups. Firing alerts are routed by Alertmanager. Each rule carries a severity label and a for duration (how long the condition must hold before firing).

%%{init: {'theme': 'default', 'themeVariables': {'fontSize': '12px'}}}%% graph TD m["LogScale metrics\n(humio_* / node_* / kube_*)"] prom["Prometheus\n(rule evaluation)"] rules["PrometheusRule\n(9 groups)"] am["Alertmanager"] recv(["Receivers\n(severity-routed)"]) m -->|scraped| prom prom -->|evaluates every interval| rules rules -->|fires after 'for'| am am -->|dedup / group / route| recv
%%{init: {'theme': 'default', 'themeVariables': {'fontSize': '12px'}}}%% graph TD m["LogScale metrics\n(humio_* / node_* / kube_*)"] prom["Prometheus\n(rule evaluation)"] rules["PrometheusRule\n(9 groups)"] am["Alertmanager"] recv(["Receivers\n(severity-routed)"]) m -->|scraped| prom prom -->|evaluates every interval| rules rules -->|fires after 'for'| am am -->|dedup / group / route| recv

logscale_data_integrity

Alert Severity For Summary
SegmentSizeAnomaly critical 15m Segment sizes approaching 2GB (p99 compressed size above 2GB). Check segment write patterns and that bucket-storage uploads are healthy.
BucketStorageFetchFailure warning 10m S3 SDK retry count per request over a 5-minute window exceeds 2 - S3 objects missing or consistently failing to fetch, causing query gaps.
UnderReplicatedSegments warning 30m Segments without a known second replica awaiting bucket-storage upload. Expected briefly during rolling restarts; sustained means no redundancy.

logscale_cluster_stability

Alert Severity For Summary
FrequentRestarts critical 0s humio container restarted more than twice in 1 hour - crash loop, typically OOM kills or bad config after an upgrade.
GCPausePressure warning 5m JVM hiccup latency p99 > 250ms - stop-the-world GC pressure degrading all operations; often precedes OOM.
HumioPodPendingUnschedulable critical 5m A LogScale pod has been Pending for more than 5 minutes. A common cause is a data-volume claim stuck in Terminating from a previous pod, which blocks the replacement pod from scheduling.
HumioPVCTerminating warning 10m A LogScale data-volume claim has been deleting for over 10 minutes. Healthy deletes finish in seconds; a stuck delete is usually a hung finalizer and blocks any new pod that uses the same claim.
JVMHeapPressure critical 10m JVM heap above 90% - excessive GC pauses degrade ingest and query latency. A rising post-GC floor indicates a leak.
JVMHeapWarning warning 15m JVM heap above 75% - trending high; escalates to critical if the post-GC floor keeps rising.
NodeDown critical 5m The metrics endpoint (up{job="logscale-metrics"}) unreachable > 5 min. Prolonged downtime indicates crash loops, OOM kills, or misconfiguration.
ThreadPoolSaturation critical 5m Pekko default-dispatcher delay > 500ms - causes heartbeat misses, HTTP timeouts, and false node-DOWN detection. High cascade risk.

logscale_ingest_query

Alert Severity For Summary
SearchQueueOverflow critical 5m Query segment-scan queue > 500K - queries timeout or return incomplete results. Causes: expensive regex, concurrent query burst, post-upgrade regression.
SearchQueueDegraded warning 10m Search queue > 200K - performance degraded; investigate expensive queries or contention before it reaches critical.
IngestRateDrop warning 10m Ingest rate dropped > 50% vs the previous window - usually a broken upstream pipeline, not a healthy cluster.
KafkaConsumerLagHigh warning 10m On an ingest-queue partition, the distance between the latest offset and the lowest offset LogScale may still need to read again (for failover) exceeds 100K offsets. Past Kafka topic retention, Kafka can delete records LogScale may still need.
KafkaConsumerLagGrowing critical 15m The total of that distance across ingest-queue partitions has been rising for 15 min. Past Kafka topic retention, Kafka can delete records LogScale may still need.
HighParserTime warning 10m A parser's p99 event-processing time has exceeded 100 ms for more than 10 minutes, slowing ingest for every event that uses it.

logscale_ingest_oom

Alert Severity For Summary
IngestPodOOMKilled critical 0s An ingest-only pod terminated by OOM - often high-cardinality tag grouping creating excessive in-memory state.
IngestPodMemoryPressure warning 10m Ingest-pod working set > 85% of its limit - continued growth triggers OOMKill. Check high-cardinality tag grouping.
LogScaleCorePodOOMKilled critical 0s A log-core pod was terminated for exceeding its container memory limit. The JVM is killed before it can log, so log-based out-of-memory alerts stay silent.
LogScalePodCrashLoop critical 5m A LogScale container restarted 2+ times in 15 minutes with OOMKilled as the terminating reason - a persistent crash loop that will not recover on its own.

logscale_disk_prediction

Alert Severity For Summary
DiskFillPrediction4h critical 15m 6-hour write trend predicts the data partition fills within 4 hours and free space is below 15% - immediate action to prevent ingest halt.
DiskFillPrediction24h warning 30m 6-hour write trend predicts fill within 24 hours - plan capacity expansion or cleanup.
FilesystemReadonly critical 1m Data filesystem remounted read-only (I/O errors or hardware failure) - all writes fail. Immediate investigation.
DiskUsageCritical critical 5m Data partition over 90% full - LogScale may stop accepting ingest to protect integrity. Free space or expand the volume.

logscale_hardware_health

Alert Severity For Summary
NVMeSpareCapacityLow warning 5m NVMe available spare below 10% - drive approaching end of life; replace proactively. Requires node_exporter NVMe collector.
NVMeMediaErrors warning 0s New NVMe media errors in the last hour - physical drive degradation; monitor growth and plan replacement.
NVMeEnduranceCritical critical 5m NVMe consumed over 90% of rated write endurance - planned replacement, should not be deferred.

logscale_memory_pressure

Alert Severity For Summary
IngestPodHeapCreep warning 2h Ingest-pod working set growing > 500MB/hour sustained for 2h - a steady climb without plateau precedes OOM, commonly tag-cardinality growth. Requires cAdvisor/kubelet metrics.

logscale_global_state

Alert Severity For Summary
ClusterTimeSkewHigh warning 5m Node-to-node clock skew exceeds 500 ms. Time-based queries, retention windows and coordination degrade at high skew.
GlobalPublishWaitP99High warning 10m Global-state publish wait p99 exceeds 1 s - the Kafka publish for global state is slow to acknowledge, stalling coordination cluster-wide.
GlobalSnapshotSizeHigh warning 30m Global-snapshot file exceeded 3GB - leading indicator of ingest-node memory pressure from high-cardinality tag additions (snapshot holds all datasource metadata). Check recent tag-config changes.

logscale_cluster_membership

Alert Severity For Summary
MissingStatefulNode critical 2m One or more stateful (digest and storage) nodes are marked missing. Owned Kafka partitions and local segments are unavailable until the node returns or is replaced.
FailedHttpChecksSustained warning 10m Nodes have been unable to reach each other over the API for at least ten minutes - a network partition or a hung node. Queries routed to the unreachable node fail.
LosableNodeHeadroomExhausted warning 15m A zone can no longer safely absorb another node loss: re-replicating that node's segments would push a remaining zone over its disk capacity. If a stateful node in the zone goes down, expect data-loss risk during recovery.