logscale/operations Dashboards
Cluster Errors & Health
Cluster Sizing & Growth
Cluster Upgrade & Regression Detection
External Feeds (SQS and FDR)
JVM & Node Stability
Kafka Consumer Lag & Broker Health
Kubernetes Operator Reconcile Health & CRD Lifecycle
Retention & Data Lifecycle Health
Scheduled Search & Workflow Health
Segment Prefetch & Download Health
Tag Cardinality & Datasource Health
Alert Status Overview
| Widget | Description | Type |
|---|---|---|
Per-action failure detail grouped by view (dataspace), actionName, and triggering category (ScheduledSearch/FilterAlert/AggregateAlert/Alert), with failure count and last occurrence. Answers which action, in which repo, driven by which trigger type, is failing and when did it last fail. Sort by failures to prioritise; use last_seen to confirm whether the issue is ongoing or already resolved.
| Table | |
Count of failed action invocations in the last 24h (subCategory=Action, severity=Error), the search/alert ran and matched, but the notification (email, webhook, Slack, PagerDuty) did not reach the customer. Non-zero = alerts firing silently. See the breakdown below.
| Single Value | |
Failed action invocations per bucket (subCategory=Action, severity=Error), split by actionName. Covers both scheduled-search and alert triggers. The match was found but the notification failed to deliver. A step change on one series isolates a single broken integration; a broad rise across series points at shared infra (action proxy, egress, DNS).
| Time Chart | |
Warning/Error activity across the three alert engines (FilterAlert, AggregateAlert, Alert) grouped by view (dataspace) and alertName. Surfaces individual alerts whose query errors, catch-up limits, missing-data warnings, or action failures are degrading coverage. Sort by event count to prioritise the noisiest failing alert. last_seen shows recency.
| Table | |
Timechart of activity events with severity=Error split by subCategory. Sustained non-zero in the Action subCategory means an outbound channel is degraded (see Action Delivery section). Sustained non-zero in the Query subCategory means one or more alert queries themselves are failing.
| Time Chart | |
Average of queryProcessedEvents per alert. Two alerts with similar p95 query time but 10x difference in scanned events tells you which one is paying for a missing tag filter or missing repository scope. High values here are the cheap next-step when tuning after Top Expensive Alerts.
| Table | |
Distribution of alert-run outcomes grouped by activity-log message. The ratio of fired vs. no-results vs. throttled is more actionable than any raw count. A dashboard dominated by throttled means user-configured throttleTime is misaligned with match rate; a dashboard dominated by no results means alerts are cheap but useless.
| Table | |
Timechart of activity-event volume split by alert type. Use this to spot when a specific category (Alert / FilterAlert / AggregateAlert) goes silent while others continue, points at one alert-runner subsystem rather than a cluster-wide outage.
| Time Chart | |
Groups fire volume by view. Shows which views dominate the alert engine's workload. A single view holding most of the fire volume is a scaling concern; a sudden shift in the distribution correlates with a recent alert configuration change on that view.
| Table | |
Timechart of queryTimeMillis percentiles across all alert executions. This is the leading indicator of execution health, spikes here forecast schedule drift and silent alerts several intervals before the drift itself becomes visible.
| Time Chart | |
Alerts sorted ascending by the timestamp of their most recent activity event. Any polling alert should emit at least one event per poll interval, even a no results one. The top rows here are the alerts to investigate first, the alert is either disabled, its scheduler is stuck, or its query is failing.
| Table | |
Ranks alerts by the p95 of their queryTimeMillis. The top three are the cost-attribution list, the ones to tune or throttle when the engine backs up.
| Table | |
Ranked count of failed action invocations by actionName over the window (subCategory=Action, severity=Error). Use to triage which integration is most broken and how widely it is affecting deliveries. Pair with the breakdown table below to see the owning view and last failure time.
| Bar Chart | |
Ranks the actual failure text on error-severity events. One dominant message means one root cause across many alerts (e.g. one webhook receiver down). Many small counts across many messages means several independent failures, a broader problem, escalate.
| Table | |
Ranked table of alerts by fire count. Use to identify noisy alerts that should be tuned or throttled. Complements humio-activity's Alerts Triggered table by ranking by count directly rather than requiring manual sort.
| Table | |
Count of activity events where an alert matched its query and invoked at least one action in the last 24h. Dropping to zero is a strong signal the polling loop stalled, cross-check the alert-heartbeats scalar on the Scheduled Search & Workflow Health dashboard.
| Single Value |
Cluster Errors & Health
| Widget | Description | Type |
|---|---|---|
Currently-running live query count (metric: livequery-count). Detail: Query Performance Overview; Grafana logscale-live-queries + logscale-query-execution.
| Time Chart | |
Per-node bucket-storage-upload-latency-max-ms. Rising p99 = object storage is slow to accept segments, delaying offload.
| Time Chart | |
Total ingest bytes/sec across the cluster (metric: ingest-bytes-total, m1 rate).
| Time Chart | |
One series per #vhost showing metric-event volume per hour. A series dropping to zero = that node stopped reporting.
| Time Chart | |
Per-vhost CPU utilisation Hot rows = nodes under sustained CPU pressure.
| Heat Map | |
Per-vhost CPU utilisation
| Time Chart | |
Digest partition leadership moves as a Sankey diagram: each ribbon goes from the node that lost a partition (fromVhost) to the node that took it over (toVhost); width is the number of moves. Moves are expected around restarts and deploys.
| Sankey | |
Cluster-wide written-events rate, the ingest pipeline's outflow. Drops correlate with digest stalls, GC storms, or writer-queue overflow.
| Time Chart | |
Most common exceptions grouped by class and message. ago_sec = seconds since last occurrence, recent + frequent = investigate first.
| Table | |
Timechart of activity events with severity=Error split by subCategory. Sustained non-zero in the Action subCategory means an outbound channel is degraded (see Action Delivery section). Sustained non-zero in the Query subCategory means one or more alert queries themselves are failing.
| Time Chart | |
Per-vhost inter-node health-check failures (m1 rate).
| Time Chart | |
Sum of failed-http-checks m1 rate across all vhosts. Any sustained non-zero = at least one node is unreachable from another node. Detail: Cluster membership section; Grafana logscale-internode-network.
| Time Chart | |
Inflight FDR messages across feeds. Sustained rising = FDR pulling faster than LogScale can ingest.
| Time Chart | |
Time spent waiting to read a global-state update back from Kafka. Rising p99 = coordination stalls cluster-wide.
| Time Chart | |
HTTP 400-499 responses per status code.
| Time Chart | |
HTTP 500+ responses split per status code. LogScale exports no per-status HTTP metric, so this is the only per-code view.
| Time Chart | |
Which API paths return 500+ responses most often. Pinpoints the request-path regression when 5xx volume climbs.
| Table | |
Per-vhost event-latency p99. Red row = laggy node; over 3M ms fires an alert.
| Heat Map | |
Seconds until an ingested event is searchable, counted from when it enters the ingest queue (parsing not included). The value is the 99th percentile: 4.5 on worst node means that on the slowest node, 99% of events were searchable within 4.5 seconds and 1% took longer. median node is the middle node: half the nodes are faster, half slower. Some delay is normal because events are buffered briefly before they are written; what matters is change compared with the days before. Worst high and median low means one node is behind; both rising means the whole cluster is falling behind. If latency is going up, use Open Ingest Overview to drill down into queue, partition and parser latency.
| Time Chart | |
Per-vhost ingest-queue-latency over 24h. Red rows = nodes falling behind on Kafka consume.
| Heat Map | |
Highest jvm-hiccup-latency p99 across nodes (10m). Rising = GC pauses or thread scheduling stalls that block user work. Detail: JVM & Node Stability; Grafana logscale-jvm-internals.
| gauge | |
ERROR vs WARN vs INFO log volume per hour. A shifting ratio (WARN climbing before ERROR) is a leading indicator of degrading paths firing fallback branches.
| Time Chart | |
Highest primary-disk-usage across nodes (0-100). Green under 85, yellow 85-95, red at 95+. Detail: Retention & Data Lifecycle; Grafana logscale-cluster-overview.
| gauge | |
Highest jvm-heap-usage-percent across nodes (normalized 0-100). Green under 85, yellow 85-95, red at 95+. Detail: JVM & Node Stability; Grafana logscale-jvm-internals + logscale-memory-detail.
| gauge | |
Highest lag of LogScale's own ingest reader on any ingest-topic partition (records-lag-max from the ingest consumers' Kafka client metrics), per 5-min bucket.
| Time Chart | |
Nodes considered dead by the cluster (metric: missing-cluster-nodes). Non-zero = investigate; a missing node cannot serve queries or ingest. Detail: humio-insights Errors; Grafana logscale-global-state.
| Single Value | |
Log lines reporting a missing segment, split by reason. Lost means no node has a copy and it is not in bucket storage. Unavailable means copies exist but every node holding one is down; the data returns when the node does. Counts are log lines, and each check logs only a limited number of segments, so read them as yes or no, not as a segment count. The node that reports the line is the node running the check, not the node that holds the segment; the table of missing segments shows the hosts.
| Time Chart | |
ERROR lines in the last 24 hours reporting a segment that no node has and that is not in bucket storage: the data cannot be read. Lost means no node has a copy and it is not in bucket storage. Unavailable means copies exist but every node holding one is down; the data returns when the node does. Counts are log lines, and each check logs only a limited number of segments, so read them as yes or no, not as a segment count. Green: none. Red: one or more.
| Single Value | |
Each segment reported missing, with the reason, the hosts that hold a copy (currentHosts) and the hosts that should hold one (ownerHosts). For Unavailable, bringing back a host in currentHosts makes the data readable again. Lost segments have an empty currentHosts. Newest first.
| Table | |
Lines in the last 24 hours reporting a segment whose copies are all on nodes that are down. Common during restarts; the data returns when the node does. Lost means no node has a copy and it is not in bucket storage. Unavailable means copies exist but every node holding one is down; the data returns when the node does. Counts are log lines, and each check logs only a limited number of segments, so read them as yes or no, not as a segment count. Green: none. Yellow: one or more.
| Single Value | |
Dead stateful (digest and storage) nodes (metric: missing-cluster-nodes-stateful). More serious than the total count, owned partitions and local segments go dark. Detail: Cluster membership section (below).
| Single Value | |
Exception classes with first occurrence in the last 24h, newest first. After a deploy, an entry near the top with high occurrence count is a strong regression candidate.
| Table | |
Node exits per hour from ShutdownHook. graceful = server closed. failure = terminated without proper shutdown. Hard crashes (SIGKILL/OOM) log nothing here.
| Time Chart | |
Distinct #vhost values currently emitting metrics. Sanity-check the cluster size against expected node count. Detail: JVM & Node Stability; Grafana logscale-infrastructure.
| Single Value | |
Count of ingest-topic partitions where LogScale's own ingest reader is behind the latest offset, using the last records-lag value per partition from the ingest consumers' Kafka client metrics (last 5 min). Zero = the ingest reader is caught up on every partition.
| Single Value | |
Ingest parser error rate (metric family: data-ingester-errors and data-ingester-parser-errors). Any sustained non-zero = malformed input reaching a parser.
| Time Chart | |
Live-tailing raw ERROR-level events. Row-level severity coloring is applied by LogScale's list-view renderer. The diagnostic value is in the row content, read the actual failure message text. Detail: Application errors detail section.
| Event List | |
Per-vhost local-query-segments-queue over 24h. Red rows = nodes with queued search work; precursor to search timeouts. Set palette to Green yellow red for temperature effect. Detail: Query Performance Overview; Grafana logscale-query-execution.
| Heat Map | |
Rate of segment operations by type (upload / merge / delete / missing: lost / missing: unavailable) as a stacked area chart. The mix reveals which lifecycle stage dominates activity.
| Time Chart | |
Share of ERROR-level log volume by dataspace. A single repo carrying >70% of the pie = focus tuning there first. Detail: Application errors detail section.
| Pie Chart | |
Which repositories dominate ingest volume today (event-latency-repo rate summed per repo).
| Pie Chart | |
ingest-writer-queue-full rate as events/hour (m1 × 3600). Over 12k = nodes are laggy.
| Time Chart |
Cluster Sizing & Growth
| Widget | Description | Type |
|---|---|---|
Average CPU % of the nodes in each node role, same formula as the trend chart above. The role is taken from the line each node logs when it starts; nodes that have not started in the last 30 days show as unknown.
| Time Chart | |
Nodes ranked by 24h-average CPU% (same formula as the trend chart above). Identifies the hottest nodes, candidates for scale-up or load-shedding. Same source as cluster-errors-health, so numbers should match there.
| Table | |
Per-#vhost CPU p95/p90 over 2 days with a scaleStatus verdict. Scale up when p95 > 90 AND p90 > 80 (both tail and body are hot). Scale down when p95 < 75 AND p90 < 50 (persistently cool). Otherwise Normal. Sorted so nodes needing action float up. Adopted from the internal Scaling Dashboard with parameters stripped (single-cluster deployment).
| Table | |
Per-node CPU% over 24h. Formula, (user + system + nice) / (user + system + nice + idle) * 100. Matches cluster-errors-health and humio-insights cpuUsagePercent. Sustained > 80% (horizontal SLO line) = node needs relief (reduce queries, add capacity, or rebalance ingest).
| Time Chart | |
Per-node disk usage vs the eviction watermark (85%) and hard ceiling (95%). status = OK when at/below the watermark, WATCH between watermark and ceiling, CRITICAL near/above 95% (ingest stall risk). headroom_to_ceiling is the raw gap to 95%.
| Table | |
Per-node local-disk usage (primary-disk-usage gauge) over 7 days. Expect flat lines near the 85% eviction watermark, that is the designed steady state, NOT a warning. Local disk is a cache backed by bucket storage; LogScale evicts cold segments at the watermark. Danger signs: sustained climb ABOVE 85% (eviction behind), approach to the 95% ceiling (ingest can stall).
| Time Chart | |
Count of nodes currently above 90%, past the eviction watermark and approaching the 95% ceiling. 0 = eviction is comfortably holding. Non-zero → check local disk eviction and bucket-storage uploads on those nodes. Colour, green at 0, red at ≥1.
| Single Value | |
Count of nodes exceeding the 2.25 TB/day recommended per-node ingest limit. Non-zero = that node is at risk of falling behind on digest. Colour, green at 0, red at ≥1.
| Single Value | |
Nodes whose 7-day p95 event latency is above 60 s. 0 means every node keeps up with incoming events. Ingest Latency Percentiles shows when latency was high.
| Single Value | |
p50 to p100 of event latency in ms, on a log scale: time from insert into the ingest queue until events are in a segment block and live queries are updated. The line at 60 000 ms is the alert threshold.
| Time Chart | |
Ingest throughput per node over 7 days (5-minute rate m5, converted to GB/day). A single high line = a hot node that is doing more digest work than peers, candidate for ingest rebalancing. Roughly-flat lines across nodes = balanced.
| Time Chart | |
Per-node JVM heap utilisation from the jvm-heap-usage-percent gauge. This is the LogScale process's own memory, the ONE memory signal that maps to real OOM risk. Sustained > 85% and long GC pauses (see JVM & Node Stability) means the heap is undersized OR a query is memory-heavy. cluster-errors-health uses the max of this metric as its top-level gauge.
| Time Chart | |
Distinct #vhost count over 30 days, hourly buckets. Drops = unexpected node loss (correlate with the JVM/Node Stability dashboard). Step-ups confirm scale-up events completed. Uses stat_cpu because every node reports it every minute, no gaps.
| Time Chart | |
Share of RAM the operating system uses to cache file data: 100 × Cached / MemTotal. On LogScale nodes the cache holds recently used segment files, so a high value is expected. Not a capacity signal on its own; use JVM heap % and System Memory Used for that.
| Time Chart | |
One row per resource: current cluster maximum for CPU %, JVM heap %, ingest latency (s), disk % and ingest per node (TB/day), compared with its scale-up threshold. Green OK, amber WATCH (85% of threshold), red SCALE UP. Memory uses JVM heap, not page cache.
| Table | |
Memory used by processes per node, excluding reclaimable page cache: 100 × (MemTotal − MemFree − Buffers − Cached) / MemTotal. On a LogScale node this is mostly JVM heap and off-heap memory. Compare with JVM heap %.
| Time Chart | |
Total cluster ingest rate in TB/day, summed over all nodes from the ingest-bytes counter's 5-minute rate (m5). Use for capacity planning against licensed limits and hardware sizing.
| Single Value |
Cluster Upgrade & Regression Detection
| Widget | Description | Type |
|---|---|---|
Segments named in ERROR or WARN lines about offsets older than Kafka retention, or about digest failing to fetch the segment it needs to start at the right offset. A digest-start fetch failure makes the partition replay from the oldest event in Kafka. Most events first.
| Table | |
Segments a query skipped on this node because downloading them from bucket storage kept failing until the retry timeout. The affected queries return partial results. Empty is normal: no segment was skipped.
| Time Chart | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Completed segment uploads to bucket storage across all nodes in each period. A drop compared with the days before the upgrade means fewer segments reach bucket storage; Nodes That Stopped Uploading Segments shows which nodes. Empty when bucket storage is not configured.
| Time Chart | |
Warnings and errors logged while uploading to bucket storage, by type, for example a segment upload that failed, a global snapshot that failed to upload, or a file that failed a checksum check. Messages that match no known type are shown by their own text. The table below shows the cause and the status code returned by bucket storage. Empty is normal: no upload problems.
| Time Chart | |
Upload problems grouped by type, with the cause logged with them (for example the error returned by bucket storage), the HTTP status code bucket storage returned, how many log lines, how many nodes, and when each was last seen. A status code of 500 or 503 from bucket storage means the storage service itself failed the request. Empty is normal: no upload problems.
| Table | |
Nodes that were uploading segments to bucket storage but have not uploaded in the last hour. Blue: none. Red: one or more nodes stopped uploading. Without bucket storage this reads 0.
| Single Value | |
Seconds a segment upload to bucket storage took: median and the slowest 1%. Rising upload time means bucket storage or the network to it got slower. Compare with the days before the upgrade. Empty when bucket storage is not configured.
| Time Chart | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Seconds the slowest events took to digest beyond the planned buffering delay, on the worst node in the last 15 minutes (95th percentile of its 10-second checks, so one odd spike does not count). Green: under 3 seconds. Yellow: 3 to 60 seconds. Red: 60 seconds or more.
| Single Value | |
Seconds the slowest event in each period took to digest beyond the planned buffering delay, for the worst node and the median node. Worst high and median low means one node is the problem; both rising together means the whole cluster. Compare with the days before the upgrade.
| Time Chart | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
ERROR log lines reporting snapshot files in bucket storage with an epoch higher than the cluster's current epoch, across all nodes. Empty is normal.
| Time Chart | |
ERROR log lines per hour. The table below shows which loggers and messages make up the count.
| Time Chart | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Nodes that logged a health check at ERROR level in the last 15 minutes. Each node logs its health-check state every 10 seconds; ERROR means a check reported DOWN or failed to run. Green: no node. Red: one or more.
| Single Value | |
Highest backlog (pressure) reported by any ingest feed. Higher values mean the feed expects more polls to return data before it is caught up. Only ingest feeds that LogScale polls itself (AWS S3/SQS, Azure Event Hub) report pressure; data pushed over HTTP or Kafka does not. Empty is normal when no ingest feeds are configured.
| Time Chart | |
Log lines where a node found that other nodes had taken over its digest partitions because they considered it dead, per node. The node then shuts itself down. Empty is normal.
| Time Chart | |
Growth of the global snapshot file between the first and the last hour of the last 24 hours, on the node where it grew most. Every node holds global in memory, so steady growth means steady heap growth. Lightweight ingest nodes write a partial snapshot and grow less. Green: under 25%. Yellow: 25% to 50%. Red: 50% or more.
| Single Value | |
Size of the global snapshot file, which every node holds in memory. Compare the level before and after the upgrade; a steady rise without more datasources or repositories points at segment metadata building up.
| Time Chart | |
Share of time the node with the most garbage collection spent collecting, in the last 15 minutes. Green: under 25%. Yellow: 25% to 30%. Red: 30% or more, the same levels as the GC alerts. Readings from a node restart are left out. GC Time, Worst and Median Node shows the trend.
| Single Value | |
Milliseconds per second spent in garbage collection, all collectors added together, for the worst node and the median node. 100 means 10% of the time, 1000 means one collector was busy the whole time. Worst high and median low means one node is the problem; both rising together means the whole cluster. Rising GC time usually comes before out-of-memory crashes; compare with the OOM chart next to it and with the days before the upgrade. Readings above 1000 per collector are left out: they come from the counter starting again at zero when a node restarts.
| Time Chart | |
Requests answered with a 4xx status, one series per status code. Empty means no 4xx responses.
| Time Chart | |
Minutes since the last minute in which more than 10% of requests were answered with a 4xx status, within the last hour. Some 4xx responses are normal client behaviour, so only minutes above 10% count. 0 means it is happening right now. Red: under 10, the problem is happening now. Yellow: 10 to 59, it happened in the last hour but has stopped for at least 10 minutes. Green: 60, none in the last hour.
| Single Value | |
Minutes since LogScale last failed to answer a request in time (returned with status 503), within the last hour. 0 means it is happening right now. Red: under 10, the problem is happening now. Yellow: 10 to 59, it happened in the last hour but has stopped for at least 10 minutes. Green: 60, none in the last hour.
| Single Value | |
Requests answered with a 5xx status, one series per status code. Routine 504 replies to cache-only query requests between nodes are excluded. Empty means no 5xx responses.
| Time Chart | |
Minutes since a request was last answered with a 5xx status, within the last hour. Any 5xx counts. Routine 504 replies to cache-only query requests between nodes are excluded. 0 means it is happening right now. Red: under 10, the problem is happening now. Yellow: 10 to 59, it happened in the last hour but has stopped for at least 10 minutes. Green: 60, none in the last hour.
| Single Value | |
Ingest requests from log shippers that were answered with an error status (400 or higher), per status code. Rising errors mean shippers are being turned away, for example because the cluster is overloaded or the requests are too large. Requests rejected before a status was set are not counted. Compare with the days before the upgrade.
| Time Chart | |
Seconds the cluster takes to answer ingest requests from log shippers: median and the slowest 1%. Only external requests are included. This is the delay shippers see before an event is accepted, which Event Latency does not include. Rising here while Event Latency is flat points at the ingest front end, including parsing. Compare with the days before the upgrade.
| Time Chart | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Live queries cancelled because ingest delay on the node exceeded the configured threshold and was still increasing, per node. Empty is normal: no live queries were cancelled.
| Time Chart | |
Seconds from an event entering the ingest queue until it is searchable, parsing not included, for the slowest 1% of events, for the worst node and the median node. Worst high and median low means one node is the problem; both rising together means the whole cluster. Compare with the days before the upgrade.
| Time Chart | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Ingest partitions whose lowest queue position LogScale may still need to read again did not move forward in each 2-hour period. It should keep moving on every partition. A few partitions here are stuck; all partitions here means progress stopped across the cluster. The position updates about every 40 minutes, so shorter stalls do not show. Empty is normal: every partition progressed.
| Time Chart | |
Data ingested across the cluster, as a daily rate. Shows the load behind the latency charts: latency that rises with ingest volume points at load; latency that rises while volume stays flat points at the cluster itself, for example a regression after the upgrade.
| Time Chart | |
Seconds an event waits in the ingest queue, from after parsing until a digest node has read it. This is only the queue part of Event Latency: if both rise together, the delay is in the queue or in reading from it. Shown for the slowest 1% of events, for the worst node and the median node. Worst high and median low means one node is the problem; both rising together means the whole cluster. Compare with the days before the upgrade.
| Time Chart | |
Records in the Kafka ingest queue that LogScale has not read yet, summed over all partitions. A growing line means digest is not reading the queue fast enough. Compare with the days before the upgrade.
| Time Chart | |
How often per second writing ingested data to the Kafka ingest queue timed out, summed over all nodes and averaged over a minute. Timeouts mean data received over HTTP could not be handed to the queue in time. Compare with the days before the upgrade.
| Time Chart | |
ERROR log lines from the Kafka ingest queue reader, ranked by node (top 15). Empty is normal: no errors.
| Bar Chart | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Kafka partitions per topic, per minute, where one or more replicas are not in sync with the leader, as reported by LogScale. Empty is normal: all replicas were in sync.
| Time Chart | |
Digest partitions whose leadership moved to another node, split by the node receiving them. Empty is normal outside restarts and deploys.
| Time Chart | |
Retention configured on each Kafka topic, as logged by the cluster. The ingest topic must keep events longer than it takes to replicate and merge segments; a short retention removes the safety net for segments that are not yet durable. distinct_values_count above 1 means the setting changed in the time window.
| Table | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Cluster-wide mini-segment counts: waiting to be merged, already merged but not yet deleted, and all mini-segments including tombstones. Mini-segments are merged only after they are replicated, so a waiting count that keeps rising means the merge pipeline is falling behind, and global grows with it. A merged-not-deleted count that keeps rising means deletes are behind.
| Time Chart | |
Failed segment downloads from bucket storage whose error mentions unexpected message type: LastHttpContent, per node. Empty is normal: no such download errors. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and this widget is empty.
| Time Chart | |
Nodes that shut themselves down because of an internal failure, per 10 minutes, split by cause: out of memory, rejected transaction, digest partitions taken over by other nodes, stuck ingest, stuck dispatcher threads, or a Kafka epoch change. other is any other cause; the table below shows its full text. A graceful shutdown or a killed process is not counted here. Empty is normal: no node failed. Only shutdowns LogScale starts itself. Kills such as a forced stop or SIGKILL, and certain crashes, do not log this; they show as unclean in All Restarts by Host and in Unclean Shutdowns (Segment Prefetch & Download Health).
| Time Chart | |
Failure shutdowns per host with the cause and its full text, how many, and the last one. Use the full text for rows marked other. Empty is normal: no node failed. Only shutdowns LogScale starts itself. Kills such as a forced stop or SIGKILL, and certain crashes, do not log this; they show as unclean in All Restarts by Host and in Unclean Shutdowns (Segment Prefetch & Download Health).
| Table | |
Every node start with the host, restart type, cause and the LogScale version the node started with. The restart type comes from the last line the node logged before starting: a graceful shutdown, a failure shutdown, an out-of-memory error, or nothing. no shutdown logged means the process stopped without logging anything, for example a kill or a crash. graceful means a clean shutdown started; it does not prove it finished. A graceful shutdown just before the time range can make a normal restart show as no shutdown logged. Needs LogScale 1.174 or later. Empty means no node started.
| Table | |
How many node starts there were in each period, across all nodes. Line it up with the charts above: GC time climbing, then an out-of-memory error, then a restart is the out-of-memory crash pattern. All Restarts by Host shows which nodes and why. Empty is normal: no node started.
| Time Chart | |
Share of nodes not running the version most nodes run, in the last 15 minutes. Green: all nodes run the same version. Yellow: under 20% of nodes differ. Red: 20% or more differ. A short mismatch is expected during a rolling upgrade; a lasting one means the upgrade stalled or a node did not update. The label is the newest version running on any node.
| Single Value | |
Number of nodes running each LogScale version in the last 15 minutes; the label is the version. During a rolling upgrade there is more than one version, and LogScale Version Mismatch next to it turns yellow or red.
| Single Value | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Hosts with an out-of-memory error: whether the node crashed, the error came from the actor system (check Node Restarts vs GC Pressure), or only a query was cancelled, the error and its message, how often, and when it was first and last seen. Compare first seen with the upgrade time. A process killed by the operating system for using too much memory logs nothing; it shows as no shutdown logged in the Nodes section. Empty is normal: no out-of-memory errors.
| Table | |
Out-of-memory errors: node crashes caused by a Java out-of-memory error, out-of-memory errors in the actor system, and queries cancelled because direct memory could not be allocated. Green: none. Red: one or more; see OOM Errors by Host for which hosts and whether the node crashed. A process killed by the operating system for using too much memory logs nothing; it shows as no shutdown logged in All Restarts by Host.
| Single Value | |
Out-of-memory errors per hour, split by impact: node crashed, actor system, or query cancelled with the node still up. OOM Errors by Host shows which hosts. Empty is normal: no out-of-memory errors.
| Time Chart | |
Seconds of parsing per second across the cluster, all parsers and nodes added together. 10 means about ten threads were busy parsing the whole time. A rise after the upgrade without more ingest means parsing got more expensive. Parsing happens before an event enters the ingest queue, so it shows in HTTP Ingest Response Time, not in Event Latency.
| Time Chart | |
Seconds for the slowest event in each ingest partition, for the worst partition and the median partition. Worst high and median low means a few hot partitions; both rising together means a cluster-wide delay.
| Time Chart | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Share of processed data that historical queries took from cached query results instead of processing again, per hour. LogScale treats this as an approximation.
| Time Chart | |
Share of historical query cost spent on queries that were cancelled, per hour.
| Time Chart | |
Query cost per user or token and repository, counted while their queries were among the cluster's 5 most expensive, with how many different queries that was. One user, token or repository far above the rest is a noisy neighbour: its queries take the CPU and memory other searches need. Empty when no query was expensive enough to be reported.
| Table | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Queries the query coordinator held back for a later start, per hour, split by the logged reason. Empty is normal: no queries were queued.
| Time Chart | |
Time historical queries spent queued in the query coordinator before they started, per hour.
| Time Chart | |
Running queries that LogScale restarted, grouped by the first word of the logged restart reason. Empty is normal: no queries were restarted.
| Time Chart | |
Runtime of completed, not cancelled historical queries per hour. Does not include time spent queued before the query started. Compare with the days before the upgrade.
| Time Chart | |
Completed historical queries grouped by query text, slowest median runtime first, with the slowest run. Inspect Query opens that run in the Search Performance Overview dashboard (LogScale 1.207 or later).
| Table | |
Logged requests per status class. rejected means the request was turned away before a response status was set. Some successful requests between nodes are not logged.
| Time Chart | |
Segments written by digest, mini-segments merged into a full segment, and segments marked deleted, per hour. In a healthy cluster the three lines move together. Created staying above merged means mini-segments pile up; merged staying above deleted means replaced mini-segments are not removed. Deletes include retention, not only replaced mini-segments.
| Time Chart | |
Segments downloaded from bucket storage and segments queued for local deletion per hour, cluster-wide. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and only the delete line shows.
| Time Chart | |
Nodes whose queue of segments to copy from other nodes was at or near its limit of 10,000 in the last 15 minutes. A full queue means the node cannot keep up with node-to-node replication. Clusters without bucket storage depend most on this queue. Green: no node. Red: one or more. Per-node charts: Segment Prefetch & Download Health.
| Single Value | |
Share of the segment fetch queue additions in the last hour that were retries of earlier requests, on the node with the highest share. Retries happen when a copy of the segment is already being fetched, the fetch threads are all busy, or the fetch failed. Healthy clusters stay near 0. Green: under 25%. Yellow: 25% to 50%. Red: 50% or more, meaning retries outnumber new requests. Per-node charts: Segment Prefetch & Download Health.
| Single Value | |
Time from the last mini-segment of a group being written until the group is merged, 99th percentile, for the worst node and the median node. Merge latency rises when mini-segments wait for replication before they can be merged, even if each merge itself is fast.
| Time Chart | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Size of data with fewer copies than the replication factor, and size of data that no live node can serve (missing includes both lost segments and segments whose only holders are down). Works with and without bucket storage. Briefly above 0 during restarts is normal; a value that stays up or grows means replication is not keeping up. The exact number of missing segments is the missingSegments field of the same log line.
| Time Chart | |
Queries LogScale cancelled because they made no progress for 30 minutes. A query that waits more than 15 seconds for its turn is normal on a busy cluster; one that waits 30 minutes is given up and the user gets a warning that the query was cancelled. Green: none. Red: one or more. The charts in this section show how long searches wait and how much of that is waiting for bucket storage.
| Single Value | |
Per node: starved-search log lines, the longest time a search went without progress, and the longest total time a search spent waiting for segment downloads from bucket storage (minutes). Empty is normal: no search stalled.
| Table | |
Log lines for searches with no progress for more than 15 seconds, per node. A stalled search is logged repeatedly; after 30 minutes it is cancelled. Empty is normal: no search stalled.
| Time Chart | |
Request URIs with the most 5xx responses, with the time each was last seen. Routine 504 replies to cache-only query requests between nodes are excluded. Empty means no 5xx responses.
| Table | |
Most frequent ERROR log lines in the last 6h, grouped by logger, start of message and exception type. ago_sec is seconds since the line was last seen. Empty means no ERROR lines in the last 6h.
| Table | |
Queries that appeared in the cluster's top 5 by query cost, ranked by their highest cost in any 30-second window, with the user or token that ran them and the repository. Empty when no query was expensive enough to be reported.
| Table | |
ERROR log lines in the last 24 hours reporting a segment whose Kafka offsets are older than the oldest event Kafka still holds. Kafka deleted the events before the segment was safely replicated, so the cluster can no longer replay that data if the segment is lost. Green: none. Red: one or more.
| Single Value | |
ERROR lines per node reporting a segment whose Kafka offsets are older than the oldest event Kafka still holds. One node means that node is the problem; every node means a cluster-wide cause such as a short Kafka retention or stalled replication.
| Time Chart | |
WARN log lines per hour for the 10 loggers with the most WARN lines. A logger that appears or grows after the upgrade is a new or louder warning path. Compare with the days before the upgrade.
| Time Chart | |
How often per second the digest writer queue was full when adding work, summed over all nodes and averaged over a minute. Some overflows are normal on a busy cluster: the work waits briefly or goes to another writer. What matters is change: a rise compared with the days before the upgrade means digest is falling behind ingest.
| Time Chart |
Digest Overview
| Widget | Description | Type |
|---|---|---|
Number of writer threads currently active (ingest-writer-threads, field max). Expected ≈ DIGEST_EXECUTOR_CORES (default = CPU cores / 2). A drop means writer threads died or are permanently blocked on segment I/O or entity locks; zero means the node cannot digest at all.
| Time Chart | |
Active datasources per node (digest-active-datasources, field max). Each datasource generates periodic flush and cleanup packets that compete for writer-queue slots, so a node with disproportionately many datasources overflows first, even at low ingest.
| Time Chart | |
Time spent deserialising events and Kafka records during digest (name=/^digest-(event|record)-deserialization-ms/, field p99; excludes Kafka client decompression). If overall digest is slow but the other digest panels are not, the cost is here, large events, complex schemas, or CPU pressure.
| Time Chart | |
Time to apply an internal bulk to live / real-time queries (digest-live-latency, field p99). High here = real-time dashboards and streaming queries are the bottleneck. Correlates with GC pauses on the JVM dashboard.
| Time Chart | |
Bytes digested per CPU-second (digest-rate-per-cpu-second, field p99). Efficiency signal independent of load, a drop here at constant CPU means digest became less efficient (a regression), not just busier. Rising or flat is healthy.
| Time Chart | |
Time to write an internal bulk into segment files (digest-segment-latency, field p99). High here points to disk I/O, segment flush, or writer-queue pressure on the affected node. One node 10x its peers = its disk is failing or a background job is eating IOPS.
| Time Chart | |
Events per second each node publishes to the Kafka ingest queue (requests per second × mean events per request, from kafka-request-events). This is the input to digest.
| Time Chart | |
Log lines where a node found that other nodes had taken over its digest partitions because they considered it dead, per node. The node then shuts itself down. Empty is normal.
| Time Chart | |
Events per second written into the local ingest queue per repository (written-events per-repo variant, m5 = 5-min rate). Repo name is extracted from the metric name via a named-capture regex. A single repo dominating = that tenant is driving load; use to identify noisy repos before they cause writer overflow.
| Time Chart | |
Highest lag of LogScale's own ingest reader on any ingest-topic partition (records-lag-max from the ingest consumers' Kafka client metrics), cluster-wide max per 5-min bucket. Per-partition detail is on the Kafka Consumer Lag & Broker Health dashboard.
| Time Chart | |
Number of events between the newest offset in each Kafka ingest partition and the oldest offset the cluster may still need to replay (ingest-queue- lowest-offset-lag). Events below that point are safe to delete from Kafka. A line that keeps rising means the safe-to-delete point is not moving: segments are not being replicated, uploaded or merged, and Kafka retention can run out before the data is safe. A sawtooth that drops back is healthy. One node reports this metric for the whole cluster, about once a minute; if the widget is empty, check that the view includes the metrics repository.
| Time Chart | |
Ownership handoffs of digest partitions between nodes, Changing digest partition leadership or Topology changed log lines. 1 to 5 after a restart is normal. A storm (> 20/hour) means nodes are flapping or the coordinator is looping.
| Time Chart | |
CPU time spent by the segment-merge thread group per node (stat_tasks, threadGroup=/segment-merge/, ticks = utime + stime). Rising CPU together with a growing merge queue = the merger is working but cannot keep up (CPU-bound); rising queue with flat CPU = merges are blocked on disk I/O.
| Time Chart | |
Depth of the segment-merge thread pool queue per node (thread-pool-queue-size/segment-merge, field value). Segment merge is the background job that consolidates small segment files into larger ones for query efficiency. A growing queue means merges are falling behind, which delays segment availability for the local cache and inflates disk usage. Correlate with the merge-CPU chart on the right.
| Time Chart | |
Size of data packets enqueued to writer threads (ingest-writer-bulksize, field p99). A single oversized bulk can block a writer for a long time while it serialises and saves, starving other queue slots. Correlate spikes with the overflow rate on the left.
| Time Chart | |
Writer queue fill rate: enqueue attempts per second minus accepted jobs per second, per minute. About 0 = draining as fast as it fills; above 0 = the queue is backing up; below 0 = draining a backlog.
| Time Chart | |
Which nodes are overflowing the digest writer queue and when (ingest-writer-queue-full, field m1 = 1-min rate). Each increment = a consumer hit a full writer queue while pushing work to the digest writers. Heat-map so one row per node makes hot nodes obvious as a bright band.
| Heat Map | |
Events added to the segment blocks being built, 1-min rate summed per node (written-events-after-queue meter). Zero on a node while its ingest-reader lag grows suggests that node's digest is not progressing.
| Time Chart |
Errors Enhanced
| Widget | Description | Type |
|---|---|---|
Action-delivery failures from any trigger source (scheduled search, filter alert, aggregate alert, legacy alert), split by repository. Errors here mean the match was found but the notification did not reach the destination (webhook down, expired token, mail relay etc.).
| Time Chart | |
Warning-severity action events, split by repository, typical: Unknown action (a trigger references an action that no longer exists), retries in progress.
| Time Chart | |
AggregateAlert engine errors, split by repository. AggregateAlerts evaluate on a time window aggregate; errors here typically mean the underlying query failed or the alert's window overran.
| Time Chart | |
Warning-severity events from the AggregateAlert engine, split by repository. Read alongside the Errors chart on the left to spot rising trouble before an outage.
| Time Chart | |
Failed segment downloads from bucket storage whose error mentions unexpected message type: LastHttpContent, per node. Empty is normal: no such download errors. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and this widget is empty.
| Time Chart | |
Node counts by LogScale version (name=cluster_versions), hourly buckets. Steady state = a single version. Two versions appearing simultaneously = a rolling upgrade is in progress. Correlate error timing above with a version transition here to identify a regression window.
| Time Chart | |
For classes matching the ?class_regex parameter, this widget groups by class + the first 120 non-newline non-period characters of the message. That exposes patterns hidden by variable content (per-request IDs, timestamps, node names), e.g. Failed to connect to broker … over many peers becomes one row. Adjust the class_regex parameter to focus on a specific error family.
| Table | |
All ERROR log lines per hour, cluster-wide, with a trend line. Errors Over Time by Class shows which classes make up the count; Top ERROR Messages by Logger shows the messages.
| Time Chart | |
Total error events per week over the last 30 days. Use this to spot the quiet regression, a version deployed three weeks ago that gradually leaked errors over subsequent weeks. A single high week that then drops is usually a spike that got fixed; a rising trend line is the real problem.
| Time Chart | |
Error-level log volume per bucket, split by the emitting Java class. Baseline shape reveals which components error steadily (routine noise) vs which are new sources of trouble. Steep climbs are the primary anomaly signal.
| Time Chart | |
FilterAlert engine errors, split by repository. FilterAlerts fire on individual matching events; errors here mean the alert evaluation itself failed (bad query, permission, resource exhaustion), NOT that an alert triggered.
| Time Chart | |
Warning-severity events from the FilterAlert engine, split by repository. Typical warnings, running behind, catch-up limit reached, missing-data condition detected.
| Time Chart | |
The pre-filter/aggregate legacy alert engine, still used by alerts created before the split, split by repository. Errors here won't move if you've migrated all alerts to FilterAlert / AggregateAlert.
| Time Chart | |
Warning-severity events from the legacy alert engine, split by repository.
| Time Chart | |
Full log volume broken out by loglevel (INFO / WARN / ERROR / DEBUG). Info is the baseline noise; the ratio of Error to Info tells you whether error volume is rising because of new failures or because overall log volume is up (deploy, config change, backfill). Percentile series coloured red → green so severity is visually obvious.
| Time Chart | |
Log lines from query functions that mention an exception but are logged at WARN or INFO rather than ERROR. Empty is normal; repeated entries indicate recurring issues worth investigating.
| Table | |
Exceptions from LogScale query functions (package com.humio.query.functions) that WERE promoted to error-level logs. These are the failures users see back on their query job, bad arguments, unsupported operations, downstream panics.
| Table | |
Percentiles of external query API request latency (/api/v1/queryjobs/). Rising p99 alongside error volume signals users are seeing timeouts; rising p95 with flat p99 = broader slowdown. Percentile series coloured green → red.
| Time Chart | |
QuerySessions query was restarted log lines grouped by reason. A restart discards the CPU already spent, so a sustained non-zero rate here inflates query cost and points at planner instability or segment-availability churn.
| Time Chart | |
Error-severity activity events for scheduled searches, split by repository (dataspace). Concentration on one repo = a repo- specific query or permission issue; spread across many repos = cluster-wide problem.
| Time Chart | |
Warning-severity counterpart to the errors chart on the left, split by repository. Rising warnings on a repo before errors appear = leading indicator of impending failure.
| Time Chart | |
Segment fetching log events. They cover several distinct message patterns (fetch start, postpone-not-in-bucket, could-not-fetch, delete-then-fetch); this widget classifies each event by regex-matching the message text and buckets them. High fetch failure volume alongside slow query latency = the local segment cache is undersized or bucket storage is degraded. Correlate against Query Request Timings in Section 3.
| Time Chart | |
stat_tasks probe grouped by threadGroup, showing CPU ticks per group. A single group dominating (Ingest, MapReduce, SegmentMerge, GC) tells you WHERE cluster CPU is going, and by exclusion, what error class in Section 1 is likely to be resource-bound rather than logic-bound.
| Time Chart | |
Log-based count of TimeSlicerActor restart events grouped by the reason field it emits. TimeSlicer manages time-based partitioning of running queries; restarts are usually driven by state-refresh conflicts, coordinator changes, or resource pressure.
| Time Chart | |
Most frequent ERROR log lines in the last 6h, grouped by logger, start of message and exception type. ago_sec is seconds since the line was last seen. Empty means no ERROR lines in the last 6h.
| Table | |
Most frequent ERROR log lines in the last 6h, grouped by logger, start of message and exception type. ago_sec is seconds since the line was last seen. Unlike Top Error Classes, two different messages from the same class are shown separately. Empty means no ERROR lines in the last 6h.
| Table | |
Bar chart of top-N error classes in the window, ranked by count. Complements the top-error-classes table in Section 1, same signal, visual form. Long-tailed distribution = many small errors; single dominant bar = one component is on fire.
| Bar Chart | |
Same shape as top-error-classes but at warn level. Classes here that ALSO appear in top-error-classes are the immediate escalation candidates, same component is warning and erroring.
| Table | |
Warning-level log volume per bucket, split by class. Rising warnings ahead of errors is a leading indicator, the system is signalling soft-failure before hard failure. If a class shows up here and later in the Errors chart, that transition is the story of the incident.
| Time Chart |
External Feeds (SQS and FDR)
| Widget | Description | Type |
|---|---|---|
FDR activity events that mention NoMoreElementsNeeded, per feed. NoMoreElementsNeeded is the stream-cancellation signal raised when a downstream stage stops consuming; it can occur during normal cancellation.
| Time Chart | |
Non-zero means two or more feeds share the same name across different orgs, and the join-based widgets on this dashboard will show wrong data. Investigate before trusting the other panels.
| Single Value | |
Number of SQS messages currently in-flight (received but not yet deleted), per feed. Extracts fdrFeedId from the metric name and joins to the activity log for the human-readable feed name. Rising = the consumer is holding messages longer than usual.
| Time Chart | |
End-to-end delay from producer timestamp inside the SQS message body to LogScale activity-log ingest, seconds. Uses max so a single stuck message with many fast ones does not hide in an average. Six segments contribute, see the trace table below to attribute a spike.
| Time Chart | |
One row per feed ingest activity event with a total delay over 60 s, split into time on the queue, processing (download, decompress, ingest) and activity-log ingest, with the largest part named.
| Table | |
The same SQS message ingested by more than one host. LogScale leaves a message on the queue after an error so it is retried later, and SQS redelivers messages whose visibility timeout runs out; either can lead to the message being ingested again. Column timeBetweenStarting shows how far apart the two receives landed. Cross-check with the trace table.
| Table | |
Rate of SQS messages that completed the full ingest cycle and were deleted from the queue, per feed. Filters on log message string SQS message ingested and deleted, fragile to LogScale source changes; if the value goes flat-zero, check the emit site.
| Time Chart | |
Depth of the SQS queue per feed. If in-flight is normal but on-queue climbs, the producer is faster than the consumer. Cross-check with the delay widget below.
| Time Chart |
Ingest Overview
| Widget | Description | Type |
|---|---|---|
Effective download speed = partLength ÷ (LogScale ingest time − source eventTime), in bytes/second. Low value = either slow source, long queue sit, or slow download. Min/avg/max/percentile for the tail.
| Time Chart | |
Latest overall event-latency p99 snapshot (cluster max, last 1h) in seconds. Colour thresholds, green under 3s, amber 3-6s, red above 6s. Matches the event-latency-critical alert threshold.
| Single Value | |
Highest p99 event latency of any digest node in the time range, in seconds: time from insert into the ingest queue until events are in a segment block and live queries are updated. The line at 3 s is the alert threshold.
| Time Chart | |
Same backpressure signal (PollCoordinator pressure field) broken out per feedId, for S3/SQS-based feeds the pressure value is the number of messages on the queue. Identifies which specific feed is behind.
| Time Chart | |
Ingest-feed backpressure per node (PollCoordinator, pressure field). Backlog depth, N = LogScale expects N more non-empty polls before the feed drains. Sustained climb = a feed's consumer is falling behind (above 1 the session is demoted to the low-priority poll queue).
| Time Chart | |
Ingest is controlled by a quota, increased by the rate value at a configurable interval (default 2s). When quota reaches 0, ingest stops. This shows how the governor's brake system is currently configured and consumed.
| Time Chart | |
Ingest request rate broken down by API kind, HEC (Splunk compat), elastic_bulk, humio-structured / -unstructured, logplex, raw. Kind is derived from the URI. Useful for planning parser capacity and for spotting when a new integration goes live.
| Time Chart | |
Rate of ingest-API requests that returned an error status code (≥ 400). 4xx = client-side (auth, payload validation, rate limit); 5xx = server-side (planner crash, timeout, backpressure). Any sustained non-zero series is a customer-visible ingest failure.
| Time Chart | |
Rate of external ingest API requests per second, /api/v1/ingest/* and /services/collector (HEC compatibility). This is the CLIENT- SIDE arrival rate that feeds the parser and Kafka producer. A sudden spike here often precedes event-latency growth by seconds.
| Time Chart | |
Seconds the cluster takes to answer ingest requests from log shippers: median and the slowest 1%. Only external requests are included. This is the delay shippers see before an event is accepted, which Event Latency does not include. Rising here while Event Latency is flat points at the ingest front end, including parsing. Compare with the days before the upgrade.
| Time Chart | |
Governor throttles ingest feeds when LogScale becomes overloaded. Signal = Kafka partition with the highest ingest delay. Aim, keep ingestDelay between the Low and High reference series (both also plotted here), preferably below Low. Ingest STOPS from feeds when delay exceeds High.
| Time Chart | |
Uncompressed bytes successfully processed per ingest feed. Sourced from IngestFeed activity logs, the sum of partLength fields on Successfully processed part events.
| Time Chart | |
Error messages grouped by dataspace, feed name, ingest feed id, and message text. Each row is one distinct failure mode for one feed. Address the highest-count row first.
| Table | |
Latency from source-side eventTime to LogScale ingest, in milliseconds. Min/avg/max/percentile so a stuck long tail shows even when the average looks fine.
| Time Chart | |
Per-feed count of info, warning, and error messages from IngestFeed activity. First triage table, feeds with high error or warning counts should be investigated in the errors and warnings tables below.
| Table | |
Warning messages grouped by dataspace, feed name, ingest feed id, and message text. Rising warning volume without matching errors = a soft-failure mode that will eventually escalate; investigate before it does.
| Table | |
Ingest throughput per node, sorted by load. A large disparity means uneven partition assignment, the busiest node caps cluster throughput and is the first to build ingest-queue latency.
| Table | |
Event-latency broken down per Kafka partition. Same source as the cluster-wide p99 above, but grouped by partition, a single high partition line = that partition's leader is slow. Shows both mean (baseline behaviour) and max (worst-case tail) per partition. Partition number captured from the metric name via named-capture.
| Time Chart | |
Seconds an event waits in the ingest queue, from after parsing until a digest node has read it. This is only the queue part of Event Latency: if both rise together, the delay is in the queue or in reading from it. Shown for the slowest 1% of events, for the worst node and the median node. Worst high and median low means one node is the problem; both rising together means the whole cluster. Compare with the days before the upgrade.
| Time Chart | |
SQS messageIds that appear in more than one Successfully deleted SQS messages event, LogScale received and deleted the same message more than once, so its events can be ingested twice. Polls cancelled during a shutdown are one way this happens.
| Table | |
Bytes flowing through parsers, bytes-in (raw incoming) vs bytes-out (parsed and emitted). Divergence between the two indicates parsers are dropping or transforming heavily.
| Time Chart | |
Time spent parsing incoming events (ingest-parsing, field p99). Excluded from event-latency but bounded by disk I/O and CPU. A spike here with steady queue latency means parsers are the bottleneck, complex regex, large events, or a mis-configured parser.
| Time Chart | |
Count of SQS messages successfully deleted after processing, per feed. Tracks Successfully deleted SQS messages IngestFeed activity events.
| Time Chart | |
Feeds ranked by average backpressure (backlog depth), worst first. Top rows are the data sources whose consumers are furthest behind and most likely driving ingest delay.
| Table |
JVM & Node Stability
| Widget | Description | Type |
|---|---|---|
| Node (vhost) filters every per-node chart to one node, leave at * for the whole cluster. Heap warning threshold sets the cutoff used by the hosts over threshold tile and the critical-hosts table. | parameterPanel | |
Direct off-heap memory (name=direct-memory-allocated). NIO buffers, mmap'd files, Netty pools. NOT GC-managed, growth without matching heap growth is a native leak. The OS kills the process when RSS exceeds the cgroup limit even if heap is fine.
| Time Chart | |
Each band is a node that could not reach another node over HTTP: left is the node doing the check, right is the node it could not reach. Band width is the number of failed checks. One node on the right reached by many bands is likely down or overloaded; one node on the left failing to reach many points at that node's own network. Empty is normal: every check succeeded.
| Sankey | |
Share of time spent in garbage collection, all collectors added together, for the node with the most and for the median node. 5 means 5% of the time was spent collecting. Worst high and median low means one node is the problem; both rising together means the whole cluster. A steady rise means the collector is doing more work to reclaim less.
| Time Chart | |
CPU ticks (utime + stime) of garbage-collector thread groups, per collector. Compare with GC time to see whether collection work is rising.
| Time Chart | |
Nodes whose heap stayed above the selected warning threshold even after garbage collection in the last hour, highest first. The heap figure includes garbage that has not been collected yet, so a busy node normally reads close to 100%. The lowest reading in a period is what is left after collection. These nodes have the least room left. Empty is normal.
| Table | |
How full the heap stays after garbage collection across all nodes, per 15 minutes: the median node, and the nodes at the 90th and 99th percentile. The heap figure includes garbage that has not been collected yet, so a busy node normally reads close to 100%. The lowest reading in a period is what is left after collection. All three rising together means the whole cluster is holding more memory.
| Time Chart | |
Nodes whose heap stayed above the selected warning threshold even after garbage collection, in the last hour. The heap figure includes garbage that has not been collected yet, so a busy node normally reads close to 100%. The lowest reading in a period is what is left after collection. Green: none. Red: one or more; more than one suggests pressure across the cluster rather than one node.
| Single Value | |
Completed runs per internal job name, from Job completed log lines. Compare jobs with each other and over time; a job running far more often than usual stands out.
| Time Chart | |
How full the heap stays after garbage collection, for the worst node and the median node, per 15 minutes. The heap figure includes garbage that has not been collected yet, so a busy node normally reads close to 100%. The lowest reading in a period is what is left after collection. A floor that climbs over hours means memory is not being freed; a high but flat floor is normal. Worst high and median low means one node is the problem.
| Time Chart | |
jvm-hiccup-latency p99 as a heat map, one row per node; colour is pause length. Ignores the node parameter so all nodes can be compared.
| Heat Map | |
How long the JVM could not run a thread, for the slowest 1% of measurements, on the worst node and the median node. A background thread sleeps 10 ms in a loop; any extra time it takes to wake up is a pause: garbage collection, the JVM stopping all threads, or the node running out of CPU. The 250 ms line is a reference, not a hard requirement; the right level can differ per cluster. Worst high and median low means one node; both rising means pressure across the cluster. If pauses rise while GC time next to it does not, the cause is CPU rather than garbage collection.
| Time Chart | |
Nodes that should be present but aren't heartbeating (name=missing-cluster-nodes). Non-zero = a node crashed / OOMed (check Heap), a network partition, or a slow restart. Sustained missing nodes trigger partition reassignment storms (see Digest dashboard).
| Time Chart | |
Current count of nodes not responding to heartbeats (name=missing-cluster-nodes). Non-zero = a node is down or partitioned, often the aftermath of an OOM. Cross-check the Heap section for the cause. Colour, green at 0, red at ≥1.
| Single Value | |
Nodes that shut themselves down because of an internal failure, per 10 minutes, split by cause: out of memory, rejected transaction, digest partitions taken over by other nodes, stuck ingest, stuck dispatcher threads, or a Kafka epoch change. other is any other cause; the table below shows its full text. A graceful shutdown or a killed process is not counted here. Empty is normal: no node failed. Only shutdowns LogScale starts itself. Kills such as a forced stop or SIGKILL, and certain crashes, do not log this; they show as unclean in All Restarts by Host and in Unclean Shutdowns (Segment Prefetch & Download Health).
| Time Chart | |
Failure shutdowns per host with the cause and its full text, how many, and the last one. Use the full text for rows marked other. Empty is normal: no node failed. Only shutdowns LogScale starts itself. Kills such as a forced stop or SIGKILL, and certain crashes, do not log this; they show as unclean in All Restarts by Host and in Unclean Shutdowns (Segment Prefetch & Download Health).
| Table | |
Every node start with the host, restart type, cause and the LogScale version the node started with. The restart type comes from the last line the node logged before starting: a graceful shutdown, a failure shutdown, an out-of-memory error, or nothing. no shutdown logged means the process stopped without logging anything, for example a kill or a crash. graceful means a clean shutdown started; it does not prove it finished. A graceful shutdown just before the time range can make a normal restart show as no shutdown logged. Needs LogScale 1.174 or later. Empty means no node started.
| Table | |
JVM non-heap memory (name=jvm-NON-heap-usage). Metaspace, code cache, compressed class space. Slow unbounded growth points at a classloader or metaspace leak. Usually stable, a steady climb is the signal.
| Time Chart | |
Log lines where a scheduled task on the default Pekko dispatcher was delayed past the operator's liveness threshold, usually a GC pause blocking the actor thread pool. Great correlator for the hiccup chart above, if dispatcher delay events cluster around hiccup spikes, GC caused the stall; if they appear alone, look at CPU throttling or thread starvation instead.
| Time Chart | |
Queries whose intermediate results grew beyond the size LogScale allows to send between nodes, with how often it happened, the most memory the query had allocated, and when. These queries put heavy pressure on the heap. Empty is normal.
| Table | |
Resident memory of the LogScale process per node (VmRSS from /proc/<pid>/status): heap, off-heap, code, native buffers and mapped pages in RAM. This is the figure the operating system out-of-memory killer uses.
| Time Chart | |
How full the heap stays after garbage collection, in the last 15 minutes, on the node where it is highest. The heap figure includes garbage that has not been collected yet, so a busy node normally reads close to 100%. The lowest reading in a period is what is left after collection. Near 100 means little room is left even after collection.
| gauge |
Kafka Consumer Lag & Broker Health
| Widget | Description | Type |
|---|---|---|
Per Kafka consumer group with committed offsets on the ingest topic: latest partition offset minus the committed offset, cluster maximum per 5 minutes. Empty when no group has committed offsets. A group whose lag exceeds topic retention misses records.
| Time Chart | |
Log lines where a node found that other nodes had taken over its digest partitions because they considered it dead, per node. The node then shuts itself down. Empty is normal.
| Time Chart | |
Warning log emitted when a publish to the global-events coordination topic takes longer than LogScale's internal recommendation. Leading indicator before publishes start timing out. Sustained non-zero on one node = that node's Kafka client is degraded; cluster-wide = the broker is slow.
| Time Chart | |
Milliseconds by which the freshest ingested event on each node lags wall-clock time (name=min-ingest-timestamp, field max). This is the user-visible consequence of Kafka lag, higher = search results contain fresher events later. Rises in step with consumer lag when LogScale digest is slower than the produce side.
| Time Chart | |
Meter written-events per node, events written into LogScale's local ingest queue before Kafka publish. A spike here correlating with lag growth means the cluster is overwhelmed by upstream volume, not internally broken. If the spike is finite (a batch job), estimate time-to-recover and monitor.
| Time Chart | |
LogScale's OWN internal ingest-consumer lag, per partition, over time. Uses KafkaMetricsLogging records-lag from the Kafka client library. This is the digest side seeing how many records am I behind on this partition right now. A single high line = one partition's leader node is slow to consume; cluster-wide climb = digest is behind overall. Correlate with Digest Throughput
| Time Chart | |
ERROR log lines from the Kafka ingest queue reader, ranked by node (top 15). Empty is normal: no errors.
| Bar Chart | |
Time for LogScale to publish a message to the Kafka ingest queue, p75 per node (name=kafka-ingestqueue-put, field p75). High = Kafka slow to acknowledge writes (broker disk I/O or network). Above the 500 ms SLO line, internal buffers fill and ingest backs up at the HTTP layer.
| Time Chart | |
Rate of publish timeouts on the LogScale → Kafka path (name=ingest-kafka-timeouts, field m1 = 1-min rate, events per second). Any sustained non-zero value means LogScale is trying to publish faster than Kafka can accept, or the broker is unhealthy. Correlate with Put Latency p99 (left) and Request-Queue Fill % (top right).
| Time Chart | |
Distinct partitions with at least one out-of-sync replica per minute, per topic. Occasional single-partition blips (< 1 min) = normal ISR shrink. Sustained > 0 for multiple minutes = a broker is struggling. Climbing or affecting all partitions = broker down or severe disk/network issue.
| Time Chart | |
Topics where at least one replica is behind the partition leader (topic_num_out_of_sync_replicas > 0). Out-of-sync replicas mean a broker's disk, network, or CPU is degraded. If the leader fails while replicas are out of sync, data loss occurs. Fix Kafka first, consumer lag is a symptom.
| Time Chart | |
Same source as put p75 (kafka-ingestqueue-put) but the p99 tail. If this rises alongside lag, the bottleneck is Kafka broker/network slow acks , not the consumer/digest side. Compare against the p75 chart in this section; p99 rising while p75 stays flat = a small number of very slow acks.
| Time Chart | |
Latency per KB on the produce path: p75 put latency (ms) divided by p75 request size (KB). A single normalised number that separates Kafka is slow (higher ms/KB) from we are sending more data (higher bytes at constant ms/KB). Rising while raw put latency is also rising = broker is legitimately overloaded, not just seeing more data.
| Time Chart | |
Compressed bytes written to Kafka per ingest-queue request, p75 per node (name=kafka-request-bytes, field p75). Large sizes indicate aggressive producer batching. If size spikes correlate with lag growth, brokers may be saturating their network. Normal 50-500 KB; warning > 1 MB sustained.
| Time Chart | |
Events written to Kafka per ingest-queue request, p75 per node (name=kafka-request-events, field p75). Read alongside request size (left): bytes high with events flat = large batches; both high = genuine volume increase. Separates a batching change from a real load change.
| Time Chart | |
Percentage of the outbound queue used for tasks waiting to send to the Kafka ingest queue (name=kafka-request-queue-fill-percentage). This is the buffer that fills when put latency rises; as it approaches 100% backpressure reaches the HTTP ingest layer and clients start seeing rejects.
| Time Chart | |
Per-partition table: which broker leads the partition, its replica broker set, its in-sync-replica set, and whether it's currently in sync. Sorted by (topic, partition). Compares directly with kafka-topics.sh --describe output. Filter for a specific topic by clicking the topic_name column header when reading.
| Table | |
Count of partitions where topic_is_in_sync=false, per topic per bucket. Each is a single point of failure, if the leader dies, that partition is unavailable until a replica catches up. Sustained URPs after a broker restart usually mean the broker disk is too slow to catch up.
| Time Chart | |
Ranks topics by the number of distinct partitions reporting out-of-sync replicas. Concentration on the humio-ingest topic directly explains consumer lag; concentration on other internal topics still signals a degraded broker. Empty table = no ISR issues in the window.
| Table | |
Distinct partitions with out-of-sync replicas seen in the window. 0 = all ISR healthy, Kafka is not the problem. Non-zero = broker-side replication is behind. Colour, green at 0, red at ≥1.
| Single Value | |
Count of distinct (topic, partition) pairs each broker currently leads (topic_leader field). Healthy Kafka spreads leadership roughly evenly, one broker leading a disproportionate share means recent failovers concentrated on it, and its network/CPU is at higher risk. A preferred leader election or partition reassignment on the Kafka side redistributes these.
| Bar Chart | |
Partition coordinator events: Changing digest partition leadership... (reassignment) and Topology changed impact=... (membership change), per node. 1-5 after a restart is normal; a storm (> 20/hr) means nodes are flapping or the coordinator is looping, partitions bounce owners, causing duplicate processing and lag during handoff gaps.
| Time Chart | |
Longest time events waited in the ingest queue per partition before digest picked them up, in seconds. One high partition points at that partition; all partitions rising means digest is behind overall.
| Time Chart | |
Max kafka-request-queue-fill-percentage across nodes over the last 15 min, the busiest node's outbound-to-Kafka queue utilisation. Healthy = low. Approaching 100% = tasks waiting to publish are backing up (producer-side pressure).
| gauge | |
Same shape as humio-ingest gauge (left) but for the global-events coordination topic. LogScale nodes use global-events to broadcast cluster state changes (bucket manifest, digest partition ownership); this topic typically has 1 partition, so healthy = 1. A 0 here means cluster coordination is stalled.
| Single Value | |
Count of distinct humio-ingest partitions currently reporting topic_is_in_sync=true. A steady number equal to the topic's configured partition count is healthy. A drop = that many partitions have out-of-sync replicas , one or more brokers is behind. humio-ingest is LogScale's primary event pipeline topic; any drop here directly threatens ingest durability.
| Single Value | |
Distinct topics reporting at least one partition with topic_is_in_sync=false in the window. A complement to the Partitions Out-of-Sync scalar in Broker Health, that counts partitions, this counts topics. Concentration on one topic = topic-specific replication issue; multiple topics affected = broker-wide problem. Colour, green at 0, red at ≥1.
| Single Value | |
Ranks consumer-group offset-lag series on the ingest topic by peak value in the window. Each series name carries the partition and consumer group, so this shows which group and which partitions are furthest behind. Empty when no consumer group has committed offsets on the ingest topic.
| Table | |
Meter written-events-after-queue (field m1, 1-min rate) summed per node, events flushed to segment blocks per second after being read from Kafka. If external lag is growing AND this is ~0, digest is stuck (GC / IO / merge; check JVM & Node Stability). If lag grows while this stays high, the node works but arrival rate exceeds capacity.
| Time Chart |
Kubernetes Operator Reconcile Health & CRD Lifecycle
| Widget | Description | Type |
|---|---|---|
Count of nodeCount updates where the operator had to clamp the requested value against its allowed min/max (clamped=true). Non-zero means the HPA (or a spec change) requested a scale value the operator refused to apply, the cluster scaled to the boundary, not the requested value. Adjust operator min/max or investigate the HPA target.
| Time Chart | |
Count of log lines containing the conflict phrase, these surface when the operator writes a resource whose resourceVersion changed between get and update (status-update collision, another writer, leader-election race). A low steady rate is normal (controller-runtime retries transparently). A sudden storm can wedge reconciliation.
| Time Chart | |
Timechart of error-level log lines per 15-minute bucket. Baseline should be flat at 0 for a healthy operator. Steps up correlate with API-server outages, expired credentials, or a breaking change in a CR spec. Compare shape against the reconcile-volume chart above, errors climbing while reconciles hold steady means each reconcile is now failing.
| Time Chart | |
Count of log lines where the operator logger emitted at error level. Any non-zero value warrants inspection, controller-runtime only logs at error level when a reconcile returns an error or a finalizer fails. Colour, green at 0, red at ≥1.
| Single Value | |
Log lines matching Finalizer method returned error, the operator's cleanup logic failed on a deletion. When present, resources stay in Terminating state and cannot be removed until the finalizer succeeds or is manually stripped. Split by Request.Type so the CR kind is visible.
| Time Chart | |
Reconcile activity for HumioCluster CRs only, split by Request.Name (the LogScale cluster name). Baseline = flat moderate rate; a spike = a config change is being applied or the operator hit a requeue loop on that cluster.
| Time Chart | |
Log-line events where the operator writes one of the HumioCluster state constants (Running, Restarting, Upgrading, Pending, ConfigError). State pulled from the log message via a regex named-capture. A cluster stuck in Restarting or Upgrading for long periods = something is blocking rollout; ConfigError = the spec itself is invalid.
| Time Chart | |
Bar chart of HumioPackage CRs currently in FailedInstall or PartialInstall state, ranked by count of failure events. Empty = every package install succeeded. Top rows are the packages to investigate first, mismatch checksum, unreachable registry, or a corrupt zip.
| Bar Chart | |
HumioPackage state extraction from log messages, Installed, FailedInstall, PartialInstall, Unknown, NotFound, ConfigError. Installed is the healthy terminal state. FailedInstall / PartialInstall = a package upload or apply step failed; check the packageChecksum, package registry access, and the humio API for details.
| Time Chart | |
Current state per HumioPackage CR, taken as the last state value seen in the window. Colour-coded, green Installed, amber PartialInstall, red FailedInstall / ConfigError. Sort by last_seen to see the most recently updated packages first.
| Table | |
Top 20 individual custom resources (Request.Type + Namespace + Name) ranked by reconcile count. A single CR appearing with an order-of-magnitude higher reconcile count than its siblings is almost certainly stuck in a reconcile → requeue loop. Cross-check the recent-errors table for messages tagged with that Name.
| Table | |
Changes to node pool nodeCount by source: hpa (the HorizontalPodAutoscaler), spec (the HumioCluster resource was edited) or default (the operator applied a default).
| Time Chart | |
The productVersion tag on the operator pod that has emitted the most log events in the window. On a healthy single-version deployment this is unambiguous, one version. During a rolling upgrade the majority version is shown here; the distribution table on the left shows both, and the rolling-upgrade chart on the right shows when the transition happened.
| Single Value | |
Distinct operator pods that emitted at least one log line in the last 15 minutes. In a healthy deployment this equals the replica count of the operator Deployment. 0 = the operator is not logging (crashed, stuck in CrashLoopBackoff, or its log pipeline is broken). A drop = one or more replicas restarted.
| Single Value | |
One row per distinct operator productVersion currently emitting logs, with the pod count for that version. Version comes from the pod annotation kubernetes.annotations.productVersion set by the operator's Helm chart. Steady state, one row, N pods (matches Deployment replicas). During a rolling upgrade you will briefly see two rows; if both persist beyond the rollout window the upgrade is stuck.
| Table | |
Newest-first table of error events with the CR coordinates that triggered them. Correlate Reconcile.ID against reconciles-per-controller to find which reconcile invocation failed and what the operator was doing at the time.
| Table | |
Distinct Reconcile.ID count per bucket, split by Request.Type (controller / CR kind). Baseline shape = healthy operator; a drop-to-zero on one controller = that controller stopped reconciling (RBAC issue, informer error, or the CR type has no instances). A dominant spike on one controller = a rollout or a reconcile storm.
| Time Chart | |
Count of distinct kubernetes.annotations.productVersion values seen per 1-hour bucket. Steady state = 1 (single version deployed everywhere). > 1 for more than a few buckets = a rolling upgrade in progress; sustained > 1 = a stuck upgrade (some pods never reached the target version).
| Time Chart | |
Node pool sync failures, by pool and error type. A rising rate means the pool is not converging.
| Time Chart | |
Operator-emitted Warning event fired when a shadow node pool read is consecutively failing (staleness > threshold). This is one of only two K8s Events the operator emits and points directly to nodepool synchronisation lag. Split by pool.
| Time Chart | |
Same signal collapsed to the 24h total, ranked. HumioCluster and HumioPackage should dominate on a healthy cluster; a suddenly-hot controller for a rarely-used CR (HumioParser, HumioSavedQuery) often means someone is making rapid config changes or a reconcile loop is churning.
| Bar Chart | |
Top 15 distinct error-message texts. Message strings are truncated to the first 120 non-newline non-period characters with a regex named-capture so long stack-trace-style errors do not crowd the bars. One dominant bar = one systemic error; spread across many small bars = several independent failures.
| Bar Chart | |
Distinct Reconcile.ID values in the last 24 hours across all controllers. This is the throughput of the operator, how many reconcile invocations it has completed. A near-zero value with running pods = the reconcile loop is wedged (informer watch broken, RBAC denied, leader-election stuck).
| Single Value | |
The operator emits Unrecoverable error only on genuinely fatal conditions (token generation failure, cluster state that cannot progress). Empty is the healthy state. Any row here is operator-team-page material, the reconcile cannot succeed without human action.
| Table | |
Count of waiting for log messages, split by Request.Type. A steady rate is normal (the operator naturally requeues while pods roll, tokens generate, etc.). A rising rate with no accompanying success events = a dependency is genuinely stuck (pending image pull, blocked webhook, missing secret).
| Time Chart | |
Same shape as the error chart on the left but at warn level. Warnings are recoverable, status-update conflicts, waiting-for dependency, transient API-server hiccups. A steady non-zero baseline is often normal (retries) but a sharp climb is a leading indicator of impending errors.
| Time Chart | |
Count of warn-level log lines. Warnings do NOT mean the reconcile failed, they signal a soft condition the operator handled (retry-worthy conflicts, transient dependencies, expected wait states). Rising warning volume without errors = soft-failure mode that will eventually escalate.
| Single Value |
Query Performance Drilldown
| Widget | Description | Type |
|---|---|---|
Live query text with the longest median runtime, with the 50th and 99th percentile runtime (ms) and the fastest and slowest runs. Runtime of a live query is how long it ran before it ended. Empty is normal when no live query ended in the time range.
| Table | |
Total query cost in the time range, historical and live, excluding subqueries and cost reused from cached results. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Single Value | |
Query cost of queries that LogScale restarted while they ran, per organization and service. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge. Empty is normal when LogScale restarted no queries.
| Time Chart | |
Query cost of the top 10 users, by external username or email; queries run on behalf of the organization show its name. Excludes cost reused from cache. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Time Chart | |
Data processed by queries, split into data taken from cached query results (cached) and data processed again (uncached), in bytes. LogScale treats the cached amount as an approximation.
| Time Chart | |
Query text run most often, per service, with average processing time (ms) and memory allocated (bytes) per run, and the most recent run.
| Table | |
Total processing time of all queries in the time range, in milliseconds, summed over query phases. Includes time spent waiting.
| Single Value | |
Per service: queries, query cost, processing time (ms), memory allocated (bytes), data processed and taken from cache (bytes), share from cache, and share of total query cost. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Table | |
Per organization: queries, query cost, share of total query cost, share of data taken from cache, and share of the cluster's stored data that belongs to the organization. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Table | |
CPU cores per node role, hourly. The role is taken from the line each node logs when it starts; nodes that have not started in the last 30 days show as unknown.
| Time Chart | |
Uncompressed size of all segments in the cluster, in bytes, from the cluster segment statistics LogScale logs.
| Time Chart | |
Query cost per service. Service comes from what LogScale logs: alert, scheduled search, persisted aggregation, system query, the HTTP client polling the query (browser or API) and streaming export. Excludes subqueries and cost reused from cache. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Time Chart | |
Total memory allocated by all queries in the time range, in bytes, summed over query phases.
| Single Value | |
Time queries spent queued in the query coordinator before they started, in milliseconds, split by the reason logged when the query was queued. Empty is normal when no queries were queued.
| Time Chart | |
Cancelled queries per organization and service: queries, query cost, processing time (ms), memory allocated (bytes), share from cache, and share of total query cost. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge. Empty is normal when no query was cancelled.
| Table | |
Organizations added to or removed from this cluster by an organization transfer. Empty is normal on single-organization clusters.
| Time Chart | |
Share of data processed by queries that was taken from cached query results instead of processed again.
| Single Value | |
Historical query text with the slowest median runtime, with the 50th and 99th percentile runtime (ms) and the fastest and slowest runs. Runtime does not include time queued before the query started.
| Table | |
Query text with the highest average query cost per run, per service and searched time span, with the most recent run. Cost reused from cached results is not included. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Table | |
Per organization and service: queries, query cost, processing time (ms), memory allocated (bytes), share from cache, and share of total query cost. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Table | |
Historical query cost of cancelled queries compared with queries that were not cancelled. Includes cost reused from cached results. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Time Chart | |
Historical queries with at least one cancelled run, per repository, organization and query text: runs, query cost of cancelled and completed runs, and the costliest cancelled run. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge. Empty is normal when no historical query was cancelled.
| Table | |
Queries that were not started because they matched a blocked-query pattern, per organization. Empty is normal when no query matched a blocked-query pattern.
| Time Chart |
Query Performance Overview
| Widget | Description | Type |
|---|---|---|
Cluster-average CPU utilisation over the same 24h window, from stat_cpu (aggregate 'cpu' row): 100 - idle/(user+system+nice+idle)*100. Below 40% is the confirming reading, the cluster had ample compute while queries were cancelled as 'starved'. Above ~70% points to real compute saturation (see Search Queue), which would justify the cancellations.
| Single Value | |
Mean processedEvents across starvation log lines with low bucket-wait. This is the forward-progress signal emitted by the query scheduler (jobDetailsKVString). Non-zero means the query was consuming events at the moment it was declared 'no progress', the contradiction that marks the anomaly. Zero is consistent with genuine starvation.
| Single Value | |
query-time-spent-starved-waiting-for-bucket-storage-max (gauge → field value): max ms a query spent parked waiting on bucket-storage data before the metric was logged and reset. LOW here rules out real S3 starvation, so a 'starved' cancellation with low bucket-wait has no storage cause. HIGH here means the cancellation has a real storage cause.
| Time Chart | |
Per-node view of the monotonic query-cancelled-starved counter (a Codahale counter → field 'count'). Because it is cumulative, healthy clusters show flat lines; a step up marks a batch of cancellations. Correlate each step's timestamp against the progress charts and the no-cause-rate chart.
| Time Chart | |
Max chunksCompleted per node from the same starvation log lines. Segment chunks are the unit the bail-on-slow-chunks logic watches; completions rising while the watchdog reports 'no progress' is the direct contradiction. Cross-check against events processed, both should move together when the anomaly is present.
| Time Chart | |
p99 latency of query-state refresh responses between cluster nodes (name=query-coordinator-latency, makeHistogram → field p99, NO value field). High values mean the coordinator is slow to gather partial results, usually GC pauses or an overloaded node, distinct from raw per-query search cost.
| Single Value | |
Live cluster-average CPU% over the last 15m (stat_cpu aggregate row). Gauge so the on-call sees whether compute is idle right now: a low needle while cancellations are firing is the real-time confirmation that no CPU cause is present.
| gauge | |
Per-vhost CPU utilisation Hot rows = nodes under sustained CPU pressure.
| Heat Map | |
Max processedEvents per node from 'Starved searching' log lines. This counter only advances when the mapper consumes events, so a non-zero, rising line means the cancelled queries were doing work. When the anomaly is present this stays clearly above zero; during genuine starvation it flatlines at the pre-stall value.
| Time Chart | |
Peak jvm-heap-usage-percent (gauge → value) across nodes over the last 15m. A needle below 70% rules out GC pressure as a live contributor. If it is high, treat GC as a candidate cause before attributing cancellations to the watchdog path.
| gauge | |
External query HTTP requests per second per node, filtered to the /query and /queryjobs routes (#kind=requests, fields uri and internal). Use to spot a sudden query-load increase that explains search-queue saturation. internal=false excludes node-to-node internal calls.
| Time Chart | |
Rate of 'Starved searching' lines with msStarvedWaitingForBucketDownloads < 1000ms, cancellations that cannot be attributed to S3/bucket latency. A sustained signal here, combined with low CPU and non-zero progress above, is the anomaly. Windows with high bucket-wait are filtered out because those are legitimate starvation.
| Time Chart | |
Count of 'Starved searching' cancellation log lines where msStarvedWaitingForBucketDownloads < 1000ms, the query was NOT waiting on bucket storage. Near-zero is healthy. A rising number, together with low CPU/heap in the rule-out section, is the anomaly this section exists to surface. Cancellations with high bucket-wait are legitimate and excluded by this filter.
| Single Value | |
jvm-heap-usage-percent (gauge → field value). Moderate heap rules out stop-the-world GC pauses as the reason the watchdog missed progress updates. If heap is high and GC pauses are long (see JVM & Node Stability), investigate GC before treating the cancellations as having no cause.
| Time Chart | |
Live queries killed for excessive ingest delay, ranked by kill count with the last-killed timestamp. Search the queryID in the running-queries logs or the admin Queries page to recover the query string.
| Table | |
Total live queries killed for excessive ingest delay in 24h. Non-zero means ingest delay crossed the kill threshold and real-time dashboards were cancelled to free CPU for ingest catch-up.
| Single Value | |
Per-query table from 'Starved searching' lines. processedEvents/chunksCompleted = forward progress (non-zero = no-cause anomaly); waitMs = watchdog no-progress timer; timeSoFarMs = total query age; bucketWaitMs = msStarvedWaitingForBucketDownloads (low = no storage cause, high = legit S3 starvation). Sorted by processedEvents desc. internalQueryID is the internal query ID (starts with IQ-); externalQueryId (does not start with IQ-) is the most recent external ID for that query, and Inspect Query opens it in Query Performance Profiler.
| Table | |
Number of live (real-time) queries per node (name=livequery-count, makeGauge → field value). Rendered as a heatmap so a node holding disproportionately many live queries shows as a bright band without lines overlapping on a large cluster. Live queries consume CPU continuously and can starve batch queries and ingest.
| Heat Map | |
Current live (real-time) query count summed across nodes (name=livequery-count, makeGauge → field value). Each live query pins a worker continuously, so a rising count directly reduces the CPU available to batch queries and to digest.
| Single Value | |
Queries that appeared in the cluster's top 5 by memory allocated, with the most memory each allocated in total (GB), the user or token that ran it and the repository. Queries that allocate a lot of memory drive garbage collection and can lead to out-of-memory errors. Empty when no query was reported.
| Table | |
Median of the same secondsPerHourSearched metric as p90 (left). Compare against p90 to see whether latency degradation is broad (both rising together) or tail-only (p90 rising while p50 flat, noisy neighbour or cache miss on a small subset of queries). No SLO line, p50 is not the SLO target.
| Time Chart | |
p90 query runtime per hour of time range searched, in seconds, for historical queries over 100 ms that search less than 7 days. The line at 10 s is the SLO; the regression line shows the trend.
| Time Chart | |
Per-node rate of queries cancelled due to starvation (name=query-cancelled-starved, makeCounter → field count, monotonic). range() per bucket gives the increase so a counter reset on restart does not spike the chart. Any sustained non-zero line = the cluster is dropping query work it cannot complete.
| Time Chart | |
Queries cancelled by starvation in the last hour (name=query-cancelled-starved, makeCounter → field count, monotonic). Computed as max−min over the window so a node restart resetting the counter does not show as negative. Non-zero = the cluster cannot serve its query load at current capacity.
| Single Value | |
Query CPU cost attributed per repository. Uses the labelled histogram query-delta-cost (labelType repositoryLabel → the repo is encoded in the metric NAME, extracted here with the named-capture (?<repo>...), NOT a #repo tag; field mean since it is a histogram, no value field). Bar chart ranks the heaviest repos for chargeback or throttling decisions.
| Bar Chart | |
Top 15 errors on cancelled queries (first ~120 characters), for example a missing lookup file, a permission error or a query validation error.
| Bar Chart | |
Cluster-wide 30s-delta memory allocation on queries (name=query-delta-total-memory-allocation, makeHistogram → field mean, NO value field). Sustained high values indicate memory-heavy queries (large aggregations, joins) that drive GC pressure. Aggregated max-per-node then summed.
| Time Chart | |
Queries restarted mid-execution, grouped by reason, from the query was restarted log line (field reason). A restart discards the CPU already spent, so frequent restarts inflate cost with no user-visible result and point at planner instability or segment-availability churn.
| Time Chart | |
7-day trend of cluster-wide total query CPU cost (name=query-delta-total-cost, makeHistogram → field mean, NO value field). Aggregated max-per-node then summed, hourly. Use for capacity planning, a sustained upward trend means query load is outgrowing the cluster.
| Time Chart | |
Peak local-query-segments-queue (gauge → value) across nodes over the last 15m. A moderate current value confirms workers are available now. Spikes into saturation would redirect the investigation to capacity rather than the watchdog path.
| Single Value | |
Completed historical queries over 100 ms per 15 minutes: the number of queries behind the percentile panels. With few queries the percentiles are less reliable.
| Time Chart | |
Segments waiting to be searched, per node then summed (name=local-query-segments-queue, makeGauge → field value). High values mean search worker threads are saturated and queries will be slow. Alert fires at 200000. Correlate with CPU and the live query count below.
| Time Chart | |
Current max search-queue depth across nodes (name=local-query-segments-queue, makeGauge → field value). Segments queued for search including exports. Alert threshold is 200000; sustained values near it mean search workers are saturated and query latency will climb.
| Single Value | |
local-query-segments-queue (gauge → field value): segments queued for query per node. Moderate depth rules out worker exhaustion, workers are free and queries are running, so queuing is not why the watchdog fired. A saturated queue would point to capacity limits (see Search Queue section), a different failure mode.
| Time Chart | |
Max milliseconds a query has spent not running because it was waiting for data from bucket storage (name=query-time-spent-starved-waiting-for-bucket-storage-max, makeGauge → field value; resets when logged). Spikes mean the local segment cache is cold or object storage is slow, the bucket-bound flavor of starvation.
| Time Chart | |
Per-node breakdown of starvation events from the Starved searching log line. Fields come straight from jobDetailsKVString: waitTimeMs (time since last progress), timeSoFarMs (total query runtime), and msStarvedWaitingForBucketDownloads (time blocked on object-storage segment fetches). High maxBucketWaitMs → bucket-bound; high maxWaitMs with low bucket wait → CPU/worker-bound. Sorted by event count, worst node first.
| Table | |
Count of Starved searching warnings per node, logged when a query makes no progress past the warning threshold. A rising line marks nodes where queries are stalling. This is the log signal; the metric-based cancel counter is in the overview and below.
| Time Chart | |
Proportion of external /query and /queryjobs POST requests that returned status < 500 per 15m (#kind=requests, internal=false to exclude node-to-node). A drop below the 99% SLO line = the query API itself is erroring (auth, planner crashes, node down), distinct from queries that complete but are slow. Sustained non-100% is customer-visible even before latency SLOs breach.
| Time Chart | |
Share of queries per 15 minutes that ended without being cancelled. The line at 99% is the SLO. Query Failure Reasons shows the errors on cancelled queries.
| Time Chart | |
Queries that appeared in the cluster's top 5 by query cost, ranked by their highest cost in any 30-second window, with the user or token that ran them and the repository. Empty when no query was expensive enough to be reported.
| Table | |
Top 10 users ranked by static-query completion count in the last 24h. Same log line as the latency panels. queryRequester=organization is bucketed to organization; missing usernames go to unknown. Use to spot a single user driving abnormal query volume, often a dashboard on a short auto-refresh or a runaway script.
| Bar Chart | |
Top 10 users by SUM of query time (runtime, seconds) over 24h, the CPU-cost view of the previous panel. A user with modest count but high total runtime is running a few expensive queries; a user high on both is churning. Match against Top Expensive Queries to find the specific query strings.
| Bar Chart | |
Count of 'Starved searching' lines per node as bars. A single dominant bar means the pattern (or a saturated/unhealthy node) is localised; roughly even bars across nodes indicate a cluster-wide watchdog pattern. Use this to decide between investigating one node and treating it as a cluster-wide pattern.
| Bar Chart |
Query Performance Profiler
| Widget | Description | Type |
|---|---|---|
Share of allocated memory used for decompressing segment data.
| Single Value | |
Share of processing time spent in the reduce phase (combining results from workers).
| Single Value | |
Internal query IDs of the time slices this query was split into. One ID means the query was not sliced. Empty is normal when the query never started on workers, for example when it was stopped while waiting to start.
| Single Value | |
Bytes downloaded from bucket storage for this query because the segments were not on local disk. Empty or 0 is normal when all segments were on local disk or the query never ran on workers.
| Table | |
Per worker node: runtime, time starved, time downloading from bucket storage, segments on disk and in bucket, downloads, subquery and table registration time, failed submits and utilisation. A single slow worker points at that node. Empty is normal when the query never ran on workers, for example when it was stopped while waiting to start.
| Bar Chart | |
Runtime percentiles of runs of the same query text and searched span, grouped by the first part of the query ID, with first and last seen. Cancelled and abandoned runs are excluded. Empty when no run of this query text and span finished without being cancelled or abandoned.
| Table | |
Share of processed data that came from cached query results instead of being processed again.
| Single Value | |
Share of processing time spent in the result phase (producing the final result).
| Single Value | |
Total runtime including time queued, with the share spent queued, waiting on subqueries, starved, waiting on tables and waiting on joins.
| Single Value | |
Share of allocated memory used in the result phase.
| Single Value | |
Parent query and child queries (for example joins and defineTable subqueries) of this query, with runtime, events, results, share from cache and query cost. Empty is normal when the query has no subqueries and is not a subquery.
| Table | |
Share of processing time spent decompressing segment data.
| Single Value | |
Share of allocated memory used in the map phase.
| Single Value | |
Other runs of the same query text by the same organization, with their runtime, cost, cache share and status. Use the row action to inspect a run. Empty is normal when this query text ran only once in the time range.
| Table | |
Last HTTP status returned to the client polling this query. A request that timed out is shown as 503. Empty is normal when no client polled the query over HTTP, for example alerts and scheduled searches.
| Single Value | |
Events the query processed.
| Single Value | |
Why the query did not start right away: the reason logged when the query coordinator queued it, or waiting on child query. Empty is normal when the query started right away.
| Table | |
Another query ID that shares the internal query with this one and logged the query-end line. If shown, inspect that ID for more detail. Empty is normal when there is no sibling query.
| Single Value | |
Total physical memory of all nodes that reported in the time range, in GiB, using each node's latest value.
| Single Value | |
Nodes per LogScale version over time. Use it to see whether a runtime change lines up with an upgrade.
| Time Chart | |
Total time searches across the cluster went without progress (starved), from the starved-search log lines. Shows the whole cluster, not only this query. Empty is normal: no search stalled.
| Time Chart | |
Query cost against searched span for runs of the same query text, live and historical. Shows whether cost grows with the time range. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge. Empty when every run of this query text was cancelled.
| scatter-chart | |
Last HTTP status of each run of the same query text over time. 503 means the request timed out. Empty is normal when the runs were not polled over HTTP, for example alerts and scheduled searches.
| Time Chart | |
Organization that ran the query.
| Single Value | |
Total processing time over all query phases, including time spent waiting.
| Single Value | |
Warnings LogScale logged for this query. Empty is normal: no warnings were logged.
| Table | |
Total CPU cores of all nodes that reported in the time range, using each node's latest value.
| Single Value | |
Total memory the query allocated over all phases, in GiB.
| Single Value | |
Query cost of the historical part, including subqueries and cost reused from cache. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Single Value | |
Other query IDs that share an internal query with this one, for example when several clients use the same query. Shows their log volume over time. Empty is normal when no other query ID shares this query.
| Time Chart | |
Query text. Uses the formatted text from the query-end log line when available. Long text may be truncated in the logs.
| Table | |
Average runtime in seconds per user over time, for runs of the same query text and span that were not cancelled or abandoned. Empty when no run of this query text and span finished without being cancelled or abandoned.
| Time Chart | |
Data the query processed, in GiB.
| Single Value | |
Average events processed per user over time, for runs of the same query text and span that were not cancelled or abandoned. Empty when no run of this query text and span finished without being cancelled or abandoned.
| Time Chart | |
Share of allocated memory used in the reduce phase.
| Single Value | |
Query cost of the live part. Empty is normal for historical queries. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge.
| Single Value | |
Summary of the query: repository, kind, searched span, who started it, live or historical, events processed, results, share from cache, data size, runtime, time queued and starved, cancelled or abandoned, last poll status, coordinating node and number of slices.
| Table | |
Share of allocated memory used for initialisation and serialisation.
| Single Value | |
Bytes downloaded from bucket storage per query over time. unknown is download activity that could not be linked to a query. Empty is normal when no segments were downloaded from bucket storage.
| Time Chart | |
Share of processing time spent in the map phase (reading and filtering events).
| Single Value | |
Share of processing time spent on initialisation, waiting and serialisation.
| Single Value | |
Average query cost of the queries that were in the cluster's top 5 by cost at the same time as this query, per query. 1 cost unit ≈ 100 ms of worker CPU plus a memory charge. Empty when no query was among the most expensive in that time.
| Time Chart | |
Set internalQueryId to an internal query ID (starts with IQ-, shown as queryID in most LogScale log lines) to list the external query IDs (which do not start with IQ-) that belong to it, most recent first. The IDs are not 1:1: a query gets a new external ID each time it is submitted, and one submission can run as several internal queries. Use Show Query on a row to load that external ID into this dashboard.
| Table |
Retention & Data Lifecycle Health
| Widget | Description | Type |
|---|---|---|
min(time_remainingDays) over time, the oldest data still kept, for the most size-constrained repository. Downward trend means data is ageing out faster than before (ingest rose or disk shrank relative to policy); if it drops below the configured policy, the repo can't hold its intended window.
| Time Chart | |
Max age (converted ms → s) of the oldest event not yet uploaded to object storage (name=bucket-storage-upload-latency-max-ms, gauge → value). A rising trend means the upload pipeline is falling behind ingest; those segments are pinned on local disk and at risk if the node fails before upload completes. SLO line at 60 s.
| Time Chart | |
Age in hours of the oldest segment awaiting upload per host, from the same bucket-storage-upload-latency-max-ms gauge. High values = an upload stall on that host, segments pinned locally and unprotected until the upload lands. Sorted worst-first.
| Table | |
Each Preservation settings log event carries the enabled flag, this timechart tallies how many events per bucket reported each value. Steady bar at true or false = stable config; a transition = someone toggled preservation on / off. Correlate a spike in the preserved-segment-count chart on the left with a preservation-enabled transition here to see when policy changed.
| Time Chart | |
Sum of segments pinned by preservation config per dataspace, from RetentionJob segment preservation log. Preservation prevents retention from deleting matching segments (typically for legal hold, audit, or extended-retention overrides). Segments pinned here don't count toward the time-policy retention windows above, so a growing count reduces effective retention. Empty is normal unless preservation is intentionally in use.
| Time Chart | |
Percent used on the primary data disk per host (name=primary-disk-usage). Steady ~85% is healthy, LogScale is evicting cold segments at LOCAL_STORAGE_PERCENTAGE. A host climbing above the 85% SLO line toward 95% (PRIMARY_STORAGE_MAX_FILL_PERCENTAGE) means eviction is falling behind, investigate disk IOPS, merge backlog, and hot-segment pinning, not days until full.
| Time Chart | |
Highest current primary-disk-usage across the cluster. Expect it to sit near the 85% eviction watermark under steady state; a reading pushing toward 95% means at least one host's eviction is behind and is the fill-prediction risk. Palette, green under 85, amber 85 to 95, red at 95+.
| gauge | |
max(time_remainingDays) per dataspace from the RetentionJob Retention-stats log line, but only where reaping actually happened (reaping_count_by_time > 0), which filters out repos that haven't hit a retention policy yet. If a repository's line drops below its configured retention window here, that repo is size-bound and losing data sooner than the policy intended.
| Time Chart | |
Minimum time_remainingDays across all Retention-stats lines, the age of the oldest segment still kept, for the repository holding data the SHORTEST. If this is well below the configured retention policy, that repo is size-bound, disk too small for the ingest rate and data ages out early.
| Single Value | |
Uncompressed bytes deleted by retention in 24h, summed from the reaping_bytes_original field of the Retention-stats log. Zero with a healthy run count means nothing aged out (fine on a repo below its size / time limit); zero with retention-passes = 0 means retention isn't executing at all.
| Single Value | |
Uncompressed data deleted per time bucket (reaping_bytes_original, converted to GB). A steady sawtooth tracking ingest rate = healthy steady-state retention. Flat zero while ingest continues = retention has stopped reaping, cross-check the retention-runs scalar and the disk-cache section.
| Time Chart | |
Segments deleted by the size-based retention setting, per repository (reaping_count_by_original_size from the retention statistics log). A repository shown here but not in the by-time chart reaches its size limit before its time limit.
| Time Chart | |
Count of segments deleted per bucket by the TIME retention policy (reaping_count_by_time from RetentionJob Retention-stats), split by dataspace. This is the count complement to the reap-volume chart at the top of the dashboard, that chart shows bytes reaped cluster-wide; this shows the per-repo count broken out.
| Time Chart | |
Count of RetentionJob Retention-stats log lines in 24h (class=/RetentionJob/). ABSENCE IS THE ALARM, a zero value means retention has stopped firing, old segments aren't being deleted from global, and segment count + disk pressure climb. Colour, green ≥1 run, red at 0 (silent failure).
| Single Value | |
Segment merges per second per host (1-minute rate of the segment-merge-cpu-time timer). Few merges while the segment count grows means small segments are building up.
| Time Chart | |
Count of segments that were marked deleted then un-marked (Undoing marking segment deleted), split by the reason field. Non-zero is an edge case, the retention or deletion pipeline decided to undelete a segment already scheduled for cleanup. Useful when investigating retention anomalies or preservation-config side-effects. Healthy is empty.
| Time Chart | |
Count of segment files deleted from bucket storage (deleting segment from bucketId log line), split by dataspace. This is the durable- storage side of tombstone cleanup, the job that removes already-tombstoned segments from S3/GCS. A backlog here (RetentionJob firing but bucket cleanup stagnant) means bucket-side deletion is stuck, which inflates object-storage cost.
| Time Chart | |
Tombstoned segments deleted by the retention job, per repository (Deleting tombstoned segment). Empty when there is nothing to delete.
| Time Chart |
Scheduled Search & Workflow Health
| Widget | Description | Type |
|---|---|---|
Count of alert job-loop completion heartbeats in the last 24h (subCategory=Job, ... job run finished) across legacy, filter, and aggregate alert engines. This is emitted every poll interval regardless of whether any alert matched. A value that falls to zero means the alert poll loop is stuck, no alerts can fire.
| Single Value | |
Completed poll loops of the alert and scheduled-search engines (job run finished / loop finished), logged every poll interval whether anything matched or not. A gap means no alerts could fire in that time.
| Time Chart | |
Scheduled searches that fell behind their schedule and were rescheduled (Scheduled search lagged behind and had to be rescheduled), per view and search name.
| Table | |
Warnings where a job loop could not finish within its poll interval (... is overloaded and cannot keep up, subCategory=Job). This precedes visible alert lag: once the loop runs long, each cycle starts later than scheduled and delay compounds. Sustained overload on aggregate/filter alert loops is the known lead-in to multi-hour alert lag on high alert-count clusters.
| Time Chart | |
Count of Warning/Error activity events across all trigger types (ScheduledSearch, FilterAlert, AggregateAlert, Alert) in the last 24h. Non-zero means at least one execution logged a query error, permission failure, lag, or action-delivery failure. Drill into the per-section panels to localise, or the Alert Status Overview dashboard for per-alert breakdown.
| Single Value | |
Count of scheduled-search execution starts in the last 24h, using the per-run Starting query marker (category=ScheduledSearch, subCategory=Query). A sudden drop to near-zero with configured schedules still present means the scheduler stopped firing, check node health and the heartbeat panel in this section.
| Single Value | |
Scheduled-search execution starts per bucket, counted from the per-run Starting query marker (category=ScheduledSearch, subCategory=Query). Healthy clusters show volume proportional to the number of configured schedules. A step down to zero means the job runner stopped firing, not individual searches failing. Cross-check against node shutdowns in the JVM dashboard.
| Time Chart | |
Average delay between a scheduled search's planned run time and when its firing actions event was ingested, in seconds. Rising means scheduled searches run late.
| Time Chart | |
Scheduled search activity events per severity. Error and Warning are runs that did not complete as expected; Info is the normal baseline.
| Time Chart | |
Share of successful scheduled-search runs that triggered an action and also logged a warning, for example a missing lookup file or a truncated result.
| Time Chart |
Segment Prefetch & Download Health
| Widget | Description | Type |
|---|---|---|
Total MB read from bucket storage (size field) and written to bucket storage (size field), summed per 1-minute bucket. Read climbing while write stays flat = re-fetch storm (prefetch loop). Both climbing together = a bucket-target transfer or a heavy backfill (see Bucket Migration). Log-based so it works regardless of which storage-metric names are enabled. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and this widget is empty.
| Time Chart | |
Log lines saying an upload is delayed because the segment is not local yet. Non-zero means a bucket-target change is downloading segments to upload them again, which causes heavy downloads that are not a loop. Empty on clusters without bucket storage.
| Single Value | |
Segments downloaded and segments queued for local deletion per node, for the 20 nodes with the most. One node far above the rest means the loop is on that node. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and only deletes show.
| Bar Chart | |
Segments downloaded plus segments queued for local deletion, per node per hour, for the 20 nodes with the most. A row that stays hot for many hours is a node where segments keep cycling. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and this widget is empty.
| Heat Map | |
Delete events split by reason over time. Dominant 'on-other-hosts' = copies are dropped because ownership excludes this node; paired with prefetch downloads this is the ownership-disagreement pattern that drives the loop. Dominant 'segmentGone' / 'segmentDeleted' = normal cleanup.
| Time Chart | |
Download events split by reason over time. If 'prefetch' is the dominant series, the prefetch controller is issuing the downloads that later get evicted, the loop's driver. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and this widget is empty.
| Time Chart | |
Segments downloaded from bucket storage and segments queued for deletion from local disk, per hour, cluster-wide. Each count is one segment. Both lines high and close together over many hours means segments may be downloaded and deleted again and again. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and only the delete line shows.
| Time Chart | |
Highest concurrent bucket-storage download count observed on any single host in the window (bucket-storage-in-progress-downloads, value). A high steady reading here is the saturation ceiling of the storm. Empty on clusters without bucket storage.
| gauge | |
Global snapshot size trend (globalsnapshot-size gauge, converted to GB). Growth > 0.5 GB/h is a leading indicator of heap pressure, the snapshot holds metadata for all segments and churning ownership inflates it. No horizontal plot line, the SLO is a growth RATE, not a level.
| Time Chart | |
Segments that were both downloaded from bucket storage at least twice and queued for local deletion at least twice in the time range. Green: none. Red: one or more segments are cycling. The table in Segments and Nodes Involved lists them. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and this widget stays at 0.
| Single Value | |
Click the â‹® at the start of the row and choose the dashboard to open it in a new tab.
| Table | |
Cluster-wide mini-segment counts: waiting to be merged, already merged but not yet deleted, and all mini-segments including tombstones. Mini-segments are merged only after they are replicated, so a waiting count that keeps rising means the merge pipeline is falling behind, and global grows with it. A merged-not-deleted count that keeps rising means deletes are behind.
| Time Chart | |
Each segment reported missing, with the reason, the hosts that hold a copy (currentHosts) and the hosts that should hold one (ownerHosts). For Unavailable, bringing back a host in currentHosts makes the data readable again. Lost segments have an empty currentHosts. Newest first.
| Table | |
Size of segment files that are still waiting to be uploaded to bucket storage and are not known to have more than one copy on the cluster's own disks. This data would be lost if the node holding it failed now. 0, or briefly above 0 while segments upload, is normal. A value that stays up or keeps growing means data is at risk; check whether uploads are failing or slow. Every node that reports it gives the same cluster-wide value, so the highest is shown. Empty on clusters without bucket storage.
| Time Chart | |
Segment downloads from bucket storage that were running at that moment (not waiting in a queue): the total across all nodes and the node with the most. A total that stays high for hours, together with many segment downloads and deletes in the charts above, means nodes keep fetching segments. Empty on clusters without bucket storage.
| Time Chart | |
Segments written by digest, mini-segments merged into a full segment, and segments marked deleted, per hour. In a healthy cluster the three lines move together. Created staying above merged means mini-segments pile up; merged staying above deleted means replaced mini-segments are not removed. Deletes include retention, not only replaced mini-segments.
| Time Chart | |
Number of segments each node picked up from its queue of segments to copy from other nodes, at the last poll. Segments already in bucket storage are downloaded from the bucket instead and are not in this queue, so on clusters without bucket storage every new mini-segment passes through it. A line flat at 10,000 is a full queue. Normal is tens.
| Time Chart | |
Rate at which each node puts segments back on its fetch queue to try again. Healthy nodes stay near zero. A rate that climbs and stays high is a retry loop: the same segments are retried instead of new ones being fetched, so replication falls behind.
| Time Chart | |
Size of data with fewer copies than the replication factor, and size of data that no live node can serve (missing includes both lost segments and segments whose only holders are down). Works with and without bucket storage. Briefly above 0 during restarts is normal; a value that stays up or grows means replication is not keeping up. The exact number of missing segments is the missingSegments field of the same log line.
| Time Chart | |
Log events where a segment already marked deleted was un-marked (Undoing marking segment deleted), grouped by reason. In a healthy cluster this is empty the delete decision was final. In a loop, segments are resurrected as ownership flips back and forth, which is the exact signal that the delete job and download job are fighting over the same segments.
| Time Chart | |
Log lines reporting a missing segment, split by reason. Lost means no node has a copy and it is not in bucket storage. Unavailable means copies exist but every node holding one is down; the data returns when the node does. Counts are log lines, and each check logs only a limited number of segments, so read them as yes or no, not as a segment count. The node that reports the line is the node running the check, not the node that holds the segment; the table of missing segments shows the hosts.
| Time Chart | |
Latest max global snapshot size across hosts, converted from bytes to GB. Watch this alongside the trend chart when deciding whether the loop is driving metadata growth toward OOM.
| Single Value | |
The 25 segments downloaded and queued for local deletion most often, with how many nodes touched them and when last. A segment with several downloads and deletes is cycling. Downloads are logged only when bucket storage is used, so the download data is missing on clusters without bucket storage and this widget is empty.
| Table | |
ERROR lines per node reporting a segment whose Kafka offsets are older than the oldest event Kafka still holds. One node means that node is the problem; every node means a cluster-wide cause such as a short Kafka retention or stalled replication.
| Time Chart | |
unclean shutdown detected WARN log lines from DiscardHandling. Fires on startup when the previous JVM process exited without completing its shutdown hook, usually SIGKILL after an OOM, OOM-killer, or node crash. Multiple within a short window on the same host = crash loop. Empty = every restart was graceful.
| Time Chart |
Storage Health
| Widget | Description | Type |
|---|---|---|
Seconds a segment upload to Azure Blob Storage took: median and the slowest 1%. Rising upload time points at Azure-side network, throttling or region latency. Empty on clusters not using Azure. Empty on clusters without bucket storage.
| Time Chart | |
Segment uploads to Azure Blob Storage that failed, grouped by the error returned: the exception, the HTTP status code where the error includes one, and the start of the message. Empty on clusters not using Azure, and healthy when empty. Empty on clusters without bucket storage.
| Time Chart | |
ERROR-level log events from any bucket storage component, grouped by error: the component, the start of the message and the exception where there is one. Typical causes are connectivity, permissions, throttling or provider faults. Empty is healthy. Empty on clusters without bucket storage.
| Time Chart | |
Count per #vhost of queries that skipped a segment because downloading it from bucket storage kept failing until the retry timeout. Each event = a query that returned incomplete historical data. Correlate with object-storage health and download slots. Empty on clusters without bucket storage.'
| Time Chart | |
Completed raw-segment uploads to object storage per #vhost, from the completed for raw segment log line (works across S3, GCS, Azure, provider only appears in the message text). ABSENCE IS THE ALARM, all lines flat at zero while ingest continues means the upload pipeline has stalled. Correlate with pending backlog and upload latency below. Empty on clusters without bucket storage.
| Time Chart | |
Count of delaying upload, segment not held locally yet events from the bucket-transfer job. Each one = a segment that must be downloaded from object storage before it can be re-uploaded to the new target. High count = the transfer is pulling a lot of cold data back to local disk; correlate with fetch timeouts and upload latency above. Empty on clusters without bucket storage.'
| Time Chart | |
Segment upload activity from the bucket-transfer job. Series scheduled = uploads queued for the new target; completed = uploads finished. Completed climbing tracking scheduled = healthy migration; scheduled climbing while completed stays flat/zero = transfer is stalled (check bucket-provider errors above). Empty on clusters without bucket storage.'
| Time Chart | |
bucket-storage-max-concurrent-download-operations per #vhost. Same shape as the upload counterpart but for the read path, shows how close each node is to its download slot ceiling. Saturated slots + rising fetch timeouts (left) = the read path is capacity-bound, not I/O-bound. Empty on clusters without bucket storage.
| Time Chart | |
bucket-storage-max-concurrent-upload-operations per #vhost, the configured upload threadpool ceiling and how close the node is to it. If this is near the configured limit while pending backlog rises, the ceiling itself is the bottleneck. Empty on clusters without bucket storage.
| Time Chart | |
Current cluster-wide bytes waiting to be uploaded to object storage, max per #vhost then summed, from the bucket-storage-pending-upload gauge, converted to GB. Sustained non-zero means uploads are not keeping up with ingest; local disk fills and data may be evicted before it is durable in the bucket. Empty on clusters without bucket storage.'
| Single Value | |
Current cluster-wide bytes pending upload that lack a second known local replica, from bucket-storage-pending-upload-underreplicated (max per #vhost, summed, converted to GB). This is the subset of the backlog that is lost if that node's local disk is lost before upload completes. Green at 0, red at ≥1 GB. Empty on clusters without bucket storage.'
| Single Value | |
Total size of the segment files stored in bucket storage, in TiB (1 TiB = 1,024 GiB). Every node that reports it gives the same cluster-wide total, so the highest value is shown. Data still waiting to be uploaded is not included. Empty on clusters without bucket storage.
| Single Value | |
Current worst-node value of bucket-storage-upload-latency-max-ms, converted from ms to seconds. This is the upload pipeline's lag SLI; alert around 30 s. Rising = the writer is falling behind object storage. Empty on clusters without bucket storage.
| gauge | |
Count of Unexpectedly found snapshot with an epoch value larger than the current epoch detections from the bucket cleaning job, snapshot files whose epoch exceeds the current epoch, indicating an object-storage consistency fault. Any non-zero value warrants investigation of clock skew and recent cluster operations. Colour, green at 0, red at ≥1. Empty on clusters without bucket storage.
| Single Value | |
Bucket keys flagged with an epoch higher than the current epoch, from the cleaning job's Unexpectedly found snapshot ERROR (fields foundEpoch, lbrc_key). Grouped by key with hit count, latest foundEpoch, and seconds since last seen so you can hand exact keys to LogScale support. Do NOT delete bucket objects manually. Empty on clusters without bucket storage.
| Table | |
bucket-storage-fetch-for-query-queue per #vhost, pending fetches specifically triggered by running queries (as distinct from prefetch or transfer). Rising = queries are generating more segment fetches than the download pipeline is serving. This is the leading indicator that query latency is about to climb. Empty on clusters without bucket storage.
| Time Chart | |
Seconds a segment upload to Google Cloud Storage took: median and the slowest 1%. Rising upload time points at GCS-side network, throttling or region latency. Empty on clusters not using GCS. Empty on clusters without bucket storage.
| Time Chart | |
Segment uploads to Google Cloud Storage that failed, grouped by the error returned: the exception, the HTTP status code where the error includes one, and the start of the message. Empty on clusters not using GCS, and healthy when empty. Empty on clusters without bucket storage.
| Time Chart | |
Backlog of data awaiting upload to object storage in GB per #vhost, from bucket-storage-pending-upload (converted to GB). Growing backlog = uploads are slower than ingest; local disk fills and data may be evicted before it is durable. Empty on clusters without bucket storage.'
| Time Chart | |
Per-#vhost bytes pending upload that lack a second known local replica, converted to GB. This is the subset of the backlog that is lost if that node's local disk is lost before upload completes; it should trend to zero as uploads drain. Empty on clusters without bucket storage.'
| Time Chart | |
Max seconds a query spent not running while waiting for data from bucket storage (query-time-spent-starved-waiting-for-bucket-storage-max, converted from ms). Rising values = queries are blocking on slow object-storage fetches even when they do not time out. Empty on clusters without bucket storage.'
| Time Chart | |
S3 uploads that took longer than LogScale's logging threshold, by type: multipart uploads (large files sent in parts) and direct uploads. Each event is one slow upload. Bursts point at S3-side network, throttling or region latency rather than LogScale itself. Empty is healthy. Empty on clusters without bucket storage.
| Time Chart | |
Segment uploads to S3 that failed, grouped by the error S3 returned: the exception, the HTTP status code and the start of the message. A status of 500 or 503 means S3 itself failed the request; 403 means access was refused; no status means the error did not include a status code from S3. Empty is healthy. Empty on clusters without bucket storage.
| Time Chart | |
Cluster-wide sum of bucket-storage-total-segment-size across #vhost, converted to GB. Shows the total footprint of durable segments in object storage over time. Read alongside the retention-lifecycle dashboard's reap volume, steady growth here with steady reaping = healthy retention keeping the dataset bounded; unbounded growth = retention isn't keeping pace or preservation is retaining too much. Empty on clusters without bucket storage.
| Time Chart | |
Max age of the oldest un-uploaded segment per #vhost, from bucket-storage-upload-latency-max-ms, converted to seconds. Rising values = the upload pipeline is falling behind object storage. SLO line at 30 s. Empty on clusters without bucket storage.
| Time Chart | |
1-minute rate of bucket-storage-request-upload-queue-overflow per #vhost. Non-zero = the upload queue overflowed, meaning upstream code is being told to slow down. Sustained here = the upload pipeline can't accept work as fast as ingest is producing it. Empty on clusters without bucket storage.'
| Time Chart | |
bucket-storage-in-progress-uploads per #vhost as a heat-map, uploads actually running (not just queued). A hot band on one node with others cold = skew (slow bucket target or hot uploader); all-zero while backlog is non-zero = stalled uploader. Empty on clusters without bucket storage.
| Heat Map |
Tag Cardinality & Datasource Health
| Widget | Description | Type |
|---|---|---|
Distinct values per tag key per repository. High cardinality on a tag (e.g., 10000+ unique values) drives datasource explosion and OOM risk.
| Table | |
Events where LogScale auto-sharded a datasource to cope with cardinality. Frequent sharding indicates the tag grouping configuration is too broad.
| Time Chart | |
Active datasources on each node over time (digest-active-datasources, one series per #vhost). Each unique combination of tag values creates a datasource. Sudden spikes = new high-cardinality tag values arriving; the Tag Cardinality per Field table shows which repository.
| Time Chart | |
Cluster-wide total of active datasources right now. Use as a quick health indicator, compare against historical baseline.
| Single Value | |
Repos that have triggered LogScale's internal high-cardinality warnings. Non-zero = that repo's tag configuration needs review.
| Table | |
Repositories where LogScale is right now closing segments on datasources tagged with TooManyTagValueCombination=true, meaning the repo hit its max-datasources limit and new events are routing to the shared overflow bucket. Data landing here is NOT tagged with the original fields, so tag-filtered queries will silently miss it. Empty is healthy. Any row is actionable.
| Table | |
Nodes ranked by active datasources (digest-active-datasources, per #vhost). The Tag Cardinality per Field table shows which repositories they belong to.
| Table |