Job CPU Time vs Elapsed Time

Compare CPU time and elapsed time across jobs to identify whether different job types exhibit distinct performance patterns. Using a nested groupBy() on a low-cardinality field such as job_type produces long format data, which the scatter chart renders as distinct colored clusters — one color per job type — making it easy to see whether batch and realtime jobs occupy different regions of the chart.

Visualization: the scatter chart displays one point per job, grouped by job type (long format). The outer groupBy() on job_type becomes the series, giving each job type a distinct color:

  • X-axis (horizontal) shows CPU time in milliseconds (cpu_ms)

  • Y-axis (vertical) shows average elapsed time in milliseconds (elapsed_ms)

  • Each colored cluster represents one job type. Points within a cluster are individual jobs of that type: cluster separation reveals whether job types have fundamentally different resource profiles.

Screenshot showing a scatter chart widget displaying job execution data with two distinct colored clusters, one per job type, with average CPU time in milliseconds on the horizontal axis and average elapsed time in milliseconds on the vertical axis, with a trend line running through each cluster, and Format Scatter panel visible on the right

Figure 163. Job CPU Time vs Elapsed Time


Sample input data:

@timestampcpu_mselapsed_mshostjob_idjob_type
1970-01-01T00:00:0218001200worker-01batch-001batch
1970-01-01T00:00:0224502500worker-02batch-002batch
1970-01-01T00:00:0231003400worker-01batch-003batch
1970-01-01T00:00:021600500worker-03realtime-001realtime
1970-01-01T00:00:0219001300worker-04realtime-002realtime

Query:

logscale
groupBy(job_type, function=[groupBy(cpu_ms, function=avg(elapsed_ms, as=elapsed_ms))])

Query breakdown:

  1. The outer groupBy() on job_type is the group field: it has low cardinality (two values: batch and realtime) and becomes the series name, giving each job type a distinct color.

  2. The inner groupBy() on cpu_ms produces one result row per unique CPU time value within each job type, computing the average elapsed time for each.

  3. The result has exactly three fields — job_type (series), cpu_ms (X-axis), and elapsed_ms (Y-axis) — which is the structure required for long format to render distinct colored clusters.

Configuration:

  1. From the Search page, type your query in the Query Editor → click Run

  2. Choose Scatter Chart in the Widget selector

  3. Click the style icon : this opens the Format panel on the side where some properties are already configured by default based on the query result.

  4. Set Input data format to Long to ensure the chart renders the two job types as distinct colored clusters.

  5. In Points, increase the Point size slider and reduce Point opacity so overlapping points remain visible.

  6. In Colors, select a Palette to assign distinct colors to each job type cluster.

  7. Toggle Trend line on to display a regression line for each cluster.

  8. In X-axis, enter a custom axis title such as CPU Time (ms).

  9. In Y-axis, enter a custom axis title such as Elapsed Time (ms).

To customize this widget further, see Scatter Chart Property Reference.