Monitor Network Error Outliers

Track total network errors across Docker containers over time

This is a query example for the Total Network Error Count Outlier widget in the Docker Overview dashboard of the docker/metrics package.

Query

flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[[Expression]] 2{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> result
logscale
total_bytes:=(docker.network.in.errors+docker.network.inbound.errors+docker.network.out.errors+docker.network.outbound.errors)
|timeChart(function=max(total_bytes))

Introduction

This widget is used to visualize network error patterns in Docker containers by aggregating different types of network errors and displaying them in a time-based chart, helping identify potential networking issues and outliers.

In this widget, network errors from different sources are combined and tracked over time using the timeChart() function with max() aggregation. The query first creates a total error count and then charts its progression.

Example incoming data might look like this:

@timestamp#repo#type@id@ingesttimestamp@rawstring@timestamp.nanos@timezone@type_datagen_identifieragent.typeagent.versioncontainer.idcontainer.image.namecontainer.namedocker.container.event.actiondocker.container.event.actor.iddocker.container.event.fromdocker.container.event.statusdocker.container.event.typedocker.diskio.read.bytesdocker.diskio.read.opsdocker.diskio.summary.bytesdocker.diskio.summary.opsdocker.diskio.write.bytesdocker.diskio.write.opsdocker.event.actiondocker.healthcheck.event.end_datedocker.healthcheck.event.exit_codedocker.healthcheck.failingstreakdocker.healthcheck.statusdocker.image.createddocker.image.id.currentdocker.image.id.parentdocker.image.size.regulardocker.image.size.virtualdocker.image.tags[0]docker.info.containers.pauseddocker.info.containers.runningdocker.info.containers.stoppeddocker.info.containers.totaldocker.info.iddocker.info.imagesevent.datasetevent.modulehost.namemetricset.nameservice.type
2026-03-10T06:41:21auto-dashboard-queriesjsongtksp2dMmrYg9WSX5CeYyU5V_2_11_17731248812026-03-10T06:41:21{"docker.image.id.current":"sha256:abcd1234efgh5678","metricset.name":"image","docker.image.size.regular":"133169152","docker.image.created":"2026-03-10T06:41:21.002Z","event.module":"docker","docker.image.tags":["nginx:latest"],"@timestamp":"2026-03-10T06:41:21.002Z","host.name":"docker-host-01","agent.type":"metricbeat","@type":"docker","event.dataset":"docker.image","agent.version":"7.11.1","docker.image.size.virtual":"133169152","docker.image.id.parent":"sha256:parent1234567890ab","service.type":"docker","_datagen_identifier":"bbe9c9c08ebf329bc648a36a3991a240"}0Zdockerbbe9c9c08ebf329bc648a36a3991a240metricbeat7.11.1                   2026-03-10T06:41:21.002Zsha256:abcd1234efgh5678sha256:parent1234567890ab133169152133169152nginx:latest      docker.imagedockerdocker-host-01imagedocker
2026-03-10T06:41:21auto-dashboard-queriesjsongtksp2dMmrYg9WSX5CeYyU5V_2_12_17731248812026-03-10T06:41:22{"@timestamp":"2026-03-10T06:41:21.801Z","event.module":"docker","host.name":"docker-host-02","container.image.name":"nginx:latest","agent.type":"metricbeat","metricset.name":"event","docker.event.action":"stop","docker.container.event.action":"start","_datagen_identifier":"bbe9c9c08ebf329bc648a36a3991a240","service.type":"docker","docker.container.event.type":"container","docker.container.event.from":"redis:6.2-alpine","container.id":"a1b2c3d4e5f6","container.name":"nginx-web","event.dataset":"docker.event","docker.container.event.status":"Up 2 hours","@type":"docker","agent.version":"7.12.0","docker.container.event.actor.id":"b2c3d4e5f6a7"}0Zdockerbbe9c9c08ebf329bc648a36a3991a240metricbeat7.12.0a1b2c3d4e5f6nginx:latestnginx-webstartb2c3d4e5f6a7redis:6.2-alpineUp 2 hourscontainer      stop                docker.eventdockerdocker-host-02eventdocker
2026-03-10T06:41:22auto-dashboard-queriesjsongtksp2dMmrYg9WSX5CeYyU5V_2_13_17731248822026-03-10T06:41:23{"docker.info.containers.paused":"0","docker.info.images":"15","metricset.name":"info","docker.info.containers.running":"5","agent.type":"metricbeat","host.name":"docker-host-03","event.module":"docker","@timestamp":"2026-03-10T06:41:22.581Z","agent.version":"7.13.2","@type":"docker","event.dataset":"docker.info","docker.info.containers.stopped":"2","docker.info.containers.total":"7","_datagen_identifier":"bbe9c9c08ebf329bc648a36a3991a240","service.type":"docker","docker.info.id":"ABCD:EFGH:IJKL:MNOP:QRST:UVWX:YZ12:3456"}0Zdockerbbe9c9c08ebf329bc648a36a3991a240metricbeat7.13.2                         0527ABCD:EFGH:IJKL:MNOP:QRST:UVWX:YZ12:345615docker.infodockerdocker-host-03infodocker
2026-03-10T06:41:23auto-dashboard-queriesjsongtksp2dMmrYg9WSX5CeYyU5V_2_14_17731248832026-03-10T06:41:24{"container.id":"c3d4e5f6a7b8","container.name":"redis-cache","_datagen_identifier":"bbe9c9c08ebf329bc648a36a3991a240","service.type":"docker","docker.healthcheck.failingstreak":"0","@type":"docker","event.dataset":"docker.healthcheck","agent.version":"7.14.0","agent.type":"metricbeat","container.image.name":"postgres:14","event.module":"docker","@timestamp":"2026-03-10T06:41:23.389Z","host.name":"swarm-manager-01","docker.healthcheck.status":"healthy","docker.healthcheck.event.end_date":"2026-03-10T06:41:23.389Z","metricset.name":"healthcheck","docker.healthcheck.event.exit_code":"0"}0Zdockerbbe9c9c08ebf329bc648a36a3991a240metricbeat7.14.0c3d4e5f6a7b8postgres:14redis-cache            2026-03-10T06:41:23.389Z00healthy            docker.healthcheckdockerswarm-manager-01healthcheckdocker
2026-03-10T06:41:24auto-dashboard-queriesjsongtksp2dMmrYg9WSX5CeYyU5V_2_15_17731248842026-03-10T06:41:24{"container.id":"d4e5f6a7b8c9","container.name":"postgres-db","service.type":"docker","_datagen_identifier":"bbe9c9c08ebf329bc648a36a3991a240","docker.diskio.read.ops":"125","docker.diskio.read.bytes":"1048576","docker.diskio.summary.bytes":"3145728","@type":"docker","event.dataset":"docker.diskio","docker.diskio.write.bytes":"2097152","agent.version":"7.15.1","docker.diskio.write.ops":"234","agent.type":"metricbeat","container.image.name":"mongo:5.0","event.module":"docker","@timestamp":"2026-03-10T06:41:24.187Z","host.name":"swarm-worker-01","docker.diskio.summary.ops":"359","metricset.name":"diskio"}0Zdockerbbe9c9c08ebf329bc648a36a3991a240metricbeat7.15.1d4e5f6a7b8c9mongo:5.0postgres-db     104857612531457283592097152234                 docker.diskiodockerswarm-worker-01diskiodocker

Step-by-Step

  1. Starting with the source repository events.

  2. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[[Expression]] 2{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> result style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    total_bytes:=(docker.network.in.errors+docker.network.inbound.errors+docker.network.out.errors+docker.network.outbound.errors)

    Creates a new field named total_bytes that sums all network errors from docker.network.in.errors, docker.network.inbound.errors, docker.network.out.errors, and docker.network.outbound.errors fields.

  3. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[[Expression]] 2{{Aggregate}} result{{Result Set}} repo --> 1 1 --> 2 2 --> result style 2 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    |timeChart(function=max(total_bytes))

    Creates a timechart showing the maximum value of total_bytes for each time bucket, with results stored in _bucket and _max fields.

  4. Event Result set.

Summary and Results

The widget is used to monitor network error patterns across Docker containers, helping identify potential networking issues and their timing.

This widget is useful to detect network error spikes, track error patterns over time, and identify containers that might be experiencing network-related issues.

Sample output from the incoming example data:

_bucket_max
1773123300000 
177312420000011
1773120600000 
1773121500000 
1773122400000 

The output displays time buckets in the _bucket field and the maximum total error count in the _max field, with a peak of 11 errors detected in one time period.