How-To: Stop Running Queries using GraphQL

Functionality for stopping queries is needed when queries seem to be congesting the system. Having a some form of handle to do this is more convenient compared to restarting the entire system.

The idea behind the functionality is that when queries are requested to stop, a chatter message is sent to all nodes in the cluster and then each node will then react to the message by stopping queries.

Each node checks for new messages every 30 seconds. The delivery guarantee is at most once, meaning that if a node did not receive the message queries will not be stopped.

There are three GraphQL mutations for stopping queries:

Stop all queries
Stop streaming queries
Stop historical queries

Manage cluster permissions are required to execute any of the mutations.

When stopping queries the query stopper class (c.h.e.a.q.QueryStopper) will log that it will be stopping queries

Stopping All Queries

The stopping all queries will stop everything as the name implies, this includes streaming queries.

Mutation

graphql

mutation {
  stopAllQueries
}

Response

json

{
  "data": {
    "stopAllQueries": true
  },
  "extensions": {
    "preview": [
      {
        "name": "stopAllQueries",
        "reason": "[PREVIEW: Feature still in development]"
      }
    ]
  }
}

The log that will be displayed for regular queries is:

default

2021-10-25T13:05:29.267+0200 [humio-pekko.actor.default-dispatcher-26] INFO c.h.q.ExecutionContext$ 1 - test_pid=24903 query is cancelled queryID='<ID>'. host='localhost:8080' vhost=1 reason='Chatter message was received requesting all queries to stop'

During the query execution:

Dashboards will stop momentarily, but will refresh and continue to display new data.
A small red warning sign will appear on each widget, indicating that it has been stopped.
A similar thing will happen in the search view, but the query will resume.
Alerts will restart after they have been stopped.

Stopping Streaming Queries

This mutation will stop all streaming queries. They will not restart automatically. If a streaming query is started on curl the client will see:

shell

curl: (56) Recv failure: Connection reset by peer

When the streaming query is stopped.

The streaming query coordinator (c.h.s.e.StreamingQueryCoordinator) will fail with a run time exception with the message: Chatter message was received requesting streaming queries to stop

Mutation

graphql

mutation {
  stopStreamingQueries
}

Response

graphql

{
  "data": {
    "stopStreamingQueries": true
  },
  "extensions": {
    "preview": [
      {
        "name": "stopStreamingQueries",
        "reason": "[PREVIEW: Feature still in development]"
      }
    ]
  }
}

Stopping Historical Queries

This mutation will only stop the historical parts of queries.

The log that will be displayed when stopping historical queries is:

syslog

2021-10-25T13:18:32.264+0200 [humio-pekko.actor.default-dispatcher-19] INFO c.h.q.ExecutionContext$ 1 - test_pid=24903 query is cancelled queryID='<ID>'. host='localhost:8080' vhost=1 reason='Chatter message was received requesting historical queries to stop'

Historical queries will be restarted after they have been stopped.

Mutation

graphql

mutation {
  stopHistoricalQueries
}

Response

graphql

{
  "data": {
    "stopHistoricalQueries": true
  },
  "extensions": {
    "preview": [
      {
        "name": "stopHistoricalQueries",
        "reason": "[PREVIEW: Feature still in development]"
      }
    ]
  }
}

Knowledge Base