Polling a Query Job

This endpoint lets you poll running Query Jobs.

To poll a running Query Job, make an HTTP GET request to the job.

In the following example request, replace {id} with the ID from the response of the Running Query Jobs:

Description Request the current query status and results  
MethodGET /api/v1/repositories/repo/queryjobs/id 
Request DataQueryJobInput 
Response DataQueryResults 
Authentication Requiredyes 
Path ArgumentsDescriptionData typeRequired?
id ID of the query job stringrequired
repo Name of repository or view to be searched stringrequired
Return Codes 
200 Request complete  
400 Request is malformed and either missing critical fields or the JSON is badly structured  
500 Request failed  

When LogScale runs a search, it returns partial results, meaning the results that were found at the time query was polled. LogScale searches the newest data first, and then searches progressively backward in time.

The effect is for LogScale to produce some results as soon as possible. If there are more results in the output, the done: true property in a poll query shows if the query is finished.

For example, to run a query for a query job matching ID {id}:

shell
$ curl https://$YOUR_LOGSCALE_URL/api/v1/repositories/$REPOSITORY_NAME/queryjobs/$ID \
  -H "Authorization: Bearer $API_TOKEN"

The result may be quite large as it includes both the search results, and also the metadata about the query execution for those events found up until that point.

An example of this output is shown below, truncated for readability:

json
{
   "cancelled" : false,
   "done" : true,
   "events" : [
      {
         "#category" : "Alert",
         "#repo" : "humio-activity",
         "#severity" : "Info",
         "@id" : "uICEKBYrbcj5h4pXZ08FKI3Y_153_418_1722252213",
         "@ingesttimestamp" : "1722252213153",
         "@rawstring" : "category=\"Alert\" severity=\"Info\" @timestamp=\"1722252213152\" message=\"Alert found no results and will not trigger\" orgId=\"p1lVrKvnGK2wV7iQDSJkanF0iOwu9XJK\" subCategory=\"Alert\" alertId=\"SYMO03XiIbyDFqwF80txtebvOOJdpvOM\" alertName=\"High Cost\" viewId=\"eS7vttlsm1SvqqTcR4aUEDAB\" dataspace=\"view_asset_migration_test_gw\" status=\"Success\" externalQueryId=\"P31-TdoncOlaRBM3CJ1hQy43CCyc\" queryProcessedEvents=\"0\" queryTimeMillis=\"139616096\" alertTime=\"1722252209403\"",
         "@timestamp" : 1722252213152,
         "@timestamp.nanos" : "0",
         "@timezone" : "Z",
         "alertId" : "SYMO03XiIbyDFqwF80txtebvOOJdpvOM",
         "alertName" : "High Cost",
         "alertTime" : "1722252209403",
         "dataspace" : "view_asset_migration_test_gw",
         "externalQueryId" : "P31-TdoncOlaRBM3CJ1hQy43CCyc",
         "message" : "Alert found no results and will not trigger",
         "orgId" : "p1lVrKvnGK2wV7iQDSJkanF0iOwu9XJK",
         "queryProcessedEvents" : "0",
         "queryTimeMillis" : "139616096",
         "status" : "Success",
         "subCategory" : "Alert",
         "viewId" : "eS7vttlsm1SvqqTcR4aUEDAB"
      },
   ],
   "fieldStats": [
      {
          "fieldName": "message",
          "numOfDistinctValues": 3,
          "numOfOccurrences": 3,
          "topValues": {
              "Scheduled reports job finished": 1,
              "Scheduled reports job started": 1,
              "FDR polling loop finished": 1
          }
      },
      {
          "fieldName": "class",
          "numOfDistinctValues": 2,
          "numOfOccurrences": 3,
          "topValues": {
              "c.h.e.v.f.FdrJob": 1,
              "c.h.s.j.ScheduledReportJob": 2
          }
      }
   ],   
   "filesUsed" : [],
   "metaData" : {
      "costs" : {
         "liveCost" : 0,
         "liveCostRate" : 0,
         "staticCost" : 8,
         "staticCostRate" : 0
      },
      "digestFlow" : {
         "ingestTimeKnownGood" : 1722252211871,
         "maxIngestLatency" : 2325,
         "minIngestTimeIncluded" : 1722252211871
      },
      "eventCount" : 70735,
      "extraData" : {
         "hasMoreEvents" : "true"
      },
      "filterQuery" : {
         "allowEventSkipping" : false,
         "end" : 1722252218865,
         "includeDeletedEvents" : false,
         "ingestEnd" : 9223372036854775807,
         "ingestStart" : 0,
         "isAlertQuery" : false,
         "isInteractive" : false,
         "isLive" : false,
         "isRepeatingSubquery" : false,
         "languageVersion" : "legacy",
         "noResultUntilDone" : false,
         "queryString" : "query",
         "showQueryEventDistribution" : false,
         "start" : 1722165818865,
         "useIngestTime" : false
      },
      "isAggregate" : false,
      "pollAfter" : 300,
      "processedBytes" : 102079232,
      "processedEvents" : 94887,
      "queryEnd" : 1722252218865,
      "querySpent" : {
         "day" : {
            "liveCost" : 0,
            "queryCount" : 1,
            "staticCost" : 8
         },
         "hour" : {
            "liveCost" : 0,
            "queryCount" : 1,
            "staticCost" : 8
         },
         "oneMinute" : {
            "liveCost" : 0,
            "queryCount" : 1,
            "staticCost" : 8
         },
         "tenMinutes" : {
            "liveCost" : 0,
            "queryCount" : 1,
            "staticCost" : 8
         }
      },
      "queryStart" : 1722165818865,
      "queuedMillis" : 0,
      "quotaTotalSpent" : {
         "day" : {
            "liveCost" : 730,
            "queryCount" : 4169,
            "staticCost" : 425
         },
         "hour" : {
            "liveCost" : 27,
            "queryCount" : 159,
            "staticCost" : 145
         },
         "oneMinute" : {
            "liveCost" : 0,
            "queryCount" : 5,
            "staticCost" : 8
         },
         "tenMinutes" : {
            "liveCost" : 5,
            "queryCount" : 33,
            "staticCost" : 55
         }
      },
      "responderVHost" : 93,
      "resultBufferSize" : 200,
      "timeMillis" : 230,
      "totalWork" : 78,
      "warnings" : [],
      "workDone" : 78
   },
   "queryEventDistribution" : {
      "events" : [
         {
            "_bucket" : "1722165300000",
            "_count" : "310"
         },
         {
            "_bucket" : "1722166200000",
            "_count" : "735"
         },
         {
            "_bucket" : "1722167100000",
            "_count" : "735"
         },
         {
            "_bucket" : "1722168000000",
            "_count" : "740"
         },
         ...
         {
            "_bucket" : "1722250800000",
            "_count" : "753"
         },
         {
            "_bucket" : "1722251700000",
            "_count" : "442"
         }
      ],
      "extraData" : {
         "bucket_first_bucket" : "1722165300000",
         "bucket_last_bucket" : "1722251700000",
         "bucket_span_humanized" : "15m",
         "bucket_span_millis" : "900000"
      }
   },
   "warnings" : []
}

The response is formatted as a JSON object with the following top-level elements:

Table: QueryResult JSON Object Fields

FieldTypeDescription
cancelledboolean Indicates whether the query has been cancelled or not. If cancelled is true then done is also always true.
doneboolean True if the static part of a search is completed. For a live query done does not mean that the set of results is complete, it just means that the static part of the search (which happens at the start of the query) is complete.
eventsarray JSON of the events as an array
fieldStatsarray An array of field statistics objects. This object includes the field name, the number of unique values, the number of occurrences, and the top values.
filesUsedarray List of the files used to fulfill the query
filterMatchesarray Entries that match the supplied filter
metaData  Metadata about the result set in the form of QueryMetaDataJson
queryEventDistribution  Information used to render the distribution graph. Only supplied when the showQueryEventDistribution is set to true.
warningsarray List of warnings returned using the WarningJson.

The MetaData field contains the number of matching events, the query boundary, and information about the attributes and their unique value domains in the response. It also contains the list of warnings generated by the query.

Heartbeats

Query timeouts are in operation for query jobs. If you do not poll a query for 90 seconds, then it stops and deletes itself. Live queries keep running for an hour without being polled.

Because there may be additional delays during polling from the client, it is recommended to include a margin of error and poll for results at a shorter interval of 45 or 30 seconds.

Queries can be kept alive with the onlyHeartbeat query parameter. For full details, including an example of usage, see Heartbeats.

Query metaData

The query metadata contains detailed information about the execution of the query, including events, workload, and duration of the query processing.

Table: Metadata JSON Object Fields

FieldTypeDescription
costs The costs of executing the query
        liveCostnumber The cost of execution of a live query
        liveCostRatenumber The rate of the live cost query
        staticCostnumber The cost for executing as a static query
        staticCostRatenumber The rate for executing static queries
digestFlow Indicates how up to date the incoming data to the query is
        ingestTimeKnownGoodinteger Latest known ingest time for digested data
        maxIngestLatencyinteger Maximum ingest latency for the returned data
        minIngestTimeIncludedinteger Minimizing ingest time for data included in the results
eventCountinteger Number of events in the result set
extraData  Additional data about the returned events
fieldOrderarray Order of the returned fields
filterQuery  Filter query structure containing a definition of the query as requested. The structure matches the supplied QueryJobInput.
isAggregateboolean True if this is an aggregate query
pollAfterinteger How soon to poll the query after it has been submitted to expected results
processedBytesinteger Number of bytes processed to return results
processedEventsinteger Number of events processed in result set
queryEndinteger Timestamp of the end of the query
querySpent Structure defining the time and duration of the returned events
        day Total per day spent executing the query
                liveCostinteger Live cost of executing the query
                queryCountinteger Number of executions of the query
                staticCostinteger Static cost of executing the query
        hour Total per hour spent executing the query
                liveCostinteger Live cost of executing the query
                queryCountinteger Number of executions of the query
                staticCostinteger Static cost of executing the query
        oneMinute Total spent per one minute
                liveCostinteger Live cost of executing the query
                queryCountinteger Number of executions of the query
                staticCostinteger Static cost of executing the query
        tenMinutes Total spent per ten minutes
                liveCostinteger Live cost of executing the query
                queryCountinteger Number of executions of the query
                staticCostinteger Static cost of executing the query
queryStartinteger Timestamp of when the query started
queuedMillisinteger Number of milliseconds the query was queued before execution
quotaTotalSpent Structure definition to the total time spent used for calculating the quota
        day Total per day spent executing the query
                liveCostinteger Live cost of executing the query
                queryCountinteger Number of executions of the query
                staticCostinteger Static cost of executing the query
        hour Total per hour spent executing the query
                liveCostinteger Live cost of executing the query
                queryCountinteger Number of executions of the query
                staticCostinteger Static cost of executing the query
        oneMinute Total spent per one minute
                liveCostinteger Live cost of executing the query
                queryCountinteger Number of executions of the query
                staticCostinteger Static cost of executing the query
        tenMinutes Total spent per ten minutes
                liveCostinteger Live cost of executing the query
                queryCountinteger Number of executions of the query
                staticCostinteger Static cost of executing the query
responderVHostinteger Number of the host returning the response
resultBufferSizeinteger Size of the buffer used to store the result
timeMillisinteger Time spent executing the query in milliseconds
totalWorkinteger Total amount of work to execute the query
warningstring Deprecated: Use warnings in the message root instead
warningsarray Use 'warnings' in the message root instead
workDoneinteger Estimate of the work done to complete the query

Query extraData

The extraData contains information about first and last bucket timestamps and a humanized version of the returned data relative time span.

Table: ExtraData JSON Object Fields

FieldTypeDescription
bucket_first_bucketstring Timestamp of the first bucket in the returned events
bucket_last_bucketstring Timestamp of the last bucket in the returned events
bucket_span_humanizedstring Human-readable description of the bucket span
bucket_span_millisstring Span of the returned buckets in milliseconds

Query warnings

Warnings when executing a query are returned in a WarningJson structure within the overall query response. The warnings are included here to avoid the whole query failing and instead highlight individual warnings within the query process. The structure of the warning is shown in the table below:

Table: Warning JSON Object Fields

FieldTypeDescription
categorystring Warnings are categorised to allow you to deal with a whole set of warnings the same way. Other values may be returned if cluster nodes are out of sync, using a newer or older version of LogScale.
classificationstring 
codestring A stable message code that can be used to compare error types or look up error descriptions.
messagestring A Human readable text describing the warning.