Export Queries API
Runs a non-aggregate search query and writes the matching events to object storage (bucket storage) instead of streaming them back in an HTTP response body. Designed for result sets too large to fit in a single HTTP response.
Important
Status: Experimental - no API stability guarantee.
Prerequisites:
Feature flag: ExportQueries must be enabled globally via
mutation { enableFeature(feature: ExportQueries) }.Bucket storage: A destination bucket must be configured on the cluster.
Permission: Read access to the target view (or it must be your sandbox).
Node role: The receiving node must be query-capable (ingest-only nodes reject the request).
Available Endpoints
Table:
| HTTP Method | URI | Availability | Description |
|---|---|---|---|
GET | /api/v1/repositories/ | Â | Stream the result back through LogScale and delete them from the bucket. |
GET | /api/v1/repositories/ | Â | Poll status. Lists exported bucket objects when done. |
POST | /api/v1/repositories/ | Â | Submit a query export, receive a query ID |
DELETE | /api/v1/repositories/ | Â | Cancel an export |
Submit an Export
Submit a new export job by sending a POST request with a query definition
and export configuration. LogScale runs the query asynchronously and
writes results to object storage. The response includes a job ID you use
to poll for status or download results.
| Description | Submit a query export, receive a query ID | Â | |
| Method | POST /api/v1/repositories/ | Â | |
| Request Data | QueryJobInput, ExportConfig | Â | |
| Response Data | | Â | |
| Authentication Required | yes | Â | |
| Path Arguments | Description | Data type | Required? |
view | View or Repository | string | required |
| Return Codes | Â | ||
| 200 | Request complete | Â | |
| 400 | Bad authentication | Â | |
| 500 | Request failed | Â | |
POST /api/v1/repositories/{view}/export-query
Send a JSON body with two required objects,
QueryJobInput and ExportConfig:
{
"queryJobInput": {
"queryString": "#type=accesslog | statuscode >= 500",
"start": "7d",
"end": "now"
},
"exportConfig": {
"useConfiguredExportBucket": false,
"outputFormat": "csv"
}
}
QueryInput is as follows, but note that
isLive and includeDeletedEvents must
be absent or false:
Table: QueryJobInput JSON Object Fields
| Field | Type | Availability | Required? | Default | Description |
|---|---|---|---|---|---|
| allowEventSkipping | boolean | Â | Â | Â |
Some LogScale functions, such as
head() and tail(),
automatically stop processing all data in the selected timeframe
and exit the query early, if more data would no longer change
the result of the query. This improves performance of the query.
Setting allowEventSkipping to
true, forces event skipping in all
circumstances. For example, when the
showQueryEventDistribution or
doneCriteria.minimumMatchedEvents parameters
are set, event skipping is disabled by default to ensure
accurate event counts, but setting
allowEventSkipping to true
overrides this, and forces skipping.
|
| arguments | object | Â | Â | Â | Dictionary of arguments specified in queries with ?param or ?{param=defaultValue} syntax. Provided arguments must be a simple dictionary of string values. If an argument is given explicitly as in ?query(param=value) then that value overrides values provided here. |
| around | object | Â | Â | Â | Used to define the pagination of events in the result set within the given query when using pagination. Used for cursor-based paginating of filter query result. This cannot be used with aggregate results as all rows are always returned. For more information see api-search-request-around |
| autobucketCount | integer | Â | Â | Â |
Determines the number of buckets the system should create during
live query searches, when no other explicit bucketing aggregate
is specified (such as bucket(),
timeChart()). Higher autobucket counts
means finer granularity, but at the cost of increased memory
usage during search.
|
| computeFieldStats | boolean | Â | Â | Â |
If set to true,
computeFieldStats causes LogScale to
calculate a map of all fields of the result and their top ten
most common values, along with both the total number of
occurrences for the field, as well as its values. These results
are returned as part of the QueryResult JSON
object under the field fieldStats. This field
is always present, but empty if not requested in the query input
for the query.
|
| dashboardFilter | string | Â | Â | Â |
The ID of a query. This parameter is used to apply a dashboard
filter to a query. In LogScale, a dashboard filter is a
regular query which is applied to all queries on a dashboard,
usually prefixing the dashboard widget query. But this is not
always the case, so it can also be passed to the Query API to be
able to merge the query and the dashboard filter to form a
complete query. When querying the Query API from other clients,
that would mean that the dashboardFilter
parameter can be used when you have a set of queries where the
start of each query is identical. You can then extract the
duplicated part of the query and pass that as a
dashboardFilter instead, ensuring that it will
be applied correctly even though you use a query function with
special semantics that require it to be at the start of the
query, such as defineTable().
|
| dashboardFilterVersion | string | Â | Â | Â | Â |
| doneCriteria | | (from 1.141) | Â | Â | Â |
| Â Â Â Â Â Â Â Â fractionOfDataScanned | number | Â | Â | Â | The query is considered done when least X percent of the input data is sampled. It should be a number in the range ]0;1] where 1 mean all events (100%) and e.g. 0.1 is approximately 10% of data that will be scanned. Sampling is done based on event size and not the number of events. |
| Â Â Â Â Â Â Â Â minimumMatchedEvents | integer | Â | Â | Â | The query is done once at least X events have matched the filters before the first aggregation. This can be useful if you just want to show some candidates or you consider a result to be good enough when at least X events have been considered. |
| Â Â Â Â Â Â Â Â minimumResultRows | integer | Â | Â | Â | The query is done once at least X rows are present in the output. Note that if this criterion is used in conjunction with an aggregate function there is no guarantee for how good the output results are as we don't know how much data was processed to produce the result. But it is useful for e.g. find 10 events that match this filter, similar to tail(), but does not guarantee that the most recent matches are returned. |
| end | | Â | Â | Â | The end date and time. This parameter tells LogScale not to return results from after this date and time. See how to specify a time. |
| extraLogFields | object | Â | Â | Â |
Extra fields set on a query that are logged to the audit and activity logs as customKey. The following rules apply to both the key and value fields:
To learn more, see extraLogFields. |
| includeDeletedEvents | boolean | Â | Â | Â | Â |
| ingestEnd | | Â | Â | Â | Specifies the end time based on when the data was ingested. See how to specify a time. |
| ingestStart | | Â | Â | Â | Specifies the start time based on when the data was ingested. See how to specify a time. |
| isAlertQuery | boolean | Â | Â | Â | Indicates whether the query comes from an alert or not. |
| isInteractive | boolean | Â | Â | Â | Whether the search is being run interactively, i.e., from with the LogScale or another UI. |
| isLive | boolean | Â | Â | Â | Sets whether this query is live. Defaults to false . Live queries are continuously updated. |
| isRepeatingSubquery | boolean | Â | Â | Â | Â |
| languageVersion | string | Â | Â | Â | The version of the query language to use |
| nextRunInterval | object | Â | Â | Â |
This parameter enables you to supply a hint to the query
engine that the next run of
the query will have an interval specified by the given
This information may be used to speed up the repeated runs of the query by allowing partial reuse of query results. Usage example: json |
| noResultUntilDone | boolean | Â | Â | Â |
If true, no events are included in responses to
polls until the static part of the query has completed, at which
point the full result set is returned. Metadata and progress
information are updated on each poll.
|
| queryString | string | Â | Yes | Â | The actual query. See Query Language Syntax for details. |
| showQueryEventDistribution | boolean | Â | Â | Â |
If true LogScale will return an additional resultset
containing a histogram of the number of events over the time
interval. Use results
instead. If both this field and
results are specified,
results takes
precedence.
|
| start | | Â | Â | Â | The start date and time. This parameter tells LogScale not to return results from before this date and time. See how to specify a time. |
| timeZone | string | Â | Â | Â | The timezone to be used when returning dates. |
| timeZoneOffsetMinutes | integer | Â | Â | Â |
Set the time zone offset used for bucket()
and timeChart() time slices, which is
significant if the corresponding
span is multiples of days.
Defaults to 0 (UTC);
positive numbers are to the east of UTC, so for
UTC+01:00 timezone the value
60 should be passed.
|
| useIngestTime | boolean | Â | Â | Â |
When set to true, uses the ingest time rather
than event timestamp as the basis for the time span, to control
sorting/bucketing for functions like
bucket(). Note that when
true, the time window set by both
ingestStart and ingestEnd
will be applied, in addition to the time window set by
start and end, the
effective time window being the intersection of these two time
windows.
|
| wantsAutoBucketing | boolean | Â | Â | Â | Â |
ExportConfig is as follows:
Table: ExportConfig JSON Object Fields
| Field | Type | Availability | Required? | Default | Description |
|---|---|---|---|---|---|
| outputFormat | string | Â | Yes | Â |
Output format is one of json,
ndjson, csv, or
txt.
|
| useConfiguredExportBucket | boolean | Â | Yes | Â |
useConfiguredExportBucket
determines the destination:
|
Example:
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d @- << EOF
{
"queryJobInput": {
"queryString": "#type=accesslog | statuscode >= 500",
"start": "7d",
"end": "now"
},
"exportConfig": {
"useConfiguredExportBucket": false,
"outputFormat": "csv"
}
}
EOFcurl -v -X POST $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d @- << EOF
{
"queryJobInput": {
"queryString": "#type=accesslog | statuscode >= 500",
"start": "7d",
"end": "now"
},
"exportConfig": {
"useConfiguredExportBucket": false,
"outputFormat": "csv"
}
}
EOFcurl -v -X POST $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query ^
-H "Authorization: Bearer $TOKEN" ^
-H "Content-Type: application/json" ^
-d @'{ ^
"queryJobInput": { ^
"queryString": "#type=accesslog | statuscode >= 500", ^
"start": "7d", ^
"end": "now" ^
}, ^
"exportConfig": { ^
"useConfiguredExportBucket": false, ^
"outputFormat": "csv" ^
} ^
} 'curl.exe -X POST
-H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json"
-d '{
"queryJobInput": {
"queryString": "#type=accesslog | statuscode >= 500",
"start": "7d",
"end": "now"
},
"exportConfig": {
"useConfiguredExportBucket": false,
"outputFormat": "csv"
}
}'
"$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query"#!/usr/bin/perl
use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query';
my $json = '{
"queryJobInput": {
"queryString": "#type=accesslog | statuscode >= 500",
"start": "7d",
"end": "now"
},
"exportConfig": {
"useConfiguredExportBucket": false,
"outputFormat": "csv"
}
}';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";#! /usr/local/bin/python3
import requests
url = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query'
mydata = r'''{
"queryJobInput": {
"queryString": "#type=accesslog | statuscode >= 500",
"start": "7d",
"end": "now"
},
"exportConfig": {
"useConfiguredExportBucket": false,
"outputFormat": "csv"
}
}'''
resp = requests.post(url,
data = mydata,
headers = {
"Authorization" : "Bearer $TOKEN",
"Content-Type" : "application/json"
}
)
print(resp.text)const https = require('https');
const data = JSON.stringify(
{
"queryJobInput": {
"queryString": "#type=accesslog | statuscode >= 500",
"start": "7d",
"end": "now"
},
"exportConfig": {
"useConfiguredExportBucket": false,
"outputFormat": "csv"
}
}
);
const options = {
hostname: '$YOUR_LOGSCALE_URL',
path: 'api/v1/repositories/$YOUR_VIEW/export-query',
port: 443,
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': data.length,
Authorization: 'BEARER ' + process.env.TOKEN,
'User-Agent': 'Node',
},
};
const req = https.request(options, (res) => {
let data = '';
console.log(`statusCode: ${res.statusCode}`);
res.on('data', (d) => {
data += d;
});
res.on('end', () => {
console.log(JSON.parse(data).data);
});
});
req.on('error', (error) => {
console.error(error);
});
req.write(data);
req.end();Polling
Poll the status of an export job using the job ID returned by the submit request. When the job completes, the response lists the objects written to object storage.
| Description | Poll status. Lists exported bucket objects when done. | Â | |
| Method | GET /api/v1/repositories/ | Â | |
| Authentication Required | yes | Â | |
| Path Arguments | Description | Data type | Required? |
id | ID of query | string | required |
view | View | string | required |
| Return Codes | Â | ||
| 200 | Request complete | Â | |
| 400 | Bad authentication | Â | |
| 500 | Request failed | Â | |
GET /api/v1/repositories/{view}/export-query/{id}Example:
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID \
-H "Authorization: Bearer $TOKEN"curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID \
-H "Authorization: Bearer $TOKEN"curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID ^
-H "Authorization: Bearer $TOKEN"curl.exe -X GET
-H "Authorization: Bearer $TOKEN"
"$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID"#!/usr/bin/perl
use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID';
my $req = HTTP::Request->new("GET", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";#! /usr/local/bin/python3
import requests
url = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID'
resp = requests.get(url,
headers = {
"Authorization" : "Bearer $TOKEN"
}
)
print(resp.text)const https = require('https');
let request = https.get('$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID', (res) => {
if (res.statusCode !== 200) {
console.error(`Error from server. Code: ${res.statusCode}`);
res.resume();
return;
}
let data = '';
res.on('data', (chunk) => {
data += chunk;
});
res.on('close', () => {
console.log('Response:');
console.log(JSON.parse(data));
});
});
return(undef,undef);
}Downloading
Download a completed export result using the job ID. Use this endpoint after polling confirms the job is complete to retrieve the exported data from object storage.
| Description | Stream the result back through LogScale and delete them from the bucket. | Â | |
| Method | GET /api/v1/repositories/ | Â | |
| Authentication Required | yes | Â | |
| Path Arguments | Description | Data type | Required? |
id | ID of query | string | required |
view | View | string | required |
| Return Codes | Â | ||
| 200 | Request complete | Â | |
| 400 | Bad authentication | Â | |
| 500 | Request failed | Â | |
GET /api/v1/repositories/{view}/export-query/download/{id}Example:
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json"curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json"curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID ^
-H "Authorization: Bearer $TOKEN" ^
-H "Content-Type: application/json"curl.exe -X GET
-H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json"
"$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID"#!/usr/bin/perl
use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID';
my $req = HTTP::Request->new("GET", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";#! /usr/local/bin/python3
import requests
url = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID'
resp = requests.get(url,
headers = {
"Authorization" : "Bearer $TOKEN",
"Content-Type" : "application/json"
}
)
print(resp.text)const https = require('https');
let request = https.get('$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID', (res) => {
if (res.statusCode !== 200) {
console.error(`Error from server. Code: ${res.statusCode}`);
res.resume();
return;
}
let data = '';
res.on('data', (chunk) => {
data += chunk;
});
res.on('close', () => {
console.log('Response:');
console.log(JSON.parse(data));
});
});
return(undef,undef);
}Cancel an Export
Cancel a running or pending export job using the job ID. Cancelled jobs stop processing and do not write further output to object storage.
| Description | Cancel an export | Â | |
| Method | DELETE /api/v1/repositories/ | Â | |
| Authentication Required | yes | Â | |
| Path Arguments | Description | Data type | Required? |
id | ID of query | string | required |
view | View or Repository | string | required |
| Return Codes | Â | ||
| 200 | Request complete | Â | |
| 400 | Bad authentication | Â | |
| 500 | Request failed | Â | |
DELETE /api/v1/repositories/{view}/export-query/{id}Example:
curl -v -X DELETE $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID \
-H "Authorization: Bearer $TOKEN"curl -v -X DELETE $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID \
-H "Authorization: Bearer $TOKEN"curl -v -X DELETE $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID ^
-H "Authorization: Bearer $TOKEN"curl.exe -X DELETE
-H "Authorization: Bearer $TOKEN"
"$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID"#!/usr/bin/perl
use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID';
my $req = HTTP::Request->new("DELETE", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";#! /usr/local/bin/python3
import requests
url = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID'
resp = requests.delete(url,
headers = {
"Authorization" : "Bearer $TOKEN"
}
)
print(resp.text)const https = require('https');
let request = https.delete('$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID', (res) => {
if (res.statusCode !== 200) {
console.error(`Error from server. Code: ${res.statusCode}`);
res.resume();
return;
}
let data = '';
res.on('data', (chunk) => {
data += chunk;
});
res.on('close', () => {
console.log('Response:');
console.log(JSON.parse(data));
});
});
return(undef,undef);
}