Export Queries API

Runs a non-aggregate search query and writes the matching events to object storage (bucket storage) instead of streaming them back in an HTTP response body. Designed for result sets too large to fit in a single HTTP response.

Important

Status: Experimental - no API stability guarantee.

Prerequisites:

  • Feature flag: ExportQueries must be enabled globally via mutation { enableFeature(feature: ExportQueries) }.

  • Bucket storage: A destination bucket must be configured on the cluster.

  • Permission: Read access to the target view (or it must be your sandbox).

  • Node role: The receiving node must be query-capable (ingest-only nodes reject the request).

Available Endpoints

Table:

HTTP MethodURIAvailabilityDescription
GET/api/v1/repositories/view/export-query/download/id  Stream the result back through LogScale and delete them from the bucket.
GET/api/v1/repositories/view/export-query/id  Poll status. Lists exported bucket objects when done.
POST/api/v1/repositories/view/export-query  Submit a query export, receive a query ID
DELETE/api/v1/repositories/view/export-query/id  Cancel an export

Submit an Export

Submit a new export job by sending a POST request with a query definition and export configuration. LogScale runs the query asynchronously and writes results to object storage. The response includes a job ID you use to poll for status or download results.

Description Submit a query export, receive a query ID  
MethodPOST /api/v1/repositories/view/export-query 
Request Data QueryJobInput, ExportConfig  
Response Data
Query ID
 
Authentication Requiredyes 
Path ArgumentsDescriptionData typeRequired?
view View or Repository stringrequired
Return Codes 
200 Request complete  
400 Bad authentication  
500 Request failed  
http
POST /api/v1/repositories/{view}/export-query

Send a JSON body with two required objects, QueryJobInput and ExportConfig:

json
{
  "queryJobInput": {
    "queryString": "#type=accesslog | statuscode >= 500",
    "start": "7d",
    "end": "now"
  },
  "exportConfig": {
    "useConfiguredExportBucket": false,
    "outputFormat": "csv"
  }
}

QueryInput is as follows, but note that isLive and includeDeletedEvents must be absent or false:

Table: QueryJobInput JSON Object Fields

FieldTypeAvailabilityRequired?DefaultDescription
allowEventSkippingboolean    Some LogScale functions, such as head() and tail(), automatically stop processing all data in the selected timeframe and exit the query early, if more data would no longer change the result of the query. This improves performance of the query. Setting allowEventSkipping to true, forces event skipping in all circumstances. For example, when the showQueryEventDistribution or doneCriteria.minimumMatchedEvents parameters are set, event skipping is disabled by default to ensure accurate event counts, but setting allowEventSkipping to true overrides this, and forces skipping.
argumentsobject    Dictionary of arguments specified in queries with ?param or ?{param=defaultValue} syntax. Provided arguments must be a simple dictionary of string values. If an argument is given explicitly as in ?query(param=value) then that value overrides values provided here.
aroundobject    Used to define the pagination of events in the result set within the given query when using pagination. Used for cursor-based paginating of filter query result. This cannot be used with aggregate results as all rows are always returned. For more information see api-search-request-around
autobucketCountinteger    Determines the number of buckets the system should create during live query searches, when no other explicit bucketing aggregate is specified (such as bucket(), timeChart()). Higher autobucket counts means finer granularity, but at the cost of increased memory usage during search.
computeFieldStatsboolean    If set to true, computeFieldStats causes LogScale to calculate a map of all fields of the result and their top ten most common values, along with both the total number of occurrences for the field, as well as its values. These results are returned as part of the QueryResult JSON object under the field fieldStats. This field is always present, but empty if not requested in the query input for the query.
dashboardFilterstring    The ID of a query. This parameter is used to apply a dashboard filter to a query. In LogScale, a dashboard filter is a regular query which is applied to all queries on a dashboard, usually prefixing the dashboard widget query. But this is not always the case, so it can also be passed to the Query API to be able to merge the query and the dashboard filter to form a complete query. When querying the Query API from other clients, that would mean that the dashboardFilter parameter can be used when you have a set of queries where the start of each query is identical. You can then extract the duplicated part of the query and pass that as a dashboardFilter instead, ensuring that it will be applied correctly even though you use a query function with special semantics that require it to be at the start of the query, such as defineTable().
dashboardFilterVersionstring    
doneCriteria(from 1.141)   
        fractionOfDataScannednumber    The query is considered done when least X percent of the input data is sampled. It should be a number in the range ]0;1] where 1 mean all events (100%) and e.g. 0.1 is approximately 10% of data that will be scanned. Sampling is done based on event size and not the number of events.
        minimumMatchedEventsinteger    The query is done once at least X events have matched the filters before the first aggregation. This can be useful if you just want to show some candidates or you consider a result to be good enough when at least X events have been considered.
        minimumResultRowsinteger    The query is done once at least X rows are present in the output. Note that if this criterion is used in conjunction with an aggregate function there is no guarantee for how good the output results are as we don't know how much data was processed to produce the result. But it is useful for e.g. find 10 events that match this filter, similar to tail(), but does not guarantee that the most recent matches are returned.
end    The end date and time. This parameter tells LogScale not to return results from after this date and time. See how to specify a time.
extraLogFieldsobject   

Extra fields set on a query that are logged to the audit and activity logs as customKey.

The following rules apply to both the key and value fields:

  • Maximum length is 512 characters.

  • Only the following characters are permitted: a-z, A-Z, 0-9, and the special characters , - . _

To learn more, see extraLogFields.

includeDeletedEventsboolean    
ingestEnd    Specifies the end time based on when the data was ingested. See how to specify a time.
ingestStart    Specifies the start time based on when the data was ingested. See how to specify a time.
isAlertQueryboolean    Indicates whether the query comes from an alert or not.
isInteractiveboolean    Whether the search is being run interactively, i.e., from with the LogScale or another UI.
isLiveboolean    Sets whether this query is live. Defaults to false . Live queries are continuously updated.
isRepeatingSubqueryboolean    
languageVersionstring    The version of the query language to use
nextRunIntervalobject   

This parameter enables you to supply a hint to the query engine that the next run of the query will have an interval specified by the given start and end values.

This information may be used to speed up the repeated runs of the query by allowing partial reuse of query results.

Usage example:

json
{ [...] "nextRunInterval": { "start": 1764765006226, "end": 1764851406227 } }
noResultUntilDoneboolean    If true, no events are included in responses to polls until the static part of the query has completed, at which point the full result set is returned. Metadata and progress information are updated on each poll.
queryStringstring Yes  The actual query. See Query Language Syntax for details.
showQueryEventDistributionboolean    If true LogScale will return an additional resultset containing a histogram of the number of events over the time interval. Use results instead. If both this field and results are specified, results takes precedence.
start    The start date and time. This parameter tells LogScale not to return results from before this date and time. See how to specify a time.
timeZonestring    The timezone to be used when returning dates.
timeZoneOffsetMinutesinteger    Set the time zone offset used for bucket() and timeChart() time slices, which is significant if the corresponding span is multiples of days. Defaults to 0 (UTC); positive numbers are to the east of UTC, so for UTC+01:00 timezone the value 60 should be passed.
useIngestTimeboolean    When set to true, uses the ingest time rather than event timestamp as the basis for the time span, to control sorting/bucketing for functions like bucket(). Note that when true, the time window set by both ingestStart and ingestEnd will be applied, in addition to the time window set by start and end, the effective time window being the intersection of these two time windows.
wantsAutoBucketingboolean    

ExportConfig is as follows:

Table: ExportConfig JSON Object Fields

FieldTypeAvailabilityRequired?DefaultDescription
outputFormatstring Yes  Output format is one of json, ndjson, csv, or txt.
useConfiguredExportBucketboolean Yes  useConfiguredExportBucket determines the destination:
  • false - writes to the cluster's own bucket storage; use /download to retrieve the result

  • true - writes to a separate export bucket you configure (via S3_EXPORT_BUCKET, GCP_EXPORT_BUCKET, or AZURE_EXPORT_BUCKET); you read the objects directly - /download does not work for this mode


Example:

Mac OS or Linux (curl)
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{
  "queryJobInput": {
    "queryString": "#type=accesslog | statuscode &gt;= 500",
    "start": "7d",
    "end": "now"
  },
  "exportConfig": {
    "useConfiguredExportBucket": false,
    "outputFormat": "csv"
  }
}
EOF
Mac OS or Linux (curl) One-line
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{
  "queryJobInput": {
    "queryString": "#type=accesslog | statuscode &gt;= 500",
    "start": "7d",
    "end": "now"
  },
  "exportConfig": {
    "useConfiguredExportBucket": false,
    "outputFormat": "csv"
  }
}
EOF
Windows Cmd and curl
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{ ^
  "queryJobInput": { ^
    "queryString": "#type=accesslog | statuscode &gt;= 500", ^
    "start": "7d", ^
    "end": "now" ^
  }, ^
  "exportConfig": { ^
    "useConfiguredExportBucket": false, ^
    "outputFormat": "csv" ^
  } ^
} '
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{
  "queryJobInput": {
    "queryString": "#type=accesslog | statuscode &gt;= 500",
    "start": "7d",
    "end": "now"
  },
  "exportConfig": {
    "useConfiguredExportBucket": false,
    "outputFormat": "csv"
  }
}'
    "$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query';

my $json = '{
  "queryJobInput": {
    "queryString": "#type=accesslog | statuscode &gt;= 500",
    "start": "7d",
    "end": "now"
  },
  "exportConfig": {
    "useConfiguredExportBucket": false,
    "outputFormat": "csv"
  }
}';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query'
mydata = r'''{
  "queryJobInput": {
    "queryString": "#type=accesslog | statuscode &gt;= 500",
    "start": "7d",
    "end": "now"
  },
  "exportConfig": {
    "useConfiguredExportBucket": false,
    "outputFormat": "csv"
  }
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

const data = JSON.stringify(
    {
  "queryJobInput": {
    "queryString": "#type=accesslog | statuscode &gt;= 500",
    "start": "7d",
    "end": "now"
  },
  "exportConfig": {
    "useConfiguredExportBucket": false,
    "outputFormat": "csv"
  }
}
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL',
  path: 'api/v1/repositories/$YOUR_VIEW/export-query',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Polling

Poll the status of an export job using the job ID returned by the submit request. When the job completes, the response lists the objects written to object storage.

Description Poll status. Lists exported bucket objects when done.  
MethodGET /api/v1/repositories/view/export-query/id 
Authentication Requiredyes 
Path ArgumentsDescriptionData typeRequired?
id ID of query stringrequired
view View stringrequired
Return Codes 
200 Request complete  
400 Bad authentication  
500 Request failed  
http
GET /api/v1/repositories/{view}/export-query/{id}

Example:

Mac OS or Linux (curl)
shell
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID \
    -H "Authorization: Bearer $TOKEN"
Mac OS or Linux (curl) One-line
shell
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID \
    -H "Authorization: Bearer $TOKEN"
Windows Cmd and curl
shell
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID ^
    -H "Authorization: Bearer $TOKEN"
Windows Powershell and curl
powershell
curl.exe -X GET 
    -H "Authorization: Bearer $TOKEN"
    "$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID';

my $req = HTTP::Request->new("GET", $uri );

$req->header("Authorization" => "Bearer $TOKEN");


my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID'

resp = requests.get(url,
                     headers = {
   "Authorization" : "Bearer $TOKEN"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

let request = https.get('$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID', (res) => {
  if (res.statusCode !== 200) {
    console.error(`Error from server. Code: ${res.statusCode}`);
    res.resume();
    return;
  }

  let data = '';

  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('close', () => {
    console.log('Response:');
    console.log(JSON.parse(data));
  });
});

    return(undef,undef);
}

Downloading

Download a completed export result using the job ID. Use this endpoint after polling confirms the job is complete to retrieve the exported data from object storage.

Description Stream the result back through LogScale and delete them from the bucket.  
MethodGET /api/v1/repositories/view/export-query/download/id 
Authentication Requiredyes 
Path ArgumentsDescriptionData typeRequired?
id ID of query stringrequired
view View stringrequired
Return Codes 
200 Request complete  
400 Bad authentication  
500 Request failed  
http
GET /api/v1/repositories/{view}/export-query/download/{id}

Example:

Mac OS or Linux (curl)
shell
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Mac OS or Linux (curl) One-line
shell
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Windows Cmd and curl
shell
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"
Windows Powershell and curl
powershell
curl.exe -X GET 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    "$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID';

my $req = HTTP::Request->new("GET", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");


my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID'

resp = requests.get(url,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

let request = https.get('$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/download/$QUERY_ID', (res) => {
  if (res.statusCode !== 200) {
    console.error(`Error from server. Code: ${res.statusCode}`);
    res.resume();
    return;
  }

  let data = '';

  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('close', () => {
    console.log('Response:');
    console.log(JSON.parse(data));
  });
});

    return(undef,undef);
}

Cancel an Export

Cancel a running or pending export job using the job ID. Cancelled jobs stop processing and do not write further output to object storage.

Description Cancel an export  
MethodDELETE /api/v1/repositories/view/export-query/id 
Authentication Requiredyes 
Path ArgumentsDescriptionData typeRequired?
id ID of query stringrequired
view View or Repository stringrequired
Return Codes 
200 Request complete  
400 Bad authentication  
500 Request failed  
http
DELETE /api/v1/repositories/{view}/export-query/{id}

Example:

Mac OS or Linux (curl)
shell
curl -v -X DELETE $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID \
    -H "Authorization: Bearer $TOKEN"
Mac OS or Linux (curl) One-line
shell
curl -v -X DELETE $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID \
    -H "Authorization: Bearer $TOKEN"
Windows Cmd and curl
shell
curl -v -X DELETE $YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID ^
    -H "Authorization: Bearer $TOKEN"
Windows Powershell and curl
powershell
curl.exe -X DELETE 
    -H "Authorization: Bearer $TOKEN"
    "$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID';

my $req = HTTP::Request->new("DELETE", $uri );

$req->header("Authorization" => "Bearer $TOKEN");


my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID'

resp = requests.delete(url,
                     headers = {
   "Authorization" : "Bearer $TOKEN"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

let request = https.delete('$YOUR_LOGSCALE_URL/api/v1/repositories/$YOUR_VIEW/export-query/$QUERY_ID', (res) => {
  if (res.statusCode !== 200) {
    console.error(`Error from server. Code: ${res.statusCode}`);
    res.resume();
    return;
  }

  let data = '';

  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('close', () => {
    console.log('Response:');
    console.log(JSON.parse(data));
  });
});

    return(undef,undef);
}