Dynamic Configuration

Dynamic Configuration allows for setting configuration values for the cluster while it is running.

Table: Dynamic Configuration Parameters Table

VariableDefault ValueAvailabilityDescription
BucketStorageKeySchemeVersion2introduced in 1.51 Allows to set a new format for the keys (file names) placed in the bucket. When the new format is applied, the listing of files only happens for the prefixes tmp/ and globalsnapshots/. The new format is applied only to buckets created after this dynamic configuration has been set to 2.
BucketStorageWriteVersion3introduced in 1.68 Sets the format for files written to bucket storage to use a format that allows files larger than 2GB and incurs less memory pressure when decrypting files during download from the bucket.
FdrEnablefalseintroduced in 1.40 Used by administrators to turn FDR polling on/off on the entire cluster with a single update.
FdrExcludedNodesemptyintroduced in 1.40 Used by administrators to exclude specific nodes from polling from FDR.
FdrMaxNodes5introduced in 1.40 Used by administrators to put a cap on how many nodes at most should simultaneously poll data from the same FDR feed.
GroupDefaultLimit20,000introduced in 1.51 Default value for the limit parameter in groupBy(), selfJoin() and some other functions, when not specified. See Limits & Standards for details.
GroupMaxLimit1,000,000introduced in 1.51 Max value for the limit parameter in groupBy() function. See Limits & Standards for details.
JoinRowLimit200,000introduced in 1.51

Max number of rows that join() and selfJoin() can return. It can be set using GraphQL.

Used as an alternative to the environment variable MAX_JOIN_LIMIT

.
LiveQueryMemoryLimit100,000,000introduced in 1.51 Determines how much memory a live query can consume during its execution.
MaxIngestRequestSize33,554,432introduced in 1.64 Size limit of ingest requests after content-encoding has been applied; expressed in bytes.
QueryMemoryLimit100,000,000introduced in 1.51 Determines how much memory a non-live query can consume during its execution.
QueryResultRowCountLimit  introduced in 1.51 Globally limits how many events a query can return. This flag can be set by administrators through GraphQL.
StateRowLimit20,000introduced in 1.51

Maximum number of rows allowed in functions.

Used as an alternative for the environment variable MAX_STATE_LIMIT.

TargetMaxRateForDatasource2introduced in 1.51 Sets the target maximum rate of ingest for each shard of a datasource.
UndersizedMergingRetentionPercentage20introduced in 1.63.1 When selecting undersized segments to merge, this setting controls how wide a time span can be merged together. The setting is interpreted as a percentage of the repository's retention by time setting. A reasonable range is 0 through to 90.

Getting Dynamic Configuration List

To obtain a list of all the available dynamic configurations, use the dynamicConfigs() GraphQL query

Raw
graphql
query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}
Mac OS or Linux (curl)
shell
curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}
EOF
Mac OS or Linux (curl) One-line
shell
curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Windows Cmd and curl
cmd
curl -v -X POST $YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "query { ^
  dynamicConfigs { ^
    dynamicConfigKey,dynamicConfigValue ^
  } ^
}" ^
} '
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}'
"$YOUR_LOGSCALE_URL/graphql"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/graphql';
my $json = '{"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

const data = JSON.stringify(
    {"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL/graphql',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

This will return a list of key/value pairs for the configuration key and current setting. For example:

graphql
{
  "data": {
    "dynamicConfigs": [
      {
        "dynamicConfigKey": "MaxIngestRequestSize",
        "dynamicConfigValue": "None"
      },
      {
        "dynamicConfigKey": "JoinRowLimit",
        "dynamicConfigValue": "200000"
      },
      {
        "dynamicConfigKey": "JoinDefaultLimit",
        "dynamicConfigValue": "100000"
      },
      ...
    ]
  }
}

The exact list of configurable parameters will depend on the version, feature flags and environment.

Setting a Dynamic Configuration Value

Important

Changing Dynamic Config settings will instantly change the configuration setting and alter the operation of your LogScale instance. Contact Support if you need advice on these settings.

To set a Dynamic Config value, use the setDynamicConfig() mutation:

Raw
graphql
mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: "VALUE_FOR_CONFIG" })
}
Mac OS or Linux (curl)
shell
curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}
EOF
Mac OS or Linux (curl) One-line
shell
curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Windows Cmd and curl
cmd
curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "mutation { ^
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" }) ^
}" ^
} '
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}'
"http://$YOUR_LOGSCALE_URL/graphql"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = 'http://$YOUR_LOGSCALE_URL/graphql';
my $json = '{"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = 'http://$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

const data = JSON.stringify(
    {"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}
);


const options = {
  hostname: 'http://$YOUR_LOGSCALE_URL/graphql',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();
Example Responses
Success (200)
json
{
   "data" : {
      "setDynamicConfig" : true
   }
}

For example:

Raw
graphql
mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: "30000" })
}
Mac OS or Linux (curl)
shell
curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}
EOF
Mac OS or Linux (curl) One-line
shell
curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Windows Cmd and curl
cmd
curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "mutation { ^
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" }) ^
}" ^
} '
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}'
"http://$YOUR_LOGSCALE_URL/graphql"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = 'http://$YOUR_LOGSCALE_URL/graphql';
my $json = '{"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = 'http://$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

const data = JSON.stringify(
    {"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}
);


const options = {
  hostname: 'http://$YOUR_LOGSCALE_URL/graphql',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();
Example Responses
Success (200)
json
{
   "data" : {
      "setDynamicConfig" : true
   }
}