Dynamic Configuration

Dynamic Configuration allows for setting configuration values for the cluster while it is running.

Table: Dynamic Configuration Parameters Table

Variable	Default Value	Description
`BucketStorageKeySchemeVersion`	2	Allows to set a new format for the keys (file names) placed in the bucket. When the new format is applied, the listing of files only happens for the prefixes `tmp/` and `globalsnapshots/`. The new format is applied only to buckets created after this dynamic configuration has been set to `2`.
`BucketStorageUploadInfrequentThresholdDays`		Sets the minimum number of days of remaining retention for the data in order to switch from the default "S3 Standard" to the storage class "S3 Intelligent-Tiering" in AWS S3.
`BucketStorageWriteVersion`	3	Sets the format for files written to bucket storage to use a format that allows files larger than 2GB and incurs less memory pressure when decrypting files during download from the bucket.
`FdrEnable`	false	Used by administrators to turn FDR polling on/off on the entire cluster with a single update.
`FdrExcludedNodes`	empty	Used by administrators to exclude specific nodes from polling from FDR.
`FdrMaxNodes`	5	Used by administrators to put a cap on how many nodes at most should simultaneously poll data from the same FDR feed.
`FlushSegmentsAndGlobalOnShutdown`	false	When set, and when `USING_EPHEMERAL_DISKS` environment variable is set to true, forces all in-progress segments to be closed and uploaded to bucket, and also forces a write (and upload) of global snapshot during shutdown. When not set, this avoids the extra work and thus time shutting down from flushing very recent segments, as those can then be resumed on next boot, assuming that next boot continues on the same Kafka epoch. When set to the default, allows faster shutdown.
`GroupDefaultLimit`	20,000	Default value for the `limit` parameter in `groupBy()`, `selfJoin()` and some other functions, when not specified. See Limits & Standards for details.
`GroupMaxLimit`	1,000,000	Max value for the `limit` parameter in `groupBy()` function. See Limits & Standards for details.
`JoinRowLimit`	200,000	Max number of rows that `join()` and `selfJoin()` can return. It can be set using GraphQL. Used as an alternative to the environment variable `MAX_JOIN_LIMIT` .
`LiveQueryMemoryLimit`	100,000,000	Determines how much memory a live query can consume during its execution.
`MaxIngestRequestSize`	33,554,432	Size limit of ingest requests after content-encoding has been applied; expressed in bytes.
`QueryMemoryLimit`	100,000,000	Determines how much memory a non-live query can consume during its execution.
`QueryPartitionAutoBalance`		Turns on/off automatic balancing of query partitions across nodes. It is used whenever there are changes to the set of live nodes tasked with doing query coordination in the cluster. Existing live queries will be migrated to the new preferred query coordinator node. Queries that do not currently support migration — such as those that have not completed their historical part and those that involve Join Query Functions — will not be migrated but need to be resubmitted. You can trigger a rebalance using the GraphQL mutation optimizeQueryPartitions.
`QueryResultRowCountLimit`		Globally limits how many events a query can return. This flag can be set by administrators through GraphQL.
`StateRowLimit`	20,000	Maximum number of rows allowed in functions. Used as an alternative for the environment variable `MAX_STATE_LIMIT`.
`StaticQueryFractionOfCores`		Limits queries from one organization (user on single-organization clusters) to run on a certain percentage of mapper threads at most, effectively throttling queries to prevent one organization from consuming all capacity.
`TargetMaxRateForDatasource`	2	Sets the target maximum rate of ingest for each shard of a datasource.
`UndersizedMergingRetentionPercentage`	20	When selecting undersized segments to merge, this setting controls how wide a time span can be merged together. The setting is interpreted as a percentage of the repository's retention by time setting. A reasonable range is `0` through to `90`.

Getting Dynamic Configuration List

To obtain a list of all the available dynamic configurations, use the dynamicConfigs() GraphQL query

Raw

graphql

query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}
EOF

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Windows Cmd and curl

cmd

curl -v -X POST $YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "query { ^
  dynamicConfigs { ^
    dynamicConfigKey,dynamicConfigValue ^
  } ^
}" ^
} '

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}'
"$YOUR_LOGSCALE_URL/graphql"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/graphql';
my $json = '{"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');

const data = JSON.stringify(
    {"query" : "query {
  dynamicConfigs {
    dynamicConfigKey,dynamicConfigValue
  }
}"
}
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL/graphql',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

This will return a list of key/value pairs for the configuration key and current setting. For example:

graphql

{
  "data": {
    "dynamicConfigs": [
      {
        "dynamicConfigKey": "MaxIngestRequestSize",
        "dynamicConfigValue": "None"
      },
      {
        "dynamicConfigKey": "JoinRowLimit",
        "dynamicConfigValue": "200000"
      },
      {
        "dynamicConfigKey": "JoinDefaultLimit",
        "dynamicConfigValue": "100000"
      },
      ...
    ]
  }
}

The exact list of configurable parameters will depend on the version, feature flags and environment.

Setting a Dynamic Configuration Value

Important

Changing Dynamic Config settings will instantly change the configuration setting and alter the operation of your LogScale instance. Contact Support if you need advice on these settings.

To set a Dynamic Config value, use the setDynamicConfig() mutation:

Raw

graphql

mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: "VALUE_FOR_CONFIG" })
}

Mac OS or Linux (curl)

shell

curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}
EOF

Mac OS or Linux (curl) One-line

shell

curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Windows Cmd and curl

cmd

curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "mutation { ^
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" }) ^
}" ^
} '

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}'
"http://$YOUR_LOGSCALE_URL/graphql"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = 'http://$YOUR_LOGSCALE_URL/graphql';
my $json = '{"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = 'http://$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');

const data = JSON.stringify(
    {"query" : "mutation {
  setDynamicConfig(input: { config: NAME_OF_CONFIG, value: \"VALUE_FOR_CONFIG\" })
}"
}
);


const options = {
  hostname: 'http://$YOUR_LOGSCALE_URL/graphql',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Example Responses

Success (200)

json

{
   "data" : {
      "setDynamicConfig" : true
   }
}

For example:

Raw

graphql

mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: "30000" })
}

Mac OS or Linux (curl)

shell

curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}
EOF

Mac OS or Linux (curl) One-line

shell

curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Windows Cmd and curl

cmd

curl -v -X POST http://$YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "mutation { ^
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" }) ^
}" ^
} '

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}'
"http://$YOUR_LOGSCALE_URL/graphql"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = 'http://$YOUR_LOGSCALE_URL/graphql';
my $json = '{"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = 'http://$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');

const data = JSON.stringify(
    {"query" : "mutation {
  setDynamicConfig(input: { config: GroupDefaultLimit, value: \"30000\" })
}"
}
);


const options = {
  hostname: 'http://$YOUR_LOGSCALE_URL/graphql',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Example Responses

Success (200)

json

{
   "data" : {
      "setDynamicConfig" : true
   }
}

Falcon LogScale Self-Hosted 1.107.0-1.112.0

Self-Hosted Overview

Installing LogScale

Self-Hosted Admin.

Organization Essentials

Configuring Security

Authentication & Identity Providers

Users & Permissions

Cluster Management

Configuration Settings

Ingesting Data

LogScale Configuration Parameters

LogScale URLs & Endpoints

Limits & Standards

Data Analysis 1.107.0-1.112.0

Data Analysis Overview

LogScale User Interface

Repositories & Views

Parsing Data

Searching Data

Writing Queries

Dashboards & Widgets

Automation

Query Language Syntax

Query Functions

Template Language

Keyboard Shortcuts