Manage Cluster Segments

Please refer to the Storage Rules documentation.

When a data segment is complete, the server selects the host(s) to place the segment on by looking up a segment-related key in the storage partition table. The partitions map to a set of nodes. All of these nodes are then assigned as owners of the segment and will start getting their copy shortly after.

You can modify the storage partitions at any time. Any number of partitions larger than the number of nodes is allowed, but the recommended the number of storage partitions is 24 or a similar fairly low number. There is no benefit in having a large number of partitions.

Existing segments are not moved when re-assigning partitions. Partitions only affect segments completed after they are POST'ed.

Prune Replicas when Reducing Replica Setting

If the number of replicas has been reduced, existing segments in the cluster do not get their replica count reduced. In order to reduce the number of replicas on existing segments, invoke this:

Description	Reduce the number of replicas on existing segments.
Method	`POST /api/v1/clusterconfig/segments/prune-replicas`
Request Data
Authentication Required	yes
Return Codes
200	Request complete
400	Bad authentication
500	Request failed

http

POST /api/v1/clusterconfig/segments/prune-replicas

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Windows Cmd and curl

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas';

my $json = '';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas'
mydata = r''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');



const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Reshuffle All Existing Segments Between Nodes

This endpoint moves segments so that all nodes have their fair share of the segments, as stated in the storage partitioning setting, but as much as possible leaving segments where they are. It's also possible to apply the current partitioning scheme to all existing segments, possibly moving every segment to a new node.

Description	Reshuffle existing segments between nodes.
Method	`POST /api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all/percentage`
Request Data
Authentication Required	yes
Path Arguments	Description	Data type	Required?
`percentage`	Percentage of full set of data to which this action should be applied.	`integer`	optional
Return Codes
200	Request complete
400	Bad authentication
500	Request failed

http

POST /api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Windows Cmd and curl

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all';

my $json = '';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all'
mydata = r''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');



const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Distribute Stored Data to a Node

If a new node is added, and you want it to take its fair share of the currently stored data, use "distribute-evenly-to-host".

Description	Move existing segments between nodes.
Method	`POST /api/v1/clusterconfig/segments/distribute-evenly-to-host/id/percentage`
Request Data
Authentication Required	yes
Path Arguments	Description	Data type	Required?
`id`	ID of the node configuration	`string`	required
`percentage`	Percentage of full set of data to which this action should be applied.	`integer`	optional
Return Codes
200	Request complete
400	Bad authentication
500	Request failed

http

POST /api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Windows Cmd and curl

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID';

my $json = '';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID'
mydata = r''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');



const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Distribute Stored Data from a Node

It's possible to move all existing segments off a node. If that node is not assigned any partitions at all (both storage and ingest), this then relieves the node of all duties, preparing it to be deleted from the cluster.

Description	Move existing segments between nodes.
Method	`POST /api/v1/clusterconfig/segments/distribute-evenly-from-host/id/percentage`
Request Data
Authentication Required	yes
Path Arguments	Description	Data type	Required?
`id`	ID of the node configuration	`string`	required
`percentage`	Percentage of full set of data to which this action should be applied.	`integer`	optional
Return Codes
200	Request complete
400	Bad authentication
500	Request failed

http

POST /api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Windows Cmd and curl

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID';

my $json = '';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID'
mydata = r''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');



const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

APIs

API Authentication

Cluster Management API

Health Check API

Ingest API

Lookup API

Redact Events API

Search API

Packages API

Software Libraries