Manage Cluster Segments

Please refer to the Storage Rules documentation.

When a data segment is complete, the server selects the host(s) to place the segment on by looking up a segment-related key in the storage partition table. The partitions map to a set of nodes. All of these nodes are then assigned as owners of the segment and will start getting their copy shortly after.

You can modify the storage partitions at any time. Any number of partitions larger than the number of nodes is allowed, but the recommended the number of storage partitions is 24 or a similar fairly low number. There is no benefit in having a large number of partitions.

Existing segments are not moved when re-assigning partitions. Partitions only affect segments completed after they are POST'ed.

Prune Replicas when Reducing Replica Setting

If the number of replicas has been reduced, existing segments in the cluster do not get their replica count reduced. In order to reduce the number of replicas on existing segments, invoke this:

Description Reduce the number of replicas on existing segments.
MethodPOST /api/v1/clusterconfig/segments/prune-replicas
Request Data 
Authentication Requiredyes
Return Codes
200 Request complete
400 Bad authentication
500 Request failed
http
POST /api/v1/clusterconfig/segments/prune-replicas
Mac OS or Linux (curl)
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Mac OS or Linux (curl) One-line
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Windows Cmd and curl
cmd
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas';
my $json = '';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas'
mydata = r''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');



const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/prune-replicas',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Reshuffle All Existing Segments Between Nodes

This endpoint moves segments so that all nodes have their fair share of the segments, as stated in the storage partitioning setting, but as much as possible leaving segments where they are. It's also possible to apply the current partitioning scheme to all existing segments, possibly moving every segment to a new node.

Description Reshuffle existing segments between nodes.
MethodPOST /api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all/percentage
Request Data 
Authentication Requiredyes
Path ArgumentsDescriptionData typeRequired?
percentage Percentage of full set of data to which this action should be applied. integeroptional
Return Codes
200 Request complete
400 Bad authentication
500 Request failed
http
POST /api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all
Mac OS or Linux (curl)
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Mac OS or Linux (curl) One-line
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Windows Cmd and curl
cmd
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all';
my $json = '';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all'
mydata = r''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');



const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-reshuffle-all',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Distribute Stored Data to a Node

If a new node is added, and you want it to take its fair share of the currently stored data, use "distribute-evenly-to-host".

Description Move existing segments between nodes.
MethodPOST /api/v1/clusterconfig/segments/distribute-evenly-to-host/id/percentage
Request Data 
Authentication Requiredyes
Path ArgumentsDescriptionData typeRequired?
id ID of the node configuration stringrequired
percentage Percentage of full set of data to which this action should be applied. integeroptional
Return Codes
200 Request complete
400 Bad authentication
500 Request failed
http
POST /api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID
Mac OS or Linux (curl)
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Mac OS or Linux (curl) One-line
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Windows Cmd and curl
cmd
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID';
my $json = '';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID'
mydata = r''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');



const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-to-host/$NODE_ID',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Distribute Stored Data from a Node

It's possible to move all existing segments off a node. If that node is not assigned any partitions at all (both storage and ingest), this then relieves the node of all duties, preparing it to be deleted from the cluster.

Description Move existing segments between nodes.
MethodPOST /api/v1/clusterconfig/segments/distribute-evenly-from-host/id/percentage
Request Data 
Authentication Requiredyes
Path ArgumentsDescriptionData typeRequired?
id ID of the node configuration stringrequired
percentage Percentage of full set of data to which this action should be applied. integeroptional
Return Codes
200 Request complete
400 Bad authentication
500 Request failed
http
POST /api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID
Mac OS or Linux (curl)
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Mac OS or Linux (curl) One-line
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Windows Cmd and curl
cmd
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID';
my $json = '';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID'
mydata = r''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');



const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/segments/distribute-evenly-from-host/$NODE_ID',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();