Manage Kafka Queue Settings

The notes on this page only work in the noted versions. To adjust the number of Kafka partitions in later releases, use the appropriate tools within your Kafka deployment, for example the bin/kafka-reassign-partitions.sh and bin/kafka-topics.sh commands in the Kafka installation.

Show Kafka Queue Settings

Description Returns a list of the Kafka queue settings.
MethodGET /api/v1/clusterconfig/kafka-queues/partition-assignment
Authentication Requiredyes
Endpoint Removed in Version1.147.99 
Return Codes
200 Request complete
400 Bad authentication
500 Request failed
http
GET /api/v1/clusterconfig/kafka-queues/partition-assignment

To redirect the output to a JSON file, run the script like this:

shell
$ script >$YOUR_FILE_NAME.json

You can then use this as the payload in further requests when updating the information.

Mac OS or Linux (curl)
shell
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Mac OS or Linux (curl) One-line
shell
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"
Windows Cmd and curl
shell
curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"
Windows Powershell and curl
powershell
curl.exe -X GET 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    "$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment';

my $req = HTTP::Request->new("GET", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");


my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment'

resp = requests.get(url,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

let request = https.get('$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment', (res) => {
  if (res.statusCode !== 200) {
    console.error(`Error from server. Code: ${res.statusCode}`);
    res.resume();
    return;
  }

  let data = '';

  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('close', () => {
    console.log('Response:');
    console.log(JSON.parse(data));
  });
});

    return(undef,undef);
}

Modify Kafka Queue Settings

Description Modify or update Kafka queue settings.
MethodPOST /api/v1/clusterconfig/kafka-queues/partition-assignment
Request Data 
Authentication Requiredyes
Endpoint Removed in Version1.147.99 
Return Codes
200 Request complete
400 Bad authentication
500 Request failed
http
POST /api/v1/clusterconfig/kafka-queues/partition-assignment

Set replication defaults for Kafka Queues

Description Apply default Kafka replication settings to all nodes in a cluster.
MethodPOST /api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults
Request Data 
Authentication Requiredyes
Endpoint Removed in Version1.88.99 
Return Codes
200 Request complete
400 Bad authentication
500 Request failed

The ingest queues are partitions of the Kafka queue humio-ingest. LogScale offers an API for editing the Kafka partition to broker assignments in this queue. Note that changes to these settings are applied asynchronously, thus you can get the previous settings, or a mix with the latest settings, for a few seconds after applying a new set.

http
POST   /api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults

The format of the file should be a JSON object with partition count and replicas:

shell
$ echo '{ "partitionCount": 24, "replicas": 2 }' > YOUR_FILE_NAME.json
Mac OS or Linux (curl)
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '@$YOUR_FILE_NAME.json'
Mac OS or Linux (curl) One-line
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '@$YOUR_FILE_NAME.json'
Windows Cmd and curl
shell
curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d '@$YOUR_FILE_NAME.json'
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '@$YOUR_FILE_NAME.json'
    "$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults';

my $json = '@$YOUR_FILE_NAME.json';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults'
mydata = r'''@$YOUR_FILE_NAME.json'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

const data = JSON.stringify(
    @$YOUR_FILE_NAME.json
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL',
  path: 'api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();