Manage Kafka Queue Settings

The notes on this page only work in the noted versions. To adjust the number of Kafka partitions in later releases, use the appropriate tools within your Kafka deployment, for example the bin/kafka-reassign-partitions.sh and bin/kafka-topics.sh commands in the Kafka installation.

Show Kafka Queue Settings

Description	Returns a list of the Kafka queue settings.
Method	`GET /api/v1/clusterconfig/kafka-queues/partition-assignment`
Authentication Required	yes
Return Codes
200	Request complete
400	Bad authentication
500	Request failed

http

GET /api/v1/clusterconfig/kafka-queues/partition-assignment

To redirect the output to a JSON file, run the script like this:

shell

$ script >$YOUR_FILE_NAME.json

You can then use this as the payload in further requests when updating the information.

Show:

shell

curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

shell

curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

shell

curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"

powershell

curl.exe -X GET 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    "$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment"

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment';

my $req = HTTP::Request->new("GET", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");


my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment'

resp = requests.get(url,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

javascript

const https = require('https');

let request = https.get('$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment', (res) => {
  if (res.statusCode !== 200) {
    console.error(`Error from server. Code: ${res.statusCode}`);
    res.resume();
    return;
  }

  let data = '';

  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('close', () => {
    console.log('Response:');
    console.log(JSON.parse(data));
  });
});

    return(undef,undef);
}

Set replication defaults for Kafka Queues

Description	Apply default partition settings to all nodes in the cluster.
Method	`POST /api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults`
Request Data
Authentication Required	yes
Return Codes
200	Request complete
400	Bad authentication
500	Request failed

The ingest queues are partitions of the Kafka queue humio-ingest. LogScale offers an API for editing the Kafka partition to broker assignments in this queue. Note that changes to these settings are applied asynchronously, thus you can get the previous settings, or a mix with the latest settings, for a few seconds after applying a new set.

http

POST   /api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults

The format of the file should be a JSON object with partition count and replicas:

shell

$ echo '{ "partitionCount": 24, "replicas": 2 }' > YOUR_FILE_NAME.json

Show:

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '@$YOUR_FILE_NAME.json'

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '@$YOUR_FILE_NAME.json'

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d '@$YOUR_FILE_NAME.json'

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '@$YOUR_FILE_NAME.json'
    "$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults"

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults';

my $json = '@$YOUR_FILE_NAME.json';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults'
mydata = r'''@$YOUR_FILE_NAME.json'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

javascript

const https = require('https');

const data = JSON.stringify(
    @$YOUR_FILE_NAME.json
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

APIs

API Authentication

Cluster Management API

Health Check API

Ingest API

Lookup API

Redact Events API

Search API

Packages API

Software Libraries