Manage Kafka Queue Settings

The notes on this page only work in the noted versions. To adjust the number of Kafka partitions in later releases, use the appropriate tools within your Kafka deployment, for example the bin/kafka-reassign-partitions.sh and bin/kafka-topics.sh commands in the Kafka installation.

Show Kafka Queue Settings

Description	Returns a list of the Kafka queue settings.
Method	`GET /api/v1/clusterconfig/kafka-queues/partition-assignment`
Authentication Required	yes
Return Codes
200	Request complete
400	Bad authentication
500	Request failed

http

GET    /api/v1/clusterconfig/kafka-queues/partition-assignment

To redirect the output to a JSON file, run the script like this:

shell

$ script>$YOUR_FILE_NAME.json

You can then use this as the payload in further requests when updating the information.

Mac OS or Linux (curl)

shell

curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Mac OS or Linux (curl) One-line

shell

curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json"

Windows Cmd and curl

cmd

curl -v -X GET $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json"

Windows Powershell and curl

powershell

curl.exe -X GET 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment';
my $req = HTTP::Request->new("GET", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment'

resp = requests.get(url,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');

let request = https.get('$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment', (res) => {
  if (res.statusCode !== 200) {
    console.error(`Error from server. Code: ${res.statusCode}`);
    res.resume();
    return;
  }

  let data = '';

  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('close', () => {
    console.log('Response:');
    console.log(JSON.parse(data));
  });
});

    return(undef,undef);
}

Modify Kafka Queue Settings

Description	Adjust Kafka replication defaults.
Method	`POST /api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults`
Request Data
Authentication Required	yes
Return Codes
200	Request complete
400	Bad authentication
500	Request failed

The ingest queues are partitions of the Kafka queue humio-ingest. LogScale offers an API for editing the Kafka partition to broker assignments in this queue. Note that changes to these settings are applied asynchronously, thus you can get the previous settings, or a mix with the latest settings, for a few seconds after applying a new set.

http

POST   /api/v1/clusterconfig/kafka-queues/partition-assignment

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '@$YOUR_FILE_NAME.json'

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '@$YOUR_FILE_NAME.json'

Windows Cmd and curl

cmd

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d '@$YOUR_FILE_NAME.json'

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '@$YOUR_FILE_NAME.json'
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment';
my $json = '@$YOUR_FILE_NAME.json';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment'
mydata = r'''@$YOUR_FILE_NAME.json'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');

const data = JSON.stringify(
    @$YOUR_FILE_NAME.json
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Set replication defaults for Kafka Queues

Description	Adjust Kafka replication defaults.
Method	`POST /api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults/set-replication-defaults`
Request Data
Authentication Required	yes
Return Codes
200	Request complete
400	Bad authentication
500	Request failed

http

POST   /api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults

The format of the file should be a JSON object with partition count and replicas:

shell

$ echo '{ "partitionCount": 24, "replicas": 2 }' > YOUR_FILE_NAME.json

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '@$YOUR_FILE_NAME.json'

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '@$YOUR_FILE_NAME.json'

Windows Cmd and curl

cmd

curl -v -X POST $YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d '@$YOUR_FILE_NAME.json'

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '@$YOUR_FILE_NAME.json'
"$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;
my $TOKEN = "TOKEN";
my $uri = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults';
my $json = '@$YOUR_FILE_NAME.json';
my $req = HTTP::Request->new("POST", $uri );
$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");
$req->content( $json );
my $lwp = LWP::UserAgent->new;
my $result = $lwp->request( $req );
print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults'
mydata = r'''@$YOUR_FILE_NAME.json'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');

const data = JSON.stringify(
    @$YOUR_FILE_NAME.json
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL/api/v1/clusterconfig/kafka-queues/partition-assignment/set-replication-defaults',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

LogScale APIs 1.118.0-1.138.0

APIs

API Authentication

Cluster Management API

Health Check API

Ingest API

Lookup API

Redact Events API

Search API

Software Libraries