TOPIC_MAX_MESSAGE_BYTES - this enables the producer to send larger messages, and for LogScale to configure the topic to enable it to store larger messages. If the topic already exists, you need to edit the Kafka topic configuration - the relevant setting to change is max.message.bytes.

Refer to the Kafka documentation for more details.

KAFKA_INGEST_QUEUE_PRODUCER_SEND_BUFFER_BYTES - This enables you to increase the TCP buffer size for the producer. See also KAFKA_INGEST_QUEUE_PRODUCER_, especially on how to map between LogScale variables and Kafka variables.

Refer to the Kafka documentation for more details.

KAFKA_INGEST_QUEUE_PRODUCER_MAX_REQUEST_SIZE - This sets the maximum size (in bytes) of a request that the producer can send. If a message batch exceeds this size, the producer will split it into smaller requests.

Refer to the Kafka documentation for more details.

KAFKA_INGEST_QUEUE_CONSUMER_FETCH_MAX_BYTES - This defines the maximum number of bytes the server should return for a fetch request. If a message batch is larger than this value, the server will still return it, but only fetch one message. This setting must be larger than the largest message a broker will accept, max.message.bytes.

This setting should be coordinated with other related configurations such as max.partition.fetch.bytes and message.max.bytes for optimal performance.

Refer to the Kafka documentation for more details.

KAFKA_INGEST_QUEUE_CONSUMER_MAX_PARTITION_FETCH_BYTES - This controls the maximum number of bytes that will be fetched from each partition at a time. This setting applies at the per-partition level, unlike fetch.max.bytes which is for the entire fetch request. Must be larger than the largest message in the partition.

This should be set lower than the consumer's total memory allocation. Multiple partitions can fetch in parallel, so total memory use could be max.partition.fetch.bytes * number_of_partitions.

Default value is typically 1MB (1048576 bytes).

Refer to the Kafka documentation for more details.

KAFKA_INGEST_QUEUE_PRODUCER_BUFFER_MEMORY - This determines the total amount of memory the producer can use for buffering. When this buffer memory is exhausted, additional send() calls will block or throw exceptions depending on configuration. This parameter acts as a crucial flow control mechanism.

Should be sized based on:

This parameter works in conjunction with batch.size, linger.ms and max.block.ms.

Default is typically 32MB (33554432 bytes).

Refer to the Kafka documentation for more details.