Create a file named kafka.properties and add the following values to it:

replica.fetch.max.bytes=104857600
message.max.bytes=104857600
compression.type=producer
unclean.leader.election.enable=false

The full list of other MSK parameters that can be used can be found in AWS documentation.

Create the configuration file for use within MSK. The name and description can be anything, but the name can't contain spaces. Specify a currently supported Kafka 3.x version that matches the version you intend to deploy:

aws kafka create-configuration \
  --name "LogScale-MSK-Configuration" \
  --description "Custom LogScale configuration for MSK" \
  --kafka-versions "3.6.0" \
  --server-properties file://config-file-path

A success message similar to this should be returned:

{
  "Arn": "arn:aws:kafka:us-east-1:123456789012:configuration/LogScale-MSK-Configuration/abcdabcd-abcd-1234-abcd-abcd123e8e8e-1",
  "CreationTime": "2024-01-01T00:54:23.591Z",
  "Description": "Custom LogScale Configuration for MSK",
  "KafkaVersions": ["3.6.0"],
  "LatestRevision": {
    "CreationTime": "2024-01-01T00:54:23.591Z",
    "Description": "Custom LogScale Configuration for MSK",
    "Revision": 1
  },
  "Name": "LogScale-MSK-Configuration"
}

Login to the console, go to AWS MSK Service and then click on Create Cluster.

Give the Cluster any name. Pick the VPC created for this MSK Cluster. See the Amazon MSK page for setting up the VPC.

Select the Kafka version. Kafka 3.6.0 or later is the minimum recommended version for LogScale. Where possible, select the latest available Kafka 3.x version supported by MSK. MSK now supports KRaft mode (ZooKeeper-less architecture) for Kafka 3.x clusters, which is the recommended deployment mode. ZooKeeper-based clusters are considered legacy and should be avoided for new deployments.

Select the Availability Zones and a subnet for each one. The minimum number of Availability Zones recommended is three.

Then add the custom configuration file (see Custom MSK Configuration). Select Use a Custom Configuration and select the name of the configuration file uploaded.

Configure the brokers. Kafka brokers use m5, m6i, or m7i instance types. Current generation instances (m6i or m7i) are recommended for improved performance and cost efficiency. Specifications for these can be found on the AWS EC2 Instance Types page. Define the number of brokers for each Availability Zone.

Optionally, add tags to identify the cluster. For more information on tagging, see AWS Tagging Strategy.

Define how much Storage each broker will have. MSK uses AWS Elastic Block Storage. The amount of storage should correlate to how much data is going to be ingested. Storage cannot be decreased after the cluster has been created.

Encryption is strongly recommended for all deployments, and is required for any production or multi-node cluster. Encryption between clients and brokers requires additional configuration steps — see Configuring Encryption. TLS brokers are accessible on port 9094. PLAINTEXT brokers on port 9092 should not be used in production environments.

If TLS Client authentication is required, see Mutual TLS Authentication for more information.

Basic monitoring is available for free, but the enhanced monitoring costs extra. More information can be found on the Monitoring an Amazon MSK Cluster.

When selecting the security group, it's important to note that the LogScale instance must be able to connect to MSK. This can be either allowing inbound and outbound rules for the IP of the LogScale instance, or if LogScale is running on AWS, adding them to the same security group.