Create a file named kafka.properties and add the following values to it:

replica.fetch.max.bytes=104857600
message.max.bytes=104857600
compression.type=producer
unclean.leader.election.enable=false

The full list of other MSK parameters that can be used can be found in AWS documentation.

Create the configuration file for use within MSK. The name and description can be anything, but the name can't contain spaces.

$ aws kafka create-configuration \
  --name "LogScale-MSK-Configuration"
  --description "Custom LogScale configuration for MSK"
  --kafka-versions "2.3.1"
  --server-properties file://config-file-path

A success message similar to this should be returned:

{
  "Arn": "arn:aws:kafka:us-east-1:123456789012:configuration/LogScale-MSK-Configuration/abcdabcd-abcd-1234-abcd-abcd123e8e8e-1",
  "CreationTime": "2019-05-21T00:54:23.591Z",
  "Description": "Custom LogScale Configuration for MSK",
  "KafkaVersions": ["2.3.1"],
  "LatestRevision": {
    "CreationTime": "2019-05-21T00:54:23.591Z",
    "Description": "Custom LogScale Configuration for MSK",
    "Revision": 1
  },
  "Name": "LogScale-MSK-Configuration"
}

Login to the console, go to AWS MSK Service and then click on Create Cluster.

Give the Cluster any name. Pick the VPC created for this MSK Cluster. See the Amazon MSK page for setting up the VPC.

The Kafka version should be selected; version 2.4.0 for LogScale is the minimum recommended.

Select the Availability Zones and a subnet for each one. The minimum number of Availability Zones recommended is three.

Then add the custom configuration file (see Custom MSK Configuration). Select Use a Custom Configuration and select the name of the configuration file uploaded.

Figure 6. AWS MSK Configuration

Next, create the brokers. Kafka brokers use m5 instance types. Specifications for these can be found on the Instance Types under the m5 tab. Define the number of brokers for each availability zone.

Optionally, add tags to identify the cluster. For more information on tagging, see AWS Tagging Strategy.

Define how much Storage each broker will have. MSK uses AWS Elastic Block Storage. The amount of storage should correlate to how much data is going to be ingested. Once created, the storage cannot be decreased.

If the AWS instance as part of a cluster, encryption is recommended to be enabled. Encryption between clients and brokers is possible, but requires some additional steps. For more information, see Configuring Encryption. If plaintext brokers have been selected, these will be on available port 9092; brokers using TLS will be accessible on port 9094.

If TLS Client authentication is required, see Mutual TLS Authentication for more information.

Basic monitoring is available for free, but the enhanced monitoring costs extra. More information can be found on the Monitoring an Amazon MSK Cluster.

When selecting the security group, it's important to note that the LogScale instance must be able to connect to MSK. This can be either allowing inbound and outbound rules for the IP of the LogScale instance, or if LogScale is running on AWS, adding them to the same security group.