Set up Kafka as an ingest method
Step 1 - Create a Kafka ingest token
Why? Ingest tokens authenticate and authorize data sources to send data to your repository. They control which parsers can be used and what fields can be populated.
Detailed steps:
Sign in to Falcon LogScale, and browse to your repository.
Click Settings, Ingest Tokens.
Click Add token.
Type in a descriptive name (for example, kafka-production-logs).
Set the appropriate permissions:
Assign parser if you want Kafka to determine the parser
Assign fields to allow field creation from Kafka data
Click Create token to save the token and securely store the generated string - you'll need this when configuring LogScale to authenticate with Kafka.
Step 2 - Configure the Kafka connection
Why? These steps establish the connection between LogScale and your Kafka cluster, and define how LogScale consumes messages.
Detailed steps:
Go to Settings, Ingest Feeds.
Click New ingest feed.
Select Kafka from the dropdown.
Configure these essential parameters:
Bootstrap Servers: Entry points to your Kafka cluster (format: host1:port,host2:port)
This setting lets LogScale know where to connect to your Kafka brokers
Topic: The Kafka topic containing your log data
This setting defines which data stream LogScale should consume
Group ID: A unique identifier for LogScale's consumer group>
This setting enables Kafka to track the consumption progress, and allows multiple LogScale instances to load-balance
Parser: Select the appropriate parser for your data format
This setting determines how raw data is transformed into structured events
Token: Enter the ingest token created earlier
The token authenticates the connection to your repository
Step 3 - Advanced settings configuration
Why? These settings optimize performance, security, and reliability based on your specific environment.
Detailed steps:
Batch Settings:
Set the Max Poll Records (default: 500): Maximum number of records per polling cycle
This setting balances throughput vs. processing time
Set the Poll Interval (default: 100ms): How frequently LogScale checks for new messages
This setting affects latency and resource utilization
Authentication (if required):
Select authentication mechanism (SASL/PLAIN, SASL/SCRAM, etc.)
This setting secures the connection to Kafka if your cluster requires authentication
Provide a username/password or other credentials
These are necessary to authenticate LogScale to your secured Kafka cluster
TLS/SSL Configuration (if required):
Upload certificates or provide paths
These certificates enable encrypted communication with Kafka
Configure trust store settings
These settings verify the identity of your Kafka brokers
Error Handling:
Set Max Retries and Retry Backoff
This setting determines how LogScale handles temporary failures
Step 4 - Test and verify
Why? These steps help you confirm that the configuration is working correctly before relying on it in production.
Detailed steps:
Send test messages to your configured Kafka topic using a producer tool
In LogScale, browse to your repository and verify that:
Events are appearing with correct timestamps
Fields are parsed correctly according to your parser
Check Settings, Status, Ingest for any errors or warnings
Verify the consumer lag metrics to ensure that LogScale is keeping up with incoming data
Step 5 - Monitoring and maintenance
Why? Ensures the ongoing reliability of your ingest pipeline.
What you should do:
Set up alerts for Kafka consumer lag in LogScale
Monitor Kafka broker health separately
Periodically review and adjust batch settings based on performance
Plan for Kafka cluster scaling as log volumes grow