Set up Amazon S3 as an ingest method
Step 1 - Start the AWS ingest feed in Log Collector
Why? Starting the feed in Log Collector first generates the IAM identity and External ID that you need when you create the AWS trust relationship.
Detailed steps:
Sign in to Log Collector, and browse to your repository.
Click Settings, Ingest Feeds.
Click New ingest feed.
Type in a descriptive name (for example, s3-application-logs).
Set the appropriate permissions:
Name - Enter a unique name for the feed.
Description - Enter a meaningful description of the source.
Compression - Select the appropriate setting. Auto is the default and can detect supported compression such as Gzip.
Preprocessing - Choose how Log Collector should split the incoming data. The available options include splitting by AWS records or by new line; the correct choice depends on the log source.
Parser - Select the parser to apply to the ingested data. This field is required
Select Next.
In the Add credentials dialog, copy the
IAM identityandExternal ID. Keep both values available for the AWS configuration.
Important
The following steps are provided as a general guide to configuring Amazon S3 for use with Log Collector. As third-party systems such as Amazon S3 are updated frequently and independently, some steps, options, or UI elements may differ between what is shown below, and what you see in your environment.
If you encounter discrepancies, refer to the official Amazon S3 documentation for the most up-to-date instructions.
Step 2 - Create an SQS queue
Why? Amazon S3 sends object-created notifications to SQS. Log Collector consumes those messages to identify objects that need to be retrieved and ingested.
In the AWS Management Console, open Amazon SQS and create a Standard queue.
Enter a descriptive queue name.
Set the message retention period. We recommend a maximum retention period of 14 days.
Create and configure a dead-letter queue. We recommend a dead-letter queue so repeatedly unprocessable messages can be isolated for investigation.
After following these steps, you'll need to allow Amazon S3 to send
messages to the queue. To do this, add an SQS
queue policy that permits the Amazon S3 service to call
sqs:SendMessage for this queue, restricted to the
source S3 bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "s3.amazonaws.com" },
"Action": "sqs:SendMessage",
"Resource": "SQS_QUEUE_ARN",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "S3_BUCKET_ARN"
}
}
}
]
}
Replace SQS_QUEUE_ARN and
S3_BUCKET_ARN with the ARNs for your queue and source
bucket.
Step 3 - Configure S3 event notifications (optional but recommended)
Why? The event notification tells SQS when a new object is created in the bucket.
In the Amazon S3 console, open the bucket that contains the source logs.
Select Properties.
Under Event notifications, select Create event notification.
Enter a descriptive event name.
Optionally, configure a prefix or suffix if only specific object keys should generate notifications.
Under Event types, select All object create events.
For Destination, select SQS queue and select the queue created in Step 2.
Save the event notification.
Amazon S3 supports Standard SQS queues as direct event-notification destinations. Note that S3 event notifications are delivered at least once, so duplicate notifications are possible.
Step 4 - Create the IAM role for LogScale
Why? The IAM role lets LogScale access only the SQS queue and S3 objects required for this feed, without storing long-lived AWS access keys in LogScale.
In AWS Identity and Access Management (IAM), create a new role.
Choose a custom trust policy.
Use the IAM identity and External ID copied from Log Collector:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowLogScaleAccess",
"Effect": "Allow",
"Principal": {
"AWS": "LOGSCALE_AWS_IDENTITY_ARN"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "LOGSCALE_EXTERNAL_ID"
}
}
}
]
}
Create the role, then add an inline permissions policy.
Grant the role the permissions required to consume SQS messages and retrieve S3 objects:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReceiveS3Notifications",
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:GetQueueAttributes",
"sqs:DeleteMessage",
"sqs:ChangeMessageVisibility"
],
"Resource": "SQS_QUEUE_ARN"
},
{
"Sid": "ReadS3Objects",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "S3_BUCKET_ARN/*"
}
]
}
Save the role ARN and the SQS queue URL. You need both values to finish the LogScale configuration.
Step 5 - Finish the ingest feed in LogScale
Return to the ingest feed that you started in Step 1.
Enter the ARN of the IAM role you created in AWS.
Enter the URL of the SQS queue.
Confirm the AWS Region. LogScale normally fills this automatically; if it is empty, select the region used by the AWS resources.
Select Test to validate the configuration.
If the test succeeds, select Save.
.
Step 6 - Verify ingestion
Why? An end-to-end test confirms that the S3 notification, SQS permissions, IAM role, parser, and LogScale feed all work together.
Create or upload a representative log object in the configured S3 bucket.
Confirm that Amazon S3 sends an object-created notification to the SQS queue.
In LogScale, open the target repository and search for the newly ingested events.
Verify that timestamps and parsed fields are correct for the selected parser.
If events do not appear, review the SQS queue, IAM role permissions, S3 event notification, feed configuration, and parser.