Go to the repository in which you want to ingest the data and select Ingest feeds under the Ingest section of the side menu.

Figure 68. Ingest Feeds

Click + New ingest feed and specify the following, where only Name and Parser are required:

Figure 69. Ingest Feeds - Add

Click Next. Click the copy button to copy your IAM identity as you will need this Identity along with your External ID to configure AWS.

Figure 70. Ingest Feeds - Add Part 1

Create a standard SQS queue in AWS with the following configurations:

(recommended) Create a dead-letter queue, see Amazon documentation for more information.

Figure 72. Dead-letter Queue

You need to grant access to the S3 Bucket or SNS topic, click Advanced under Access policy and copy and paste this snippet and:

Figure 73. Ingest Feeds - Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "sqs:SendMessage",
            "Resource": "arn:aws:sqs:*",
            "Condition": {
                "ArnEquals": {
                    "aws:SourceArn": "MY_ARN"
                }
            }
        }
    ]
}

Click Create the queue.

Configure the SQS queue or SNS Topic to create an event notifications, go to Create event notification on you SQS Bucket and configure it so that All objects create events.

Figure 74. Ingest Feeds - Events

Figure 75. Ingest Feeds - Events 2

Figure 76. Ingest Feeds - Events 3

Set the SQS Queue or SNS Topic as the destination, then click Save changes. Notifications will now appear in the bucket.

Figure 77. Ingest Feeds - data

If you are using an SNS Topic you need to create a SNS topic subscription that points to the SQS bucket you created previously with an access policy that allow SourceArn on your S3 bucket.

The next step is to create an IAM role, go to Identity and Access Management (IAM), which allows you to read data. Select Roles.

Create a role and add a Custom trust policy using this snippet, where you must:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLogScaleAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "LOGSCALE_AWS_IDENTITY_ARN"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "LOGSCALE_EXTERNAL_ID"
                }
            }
        }
    ]
}

Figure 78. Ingest Feeds - Custom trust policy

Click Next.

Click Next again, you do not add permissions as we will add an inline policy later.

Specify a name for the role and click Create role.

Find and select the role, Click the Add permissions drop down and select Create inline policy

Figure 79. Ingest Feeds - Role

Figure 80. Ingest Feeds - Inline Policy

Click the JSON tab and copy and paste the snippet below which allows your read data and;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReceiveS3Notifications",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes",
                "sqs:DeleteMessage",
                "sqs:ChangeMessageVisibility"
            ],
            "Effect": "Allow",
            "Resource": "SQS_QUEUE_ARN"
        },
        {
            "Sid": "ReadS3Objects",
            "Action": "s3:GetObject",
            "Effect": "Allow",
            "Resource": "S3_BUCKET_ARN/*"
        }
    ]
}

Click Next. Specify a name for the policy and click Create policy. Copy and save the ARN of the role and the URL of the SQS bucket.

Go back to the ingest feed you started to create in part 1.

Figure 81. Ingest Feeds - Add Part 2

Copy and paste the ARN of the role you created in AWS.

Copy the paste the URL of your SQS queue from AWS. The AWS Region is normally auto-filled but if empty select a region based on your AWS Console.

(Optional) click test to see if the configuration is valid. Click Save.