Set up a New Ingest Feed

The steps required to create a new ingest feed are partially on LogScale and partially on AWS as ID's created during the creation of an ingest feed must be used during the creation of policies on AWS.

The configuration has been divided into three phases to simplify the process. Phase 1 and 3 is performed on LogScale and phase 2 must be performed on AWS.

LogScale Ingest Feeds Steps Phase 1

The first step is to create the new ingest feed configuration:

  1. Go to Repositories and views page and select a relevant repository in which you want to ingest the data.

  2. Click Settings, under Ingest on the side menu click Ingest feeds.

    Ingest Feeds

    Figure 69. Ingest Feeds


  3. On the Ingest feeds page, click + New ingest feed and specify the following:

    • Name

      Enter a name for the feed (required).

    • Description

      Enter a meaningful description for the feed.

    • Compression

      Select a compression type from the list. Auto (default) uses decompression based on MIME type and file extension.

    • Preprocessing

      Define how the data is processed prior to ingestion, it can be split by AWS records or by new line. The way data should be preprocessed depends on the log source.

    • Parser

      Select the parser to apply to ingest data (required). The parser can be any parser that is on repository, predefined, from a package, or custom. See Parsing Data for more information.

    Ingest Feeds - Add

    Figure 70. Ingest Feeds - Add


  4. Click Next.

  5. In the Add credentials dialog box, click the copy icon to copy your IAM identity as you will need this identity along with your External ID to configure AWS.

    Ingest Feeds - Add Part 1

    Figure 71. Ingest Feeds - Add Part 1


AWS Steps Phase 2

The following is an example configuration with minimal permission to access the data to ingest. You will need:

  • To create a role in AWS.

  • The ARN of the role you create for the ingest feed in LogScale.

  • To configure an SQS queue or an SNS topic with event notifications.

    The following SQS permissions are required:

    ini
    sqs:ReceiveMessage
sqs:GetQueueAttributes
sqs:DeleteMessage
sqs:ChangeMessageVisibility

  1. Create a standard SQS queue in AWS with the following configurations:

    • Name

      Specify a name for your queue.

    • Message retention period

      We recommend setting this to 14 days (max).

      Ingest Feeds - SQS Queue

      Figure 72. Ingest Feeds - SQS Queue


  2. Create a dead-letter queue (recommended). See Amazon documentation for more information.

    Dead-letter Queue

    Figure 73. Dead-letter Queue


  3. You need to grant access to the S3 Bucket or SNS topic, click Advanced under Access policy and copy and paste this snippet and:

    Ingest Feeds - Policy

    Figure 74. Ingest Feeds - Policy


    • Replace the MY_ARN with the ARN of your S3 bucket or SNS Topic.

    json
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "sqs:SendMessage",
            "Resource": "arn:aws:sqs:*",
            "Condition": {
                "ArnEquals": {
                    "aws:SourceArn": "MY_ARN"
                }
            }
        }
    ]
}

  4. Click Create the queue.

  5. Configure the SQS queue or SNS Topic to create an event notifications, go to Create event notification on you SQS Bucket and configure it so that All objects create events.

    Ingest Feeds - Events

    Figure 75. Ingest Feeds - Events


    Ingest Feeds - Events 2

    Figure 76. Ingest Feeds - Events 2


    Ingest Feeds - Events 3

    Figure 77. Ingest Feeds - Events 3


  6. Set the SQS Queue or SNS Topic as the destination, then click Save changes. Notifications will now appear in the bucket.

    Ingest Feeds - data

    Figure 78. Ingest Feeds - data


  7. If you are using an SNS Topic you need to create a SNS topic subscription that points to the SQS bucket you created previously with an access policy that allow SourceArn on your S3 bucket.

  8. The next step is to create an IAM role, go to Identity and Access Management (IAM), which allows you to read data. Select Roles.

  9. Create a role and add a Custom trust policy using this snippet, where you must:

    json
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLogScaleAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "LOGSCALE_AWS_IDENTITY_ARN"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "LOGSCALE_EXTERNAL_ID"
                }
            }
        }
    ]
}
    Ingest Feeds - Custom trust policy

    Figure 79. Ingest Feeds - Custom trust policy


  10. Click Next.

  11. Click Next again, you do not add permissions as we will add an inline policy later.

  12. Specify a name for the role and click Create role.

  13. Find and select the role, click the Add permissions drop down and select Create inline policy

    Ingest Feeds - Role

    Figure 80. Ingest Feeds - Role


    Ingest Feeds - Inline Policy

    Figure 81. Ingest Feeds - Inline Policy


  14. Click the JSON tab and copy and paste the snippet below which allows your read data and;

    • Replace S3_BUCKET_ARN -- with the ARN of your S3 bucket this must be followed by /* to access the objects in the bucket.

    • Replace SQS_QUEUE_ARN with the ARN of your SQS queue

    json
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReceiveS3Notifications",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes",
                "sqs:DeleteMessage",
                "sqs:ChangeMessageVisibility"
            ],
            "Effect": "Allow",
            "Resource": "SQS_QUEUE_ARN"
        },
        {
            "Sid": "ReadS3Objects",
            "Action": "s3:GetObject",
            "Effect": "Allow",
            "Resource": "S3_BUCKET_ARN/*"
        }
    ]
}

  15. Click Next. Specify a name for the policy and click Create policy. Copy and save the ARN of the role and the URL of the SQS bucket.

LogScale Ingest Feeds Steps Phase 3

  1. Go back to the ingest feed you started to create in part 1.

    Ingest Feeds - Add Part 2

    Figure 82. Ingest Feeds - Add Part 2


  2. Copy and paste the ARN of the role you created in AWS.

  3. Copy and paste the URL of your SQS queue from AWS. The AWS Region is normally auto-filled but if empty, select a region based on your AWS Console.

  4. Optionally, click test to see if the configuration is valid. Click Save.