createAwsS3SqsIngestFeed()

🎯API Stability

Long-TermThis API is stable and will remain supported across versions with minimal breaking changes.

Summary

The createAwsS3SqsIngestFeed() mutation creates an ingest feed that reads log data from an Amazon S3 bucket using SQS event notifications. LogScale automatically pulls data from S3 as new objects arrive in the bucket. To modify a feed, use updateAwsS3SqsIngestFeed().

Use updateAwsS3SqsIngestFeed() to modify a feed's configuration. Use testAwsS3SqsIngestFeed() to validate the feed's connectivity and credentials before enabling it. To remove an ingest feed, use deleteIngestFeed().

Hide Query Example

Show AWS Ingest Feeds Query

To get a list of AWS ingest feeds, use the repositories() query like this:

graphql

query {
     repositories
       { id, name, awsExternalId,
         ingestFeeds( typeFilter: AwsS3Sqs ) 
            { results { id, name, enabled } }
        }
   }

Syntax

graphql

createAwsS3SqsIngestFeed(
      input: CreateAwsS3SqsIngestFeed!
   ): IngestFeed!

The mutation accepts a single required argument, input, of type CreateAwsS3SqsIngestFeed!. This input includes the repository name, the parser identifier, and the AWS connection details such as authentication credentials and SQS queue configuration. The mutation returns a non-null IngestFeed! object.

Example

Raw

graphql

mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: "humio", 
          name: "main-server",
          parser: "audit-log",
          authentication: {kind: IamRole, roleArn: "arn:aws_iam::1234:user/admin"},
          sqsUrl: "http://address.com/something",
          region: "us-west-2",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}

Mac OS or Linux (curl)

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}
EOF

Mac OS or Linux (curl) One-line

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}
EOF

Windows Cmd and curl

shell

curl -v -X POST $YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "mutation { ^
  createAwsS3SqsIngestFeed(input: ^
        { repositoryName: \"humio\",  ^
          name: \"main-server\", ^
          parser: \"audit-log\", ^
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"}, ^
          sqsUrl: \"http://address.com/something\", ^
          region: \"us-west-2\", ^
          enabled: true, ^
          preprocessing: {kind: SplitAwsRecords}, ^
          compression: Auto ^
        } ) ^
  {name, enabled } ^
}" ^
} '

Windows Powershell and curl

powershell

curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}'
    "$YOUR_LOGSCALE_URL/graphql"

Perl

perl

#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/graphql';

my $query = "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}";
$query =~ s/\n/ /g;
my $json = sprintf('{"query" : "%s"}',$query);
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";

Python

python

#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)

Node.js

javascript

const https = require('https');

const data = JSON.stringify(
    {"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL',
  path: 'graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Input Parameters

Provide the repository name, the parser name or unique identifier, and the AWS connection details required to authenticate and read from the SQS queue and S3 bucket.

Table: CreateAwsS3SqsIngestFeed Input Datatype

Parameter	Type	Required	Stability	Description
authentication	IngestFeedAwsAuthenticationInput	yes	`Long-Term`	How to authenticate to AWS. See IngestFeedAwsAuthenticationInput.
compression	IngestFeedCompression	yes	`Long-Term`	Compression scheme of the file. See IngestFeedCompression.
description	String		`Long-Term`	The description of the ingest feed. See UpdateIngestFeedDescription.
enabled	Boolean	yes	`Long-Term`	Whether ingest from the ingest feed is enabled.
name	String	yes	`Long-Term`	The name of the ingest feed.
parser	String	yes	`Long-Term`	The unique identifier or name of the parser that should be used to parse the ingest feed. Parsers in packages can be referred to as `packagescope/packagename:parsername`.
preprocessing	IngestFeedPreprocessingInput	yes	`Long-Term`	The preprocessing to apply to an ingest feed before parsing. See IngestFeedPreprocessingInput.
region	String	yes	`Long-Term`	The AWS region in which to connect.
repositoryName	RepoOrViewName	yes	`Long-Term`	Name of the repository of the ingest feed. RepoOrViewName is a scalar.
sqsUrl	String	yes	`Long-Term`	The AWS SQS queue URL.

Returned Values

The mutation returns an IngestFeed! object representing the created feed. The returned object includes the feed ID, name, enabled status, and parser details.

Table: IngestFeed Datatype

Parameter	Type	Required	Stability	Description
createdAt	Long	yes	`Long-Term`	Unix timestamp for when this feed was created.
description	String		`Long-Term`	A description of the ingest feed.
enabled	Boolean	yes	`Long-Term`	Whether the ingest from the ingest feed is enabled.
executionInfo	IngestFeedExecutionInfo		`Long-Term`	Details about how the ingest feed is running. See IngestFeedExecutionInfo.
forceStopped	Boolean	yes	`Preview`	Whether the ingest feed is force stopped, meaning only a cluster manager can start the ingest feed again.
id	String	yes	`Long-Term`	The unique identifier of the ingest feed.
name	String	yes	`Long-Term`	The name of the ingest feed.
parser	Parser		`Long-Term`	The parser used to parse the ingest feed. See Parser.
source	IngestFeedSource	yes	`Long-Term`	The source of the ingest feed. See IngestFeedSource.

Versions of this Page

GraphQL Mutations

GraphQL API

GraphQL Queries

GraphQL Datatypes