The createAwsS3SqsIngestFeed() GraphQL mutation is used to create an ingest feed that uses AWS S3 and SQS.

This is an experimental feature, not ready for production.

Syntax

Below is the syntax for the createAwsS3SqsIngestFeed() mutation field:

graphql
createAwsS3SqsIngestFeed(
     input: CreateAwsS3SqsIngestFeed!
   ): IngestFeed!

The input contains data for creating an ingest feed that uses AWS S3 and SQS. Below is an example of how this mutation field might be used:

Raw
graphql
mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: "humio", 
          name: "main-server",
          parser: "audit-log",
          authentication: {kind: IamRole, roleArn: "arn:aws_iam::1234:user/admin"},
          sqsUrl: "http://address.com/something",
          region: "us-west-2",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}
Mac OS or Linux (curl)
shell
curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}
EOF
Mac OS or Linux (curl) One-line
shell
curl -v -X POST $YOUR_LOGSCALE_URL/graphql \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @- << EOF
{"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}
EOF
Windows Cmd and curl
shell
curl -v -X POST $YOUR_LOGSCALE_URL/graphql ^
    -H "Authorization: Bearer $TOKEN" ^
    -H "Content-Type: application/json" ^
    -d @'{"query" : "mutation { ^
  createAwsS3SqsIngestFeed(input: ^
        { repositoryName: \"humio\",  ^
          name: \"main-server\", ^
          parser: \"audit-log\", ^
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"}, ^
          sqsUrl: \"http://address.com/something\", ^
          region: \"us-west-2\", ^
          enabled: true, ^
          preprocessing: {kind: SplitAwsRecords}, ^
          compression: Auto ^
        } ) ^
  {name, enabled } ^
}" ^
} '
Windows Powershell and curl
powershell
curl.exe -X POST 
    -H "Authorization: Bearer $TOKEN"
    -H "Content-Type: application/json"
    -d '{"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}'
"$YOUR_LOGSCALE_URL/graphql"
Perl
perl
#!/usr/bin/perl

use HTTP::Request;
use LWP;

my $INGEST_TOKEN = "TOKEN";

my $uri = '$YOUR_LOGSCALE_URL/graphql';

my $json = '{"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}';
my $req = HTTP::Request->new("POST", $uri );

$req->header("Authorization" => "Bearer $TOKEN");
$req->header("Content-Type" => "application/json");

$req->content( $json );

my $lwp = LWP::UserAgent->new;

my $result = $lwp->request( $req );

print $result->{"_content"},"\n";
Python
python
#! /usr/local/bin/python3

import requests

url = '$YOUR_LOGSCALE_URL/graphql'
mydata = r'''{"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}'''

resp = requests.post(url,
                     data = mydata,
                     headers = {
   "Authorization" : "Bearer $TOKEN",
   "Content-Type" : "application/json"
}
)

print(resp.text)
Node.js
javascript
const https = require('https');

const data = JSON.stringify(
    {"query" : "mutation {
  createAwsS3SqsIngestFeed(input:
        { repositoryName: \"humio\", 
          name: \"main-server\",
          parser: \"audit-log\",
          authentication: {kind: IamRole, roleArn: \"arn:aws_iam::1234:user/admin\"},
          sqsUrl: \"http://address.com/something\",
          region: \"us-west-2\",
          enabled: true,
          preprocessing: {kind: SplitAwsRecords},
          compression: Auto
        } )
  {name, enabled }
}"
}
);


const options = {
  hostname: '$YOUR_LOGSCALE_URL/graphql',
  path: '/graphql',
  port: 443,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length,
    Authorization: 'BEARER ' + process.env.TOKEN,
    'User-Agent': 'Node',
  },
};

const req = https.request(options, (res) => {
  let data = '';
  console.log(`statusCode: ${res.statusCode}`);

  res.on('data', (d) => {
    data += d;
  });
  res.on('end', () => {
    console.log(JSON.parse(data).data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.write(data);
req.end();

Given Datatypes

For the input createAwsS3SqsIngestFeed(), there are several parameters. Below is a list of them:

Table: CreateAwsS3SqsIngestFeed

ParameterTypeRequiredDefaultDescription
Some arguments may be required, as indicated in the Required column. For some fields, this column indicates that a result will always be returned for this column.
Table last updated: Sep 17, 2024
authenticationIngestFeedAwsAuthenticationInputyes How to authenticate to AWS. See IngestFeedAwsAuthenticationInput.
compressionIngestFeedCompressionyes Compression scheme of the file. See IngestFeedCompression.
descriptionUpdateIngestFeedDescription  The description of the ingest feed. See UpdateIngestFeedDescription.
enabledbooleanyes Whether ingest from the ingest feed is enabled.
namestringyes The name of the ingest feed.
Parserstringyes The unique identifier or name of the parser that should be used to parse the ingest feed. Parsers in packages can be referred to as packagescope/packagename:parsername.
preprocessingIngestFeedPreprocessingInputyes The preprocessing to apply to an ingest feed before parsing. See IngestFeedPreprocessingInput.
regionstringyes The AWS region in which to connect.
repositoryNameRepoOrViewNameyes Name of the repository of the ingest feed. RepoOrViewName is a scalar.
sqsUrlstringyes The AWS SQS queue URL.

Returned Datatypes

The ingestFeed return datatype has its own parameters. Below is a list of them:

Table: IngestFeed

ParameterTypeRequiredDefaultDescription
Some arguments may be required, as indicated in the Required column. For some fields, this column indicates that a result will always be returned for this column.
createdAtlongyes Unix timestamp for when this feed was created.
descriptionstring  A description of the ingest feed.
enabledbooleanyes Whether the ingest from the ingest feed is enabled.
executionInfoIngestFeedExecutionInfo  Details about how the ingest feed is running. See IngestFeedExecutionInfo.
idstringyes The unique identifier of the ingest feed.
namestringyes The name of the ingest feed.
ParserParser  The parser used to parse the ingest feed. See Parser.
sourceIngestFeedSourceyes The source from which the ingest feed will ingest. See IngestFeedSource.