Falcon LogScale Documentation

There are two ways to quick launch this integration which will create a CloudFormation stack on your AWS account; one which creates a CloudTrail trail and a S3 bucket, and one which does not create these resources. The resources are needed if an automatic subscription to new logs is desired. If you already have these resources available, pick the option without CloudTrail, which will let you use your existing resources for this integration.

Every AWS account only allows for one free CloudTrail, and adding another will therefore be an additional cost when setting up this integration.

If your region is missing, contact us:

The integration is installed using a CloudFormation template.

The template supports the following parameters:

The integration uses a set of AWS Lambdas, where some of these might need to be manually enabled before use. In Configuring the Integration is explained how logs and metrics can be retrieved.

On Linux/MacOS, this is done by using the present makefile:

make

On Windows, create a folder named target in the project root folder, and copy all files from src into it. Then install dependencies into the target folder using pip:

pip3 install -r requirements.txt -t target

In the target folder, zip all files into v0.0.0_YOUR-ZIP-FILE.zip. Choose whatever version number is relevant to you. Then create an AWS S3 bucket using the following command:

aws s3api create-bucket --bucket YOUR-HUMIO-BUCKET --create-bucket-configuration LocationConstraint=YOUR-REGION

The name of the AWS S3 bucket must be the same as the one specified in the CloudFormation file. The default name is humio-public-YOUR-REGION.

Upload the zip file to the AWS S3 bucket:

aws s3 cp target/VERSION_YOUR-ZIP-FILE.zip s3://YOUR-HUMIO-BUCKET/

Next, create a parameters.json file in the project root folder, and specify the CloudFormation parameters, for example:

[
     {
         "ParameterKey": "EnableCloudWatchLogsAutoSubscription",
         "ParameterValue": "true"
     },
     {
         "ParameterKey": "EnableCloudWatchLogsBackfillerAutoRun",
         "ParameterValue": "true"
     },
     {
         "ParameterKey": "EnableVPCForIngesterLambdas",
         "ParameterValue": "true"
     },
     {
         "ParameterKey": "HumioCloudWatchLogsSubscriptionPrefix",
         "ParameterValue": "humio"
     },
     {
         "ParameterKey": "HumioHost",
         "ParameterValue": "cloud.humio.com"
     },
     {
         "ParameterKey": "HumioProtocol",
         "ParameterValue": "https"
     },
     {
         "ParameterKey": "HumioIngestToken",
         "ParameterValue": "YOUR-SECRET-INGEST-TOKEN"
     },
     {
         "ParameterKey": "HumioLambdaLogRetention",
         "ParameterValue": "1"
     },
     {
         "ParameterKey": "HumioLambdaLogLevel",
         "ParameterValue": "WARNING"
     },
     {
         "ParameterKey": "SecurityGroups",
         "ParameterValue": "SECURITY-GROUP-1, SECURITY-GROUP-2"
     },
     {
         "ParameterKey": "SubnetIds",
         "ParameterValue": "SUBNET-ID-1, SUBNET-ID-2, SUBNET-ID-3"
     },
     {
         "ParameterKey": "Version",
         "ParameterValue": "VERSION"
     }
]

Only the HumioIngestToken parameter is required to be specified in the parameters.json file as the rest have default values. The Version must match the version number given in the name of the zip file you have uploaded.

Now, create the stack using the CloudFormation file and the parameters that you have defined:

aws cloudformation create-stack --stack-name YOUR-DESIRED-STACK-NAME --template-body file://cloudformation.json --parameters file://parameters.json --capabilities CAPABILITY_IAM --region YOUR-REGION

The CAPABILITY_IAM is the required value here.

Next, updating the stack. To do so, add your changes and use the command:

aws cloudformation update-stack --stack-name YOUR-DESIRED-STACK-NAME --template-body file://cloudformation.json --parameters file://parameters.json --capabilities CAPABILITY_IAM --region YOUR-REGION

The stack will only register changes in the CloudFormation file or in the parameters.json file. If you have updated the lambda files, you need to change the name of the version of the zip file, and thus the parameter Version for your changes to be recognized.

To delete the stack, use the following command:

aws cloudformation delete-stack --stack-name YOUR-DESIRED-STACK-NAME

For retrieving CloudWatch Logs, the integration uses three AWS Lambdas:

If you only want specific log groups to be ingested into Humio, you can use the HumioCloudWatchLogsSubscriber as this only subscribes the log ingester to one log group at a time. If you want to subscribe to all log groups available, you can use the HumioCloudWatchBackfiller. If you have set the EnableCloudWatchLogsBackfillerAutoRun parameter to true when creating the stack, then you will not have to manually trigger it as it should already have run on creation and subscribed the log ingester to all available log groups. Otherwise, both lambdas need to be enabled using test events.

For the HumioCloudWatchLogsSubscriber lambda, configure your test event like the example below with "EXAMPLE" representing an actual log group, and click Test.

{
  "detail": { "requestParameters": { "logGroupName":"EXAMPLE" }}
}

Figure 227. Event Subscriber

For the HumioCloudWatchLogsBackfiller lambda, use the default test event and click Test. This might take awhile depending on the number of log groups that you are subscribing to.

For retrieving CloudWatch Metrics, the integration can use either one of the two AWS Lambdas:

Both lambdas can essentially retrieve the same information, but there are some differences in their limitations and cost.

There are already defined request parameters for both of these lambdas, and by clicking the Test button for either, using the default test parameters, will make a request retrieving metrics regarding the number of lambda invocations made. The default settings of the request only looks at the metrics from the last 15 minutes.

To change the API request parameters, you can edit the conf_metric_ingester.json file for the HumioCloudWatchMetricIngester lambda, and the conf_metric_statistics_ingester.json file for the HumioCloudWatchMetricStatisticsIngester lambda.

These can be found in your Lambda Console for each lambda under the Function code section.

Figure 228. Configuration File

In the conf_metric_ingester.json, the following code is present:

{
  "MetricDataQueries": [
     {
         "Id": "test_cloudwatch_metrics_lambda_invocations",
         "MetricStat": {
             "Metric": {
                 "Namespace": "AWS/Lambda",
                 "MetricName": "Invocations"
             },
             "Period": 60,
             "Stat": "Sum",
             "Unit": "Count"
         }
     }
  ]
}

This request to the AWS CloudWatch API through a boto3 client retrieves the number of lambda invocations made for the last 15 minutes. The integration will then add a Humio event for each timestamp retrieved.

Change this code accordingly to which metrics you want to retrieve. For more information regarding the different options available, consult the AWS documentation regarding metrics, and for the specifics regarding the request, consult the Boto3 Docs for the get_metric_data(**kwargs) function.

In the conf_metric_statistics_ingester.json, the following code is present:

[
  {
   "Namespace": "AWS/Lambda",
   "MetricName": "Invocations",
   "Period": 60,
   "Statistics": [
     "Sum"
   ]
  },
  {
   "Namespace": "AWS/Lambda",
   "MetricName": "Errors",
   "Period": 60,
   "Statistics": [
     "Sum"
   ]
  }
]

This request to the AWS CloudWatch API through a boto3 client retrieves the number of lambda invocations and errors made for the last 15 minutes. The integration will then add a Humio event for each timestamp retrieved.

Change this code accordingly to which metrics you want to retrieve. For more information regarding the different options available consult, the AWS documentation regarding metrics, and for the specifics regarding the request, consult the Boto3 Docs for the get_metric_statistics(**kwargs) function.

An important thing to notice here, is that the API parameters are wrapped in a list. This is so that more requests can be made with the same function at once.

This lambda handles the delivery of your CloudWatch log events to Humio.

This lambda will subscribe the HumioCloudWatchLogsIngester to a log group every time a new one is created. This is done by filtering CloudTrail events and triggering the HumioCloudWatchLogsSubscriber lambda every time a new log group is created.

This function will paginate through your existing CloudWatch log groups and subscribe the HumioCloudWatchLogsIngester to every single one unless a prefix has been specified.

This lambda handles the delivery of your CloudWatch metrics to Humio. This lambda uses the GetMetricData action from the CloudWatch API reference.

This lambda can also handle the delivery of your CloudWatch metrics to Humio. This lambda uses the GetMetricStatistics action from the CloudWatch API reference.