Amazon CloudWatch Log Format

Humio's CloudWatch integration sends your AWS CloudWatch Logs and Metrics to Humio by using AWS Lambda functions to ship the data.
The newest version of the integration has been released the 12th of
January 2021 where the version was updated to v1.2.1
.
If you have installed the integration prior to this date, you are
recommended to update your current integration to the newest version. See
the section concerning
Updating the
Integration.
Versioning
Versioning has been added to the integration so that upgrading/downgrading between versions is possible. Currently, the following versions are available:
v1.0.0: The initial version.
v1.1.0: A logging library was added so that the log-level of the lambda functions can be adjusted. This can help limit the cost in AWS when using this integration.
v1.2.0: A retention option for the lambda functions' logs has been added.
v1.2.1: A fix for a performance regression.
Quick Installation
Use the Launch Stack buttons below to install in a region of your choice.
Launching Options
There are two ways to quick launch this integration which will create a CloudFormation stack on your AWS account; one which creates a CloudTrail trail and a S3 bucket, and one which does not create these resources. The resources are needed if an automatic subscription to new logs is desired. If you already have these resources available, pick the option without CloudTrail, which will let you use your existing resources for this integration.
Every AWS account only allows for one free CloudTrail, and adding another will therefore be an additional cost when setting up this integration.
Note
If you use an existing CloudTrail it is required that it has the option management events enabled. Use a globally unique name for the stack since the integration uses an S3 bucket which needs to be globally unique.
Launch Stack
If your region is missing, contact us:
Humio Support (
<support@humio.com>
)
Launch Parameters
The integration is installed using a CloudFormation template.
The template supports the following parameters:
EnableCloudWatchLogsAutoSubscription
— Used to enable automatic subscription to new log groups when they are created. The default value istrue
.EnableCloudWatchLogsBackfillerAutoRun
— Used to enable the backfiller to run automatically once created. This will subscribe the log ingester to all present log groups specified with the subscription prefix parameter. Beware that this will delete all current subscriptions to the log groups which will be subscribed! This might mess up any current infrastructure using subscriptions that you have set up. The default value istrue
.EnableVPCForIngesterLambdas
— Used to enable an AWS VPC for all three ingester lambdas. If used, the parameters for security groups and subnet ids also need to be specified. The default value isfalse
.HumioCloudWatchLogsSubscriptionPrefix
— By adding this filter, the Humio logs ingester will only subscribe to log groups whose paths start with this prefix. The default value is "", which will subscribe the log ingester to all log groups.HumioHost
— The host you want to ship your Humio logs to. The default value iscloud.humio.com
.HumioProtocol
— The transport protocol used for delivering log events to Humio.HTTPS
is default and recommended, butHTTP
is possible as well.HumioIngestToken
— The value of your ingest token for the repository from your Humio account that you want your logs delivered to. This parameter is required as you cannot ingest log/metric events into Humio otherwise.HumioLambdaLogRetention
— Number of days that the logs generated by CloudWatch regarding integration's lambda functions will be retained.HumioLambdaLogLevel
— The level of logs that should be logged by the functions. Possible values are, from the highest level to the lowest,DEBUG
,INFO
,WARNING
,ERROR
,CRITICAL
. Default is set toINFO
. The logs are divided into a hierarchy where the higher level will also display the lower level logs, thus setting this option toDEBUG
will enable all logs, whereas setting it toCRITICAL
will only enable logs classified as critical. Setting this option is important in relation to adjust additional costs from CloudWatch since enabling any logs higher thanWARNING
will make the lambda function log more information and the cost of running the integration will be increased. To avoid rudimentary information regarding the integration, yet still log errors, it is recommended to set this parameter to eitherWARNING
orERROR
. For more information regarding how the logging works, see this link.SecurityGroups
— Security groups used by the VPC to group the ingester lambdas. Only required if you are using a VPC.SubnetIds
— Subnet ids used by the VPC to deploy the ingester lambdas into. Only required if you are using a VPC.Version
— The version of the integration to be installed. When creating a new stack, the default value is the newest version. Available releases can be found under releases in the GitHub repository.
The integration uses a set of AWS Lambdas, where some of these might need to be manually enabled before use. In Configuring the Integration is explained how logs and metrics can be retrieved.
Manual Installation
The integration is available at our GitHub repository.
Following is described how to manually install the integration without using the launch buttons.
Pre-Requisites
AWS CLI installed and setup with an AWS account allowed to create a CloudFormation stack.
Python 3x installed.
Setup
Clone the git repository:
https://github.com/humio/cloudwatch2humio
In the project folder, create a zip file with the contents of the
src
folder from the repository.
On Linux/MacOS, this is done by using the present makefile:
make
On Windows, create a folder named target in the project root folder,
and copy all files from src
into it. Then install
dependencies into the target folder using pip:
pip3 install -r requirements.txt -t target
In the target
folder, zip all files into
v0.0.0_YOUR-ZIP-FILE.zip
. Choose whatever version
number is relevant to you. Then create an AWS S3 bucket using the
following command:
aws s3api create-bucket --bucket YOUR-HUMIO-BUCKET --create-bucket-configuration LocationConstraint=YOUR-REGION
The name of the AWS S3 bucket must be the same as the one specified in
the CloudFormation file. The default name is
humio-public-YOUR-REGION
.
Upload the zip file to the AWS S3 bucket:
aws s3 cp target/VERSION_YOUR-ZIP-FILE.zip s3://YOUR-HUMIO-BUCKET/
Next, create a parameters.json
file in the project
root folder, and specify the CloudFormation parameters, for example:
[
{
"ParameterKey": "EnableCloudWatchLogsAutoSubscription",
"ParameterValue": "true"
},
{
"ParameterKey": "EnableCloudWatchLogsBackfillerAutoRun",
"ParameterValue": "true"
},
{
"ParameterKey": "EnableVPCForIngesterLambdas",
"ParameterValue": "true"
},
{
"ParameterKey": "HumioCloudWatchLogsSubscriptionPrefix",
"ParameterValue": "humio"
},
{
"ParameterKey": "HumioHost",
"ParameterValue": "cloud.humio.com"
},
{
"ParameterKey": "HumioProtocol",
"ParameterValue": "https"
},
{
"ParameterKey": "HumioIngestToken",
"ParameterValue": "YOUR-SECRET-INGEST-TOKEN"
},
{
"ParameterKey": "HumioLambdaLogRetention",
"ParameterValue": "1"
},
{
"ParameterKey": "HumioLambdaLogLevel",
"ParameterValue": "WARNING"
},
{
"ParameterKey": "SecurityGroups",
"ParameterValue": "SECURITY-GROUP-1, SECURITY-GROUP-2"
},
{
"ParameterKey": "SubnetIds",
"ParameterValue": "SUBNET-ID-1, SUBNET-ID-2, SUBNET-ID-3"
},
{
"ParameterKey": "Version",
"ParameterValue": "VERSION"
}
]
Only the HumioIngestToken
parameter is required to
be specified in the parameters.json
file as the
rest have default values. The Version
must match
the version number given in the name of the zip file you have
uploaded.
Now, create the stack using the CloudFormation file and the parameters that you have defined:
aws cloudformation create-stack --stack-name YOUR-DESIRED-STACK-NAME --template-body file://cloudformation.json --parameters file://parameters.json --capabilities CAPABILITY_IAM --region YOUR-REGION
The CAPABILITY_IAM is the required value here.
Next, updating the stack. To do so, add your changes and use the command:
aws cloudformation update-stack --stack-name YOUR-DESIRED-STACK-NAME --template-body file://cloudformation.json --parameters file://parameters.json --capabilities CAPABILITY_IAM --region YOUR-REGION
The stack will only register changes in the CloudFormation file or in the parameters.json file. If you have updated the lambda files, you need to change the name of the version of the zip file, and thus the parameter Version for your changes to be recognized.
To delete the stack, use the following command:
aws cloudformation delete-stack --stack-name YOUR-DESIRED-STACK-NAME
Configuring the Integration
The integration might need to be manually activated before logs and metrics are shipped to Humio.
Retrieving CloudWatch Logs
For retrieving CloudWatch Logs, the integration uses three AWS Lambdas:
Note
If EnableCloudWatchLogsBackfillerAutoRun is set to true an additional custom resource will be created — namely the HumioBackfillerAutoRunner.
If you only want specific log groups to be ingested into Humio, you
can use the HumioCloudWatchLogsSubscriber
as this
only subscribes the log ingester to one log group at a time. If you
want to subscribe to all log groups available, you can use the
HumioCloudWatchBackfiller
. If you have set the
EnableCloudWatchLogsBackfillerAutoRun
parameter to
true
when creating the stack, then you will not
have to manually trigger it as it should already have run on creation
and subscribed the log ingester to all available log groups.
Otherwise, both lambdas need to be enabled using test events.
For the HumioCloudWatchLogsSubscriber
lambda,
configure your test event like the example below with "EXAMPLE"
representing an actual log group, and click Test.
{
"detail": { "requestParameters": { "logGroupName":"EXAMPLE" }}
}

Figure 215. Event Subscriber
For the HumioCloudWatchLogsBackfiller
lambda, use
the default test event and click Test. This might take awhile
depending on the number of log groups that you are subscribing to.
Retrieving CloudWatch Metrics
For retrieving CloudWatch Metrics, the integration can use either one of the two AWS Lambdas:
Both lambdas can essentially retrieve the same information, but there are some differences in their limitations and cost.
There are already defined request parameters for both of these lambdas, and by clicking the Test button for either, using the default test parameters, will make a request retrieving metrics regarding the number of lambda invocations made. The default settings of the request only looks at the metrics from the last 15 minutes.
To change the API request parameters, you can edit the
conf_metric_ingester.json
file for the
HumioCloudWatchMetricIngester
lambda, and the
conf_metric_statistics_ingester.json
file for the
HumioCloudWatchMetricStatisticsIngester
lambda.
These can be found in your Lambda Console for each lambda under the Function code section.

Figure 216. Configuration File
Setting the HumioCloudWatchMetricIngester API Parameters
In the conf_metric_ingester.json
, the following
code is present:
{
"MetricDataQueries": [
{
"Id": "test_cloudwatch_metrics_lambda_invocations",
"MetricStat": {
"Metric": {
"Namespace": "AWS/Lambda",
"MetricName": "Invocations"
},
"Period": 60,
"Stat": "Sum",
"Unit": "Count"
}
}
]
}
This request to the AWS CloudWatch API through a boto3 client retrieves the number of lambda invocations made for the last 15 minutes. The integration will then add a Humio event for each timestamp retrieved.
Change this code accordingly to which metrics you want to retrieve. For more information regarding the different options available, consult the AWS documentation regarding metrics, and for the specifics regarding the request, consult the Boto3 Docs for the get_metric_data(**kwargs) function.
Setting the HumioCloudWatchMetricStatisticsIngester API Parameters
In the conf_metric_statistics_ingester.json
, the
following code is present:
[
{
"Namespace": "AWS/Lambda",
"MetricName": "Invocations",
"Period": 60,
"Statistics": [
"Sum"
]
},
{
"Namespace": "AWS/Lambda",
"MetricName": "Errors",
"Period": 60,
"Statistics": [
"Sum"
]
}
]
This request to the AWS CloudWatch API through a boto3 client retrieves the number of lambda invocations and errors made for the last 15 minutes. The integration will then add a Humio event for each timestamp retrieved.
Change this code accordingly to which metrics you want to retrieve. For more information regarding the different options available consult, the AWS documentation regarding metrics, and for the specifics regarding the request, consult the Boto3 Docs for the get_metric_statistics(**kwargs) function.
An important thing to notice here, is that the API parameters are wrapped in a list. This is so that more requests can be made with the same function at once.
Updating the Integration
When the codebase for the integration is updated, you are recommended to update your current integration setup as well. (See the Versioning section for when the integration was last updated.)
To update the cloudwatch2humio integration go to Services > CloudFormation, find the cloudwatch2humio stack and click its Update > Replace Current Template. Use either of the following links depending on which launch option you want:
Under Version in the parameter section write the version you want to use
with the format vX.X.X
but replacing the Xs with the
desired version.
Deleting the Integration
To delete the cloudwatch2humio integration, you need to find its stack in Services > CloudFormation, and click its Delete button.
The most common error that happens when trying to delete the stack
concerns the S3 bucket associated with CloudTrail. (Note: This S3 bucket
is only created if the parameter
EnableCloudWatchLogsAutoSubscription
is set to true.)
As this bucket gathers events, it tends not to be empty, and
CloudFormation is not allowed to delete a bucket which is not empty.

Figure 217. Delete Stack

Figure 218. Resources
To fix this error and delete the stack, go to your view of S3 buckets, Services > S3, and mark the offending bucket and click the "Empty" button.

Figure 219. Empty Bucket
Now you should be able to delete the stack. The bucket is automatically updated with CloudTrail events, so you have to make sure you empty the bucket right before deleting the stack.
How the Integration Works
The integration will install five lambda functions:
HumioCloudWatchLogsIngester
,HumioCloudWatchLogsSubscriber
,HumioCloudWatchLogsBackfiller
,HumioCloudWatchMetricIngester
,HumioCloudWatchMetricStatisticsIngester
.
If EnableCloudWatchLogsAutoSubscription
is set to
true
, then the CloudFormation template will also set
up CloudTrail and a S3 bucket for your account. This is needed to
trigger the HumioCloudWatchLogsSubscriber
lambda
automatically to newly created log groups.
If EnableCloudWatchLogsBackfillerAutoRun
is set to
true
, then the CloudFormation template will also
create a custom resource named
HumioBackfillerAutoRunner
. This is needed to trigger
the HumioCloudWatchLogsBackfiller
lambda
automatically when created, which will subscribe the
HumioCloudWatchLogsSubscriber
to all present log
groups in your account, and override any already existing subscriptions.
The HumioLambdaLogRetention
parameter determines how
many days the logs for the integration's lambda functions are stored in
CloudWatch.
The HumioLambdaLogLevel
parameter determines which
logs from the lambda functions are logged in Cloudwatch. The parameter
may be set to one of the following log-levels:
DEBUG
— Used in relation to diagnosing problems.INFO
— Used in relation to expected behavior.WARNING
— Used in relation to unexpected behavior.ERROR
— Used in relation to errors.CRITICAL
— Used in relation to serious errors. No logs in the integration are considered critical, so selecting this option will disable all logs created by code from the lambda functions.
See the python docs for more information regarding the logging library used by the lambda functions.
HumioCloudWatchLogsIngester
This lambda handles the delivery of your CloudWatch log events to Humio.
HumioCloudWatchLogsSubscriber
This lambda will subscribe the
HumioCloudWatchLogsIngester
to a log group every
time a new one is created. This is done by filtering CloudTrail events
and triggering the HumioCloudWatchLogsSubscriber
lambda every time a new log group is created.
HumioCloudWatchLogsBackfiller
This function will paginate through your existing CloudWatch log
groups and subscribe the
HumioCloudWatchLogsIngester
to every single one
unless a prefix has been specified.
Note
Beware that the function also will delete all current subscriptions
to any targeted log groups! This will ruin any current
infrastructure set up with subscriptions to these. Log groups will
be targeted based on the
HumioCloudWatchLogsSubscriptionPrefix
which is
set in the launch parameters.
HumioCloudWatchMetricIngester
This lambda handles the delivery of your CloudWatch metrics to Humio. This lambda uses the GetMetricData action from the CloudWatch API reference.
HumioCloudWatchMetricStatisticsIngester
This lambda can also handle the delivery of your CloudWatch metrics to Humio. This lambda uses the GetMetricStatistics action from the CloudWatch API reference.