Anyone can build a Humio package and submit it to Humio for inclusion in the marketplace. The marketplace provides a way for Humio, customers and partners to share Humio assets which save time and encourage people to find new ways to do more with Humio. Humio are keen that all packages can be freely used and edited by anyone to fuel creativity in the spirit of the open source community. Therefore all packages in the marketplace are made available under the Apache 2.0 license.
We have developed these guidelines to help developers ensure their package can be shared in the marketplace. These guidelines will evolve as the marketplace grows and we welcome your feedback on this document and the packages and marketplace as a whole. We are keen to support and help people building packages for the marketplace so please get in touch with any questions or if you require any help.
When considering building and sharing a new Humio package, it is important to evaluate if the package already exists; if it does then you should iterate the existing package rather than build a new one. The Humio Marketplace will not accept duplicate packages or packages with substantial overlapping capability.
All Marketplace submissions are made under the Apache 2.0 license. In the event that a duplicate package is submitted, or significant overlap between packages is identified, Humio may decide to combine the assets.
Some examples where an existing package should not be duplicated:
Apache httpd Web Server
AWS VPC Flow
If you did feel you had a unique package related to the functionality already included here, a good question to ask is “would my new package be useful without the existing package”, or “would the same group of people always be interested in both”.
To create a package in Humio, you first need to have a working set of Humio components such as parsers, queries, dashboards or alerts that meet a particular need. To do this you need to have access to the relevant logs and to have tested that the package works as intended. When preparing to create a package please keep a careful record of the versions and configurations that you used in testing as well as the relevant log files and how they were generated and/or collected.
When developing a package aimed at being published on the Humio Marketplace, it should preferably be developed against the latest version of Humio where possible. If the latest version is not readily available to you, it is recommended that you use the Humio docker image to deploy a temporary development environment.
Once you have created and tested the assets which you would like to include in the package, go toinside your repository, and complete the package descriptions as per the below guidance. To ensure your package can be included in the marketplace please pay close attention to the notes and explanations below.
The package scope and name provide a unique identifier for the package. For the scope, use the vendor/manufacturer of the technology to which the package relates. If the package uses logs from multiple different vendors/manufacturers, then you can enter the scope as your company name or relevant industry standard or framework. Scope should be entered as lower case with only standard text characters. See below for some examples.
Technology / Vendor
Amazon Web Services
Some AWS services are referred to as aws and some Amazon. The scope should always be aws but the right name should be used in the package README.
Azure apps are distinct from on-prem/traditional Microsoft packages.
Google Cloud Platform
This includes GSuite, as Google continues to integrate these services.
Apache Software Foundation
All projects related to the Apache Software Foundation (httpd, kafka, etc). App versions specific to a commercial variant of these technologies should use an alternative scope, for example
Where a vendor only offers a single technology, service, or tool, the scope and name may be the same. It is acceptable to therefore have a scope name combination like
Center for Internet Security (CIS)
Where the package supports multiple log source types use a scope of any related frameworks or standards if possible.
The package name should be the specific vendor’s product name to which the package relates, or the use case for the package. The name should be entered as lower case with only standard text characters.
AWS VPC Flow would use the scope
aws and the name
vpcflow, so the final package identifier becomes
aws/vpcflow. A package for Okta is likely to use the same name and scope, so the identifier becomes
okta/okta. A use-case specific package, such as one based on the CIS Foundations Benchmark for AWS, would use
cis for the scope, and
aws_foundations_benchmark or similar for the name.
Humio recommends following the [Semantic Versioning standards](https://semver.org/) for package versions. From the standards: Given a version number MAJOR.MINOR.PATCH, increment the:
MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards compatible manner, and
PATCH version when you make backwards compatible bug fixes.
“API incompatible changes” for the purposes of a Humio package would be where the log format has changed, or the way the data needs to be parsed is changed in an incompatible way with data users have already ingested. It is critical that any change to a package that is not compatible with previous parsing and/or field naming convention is released as a new major version.
Select Application for the package components to be automatically installed in the target repository; i.e. when a user deploys an “Application” type package, the assets in that package are automatically deployed into the repository.
Select Library to not install any components automatically, but to install templates from which customers can create and install the components later.
This text is displayed next to the package name in the marketplace. It should be 65 characters or less and provide a brief description of what the package does.
Example: “A parser and sample overview dashboard for Amazon VPC Flow Logs”
Provide a link to a suitable icon, that is displayed next to the package in the marketplace. For a package specific to a certain vendor/manufacturer, please use their standard logo and ensure that the logo you use conforms to their guidelines on usage, and that you are allowed to use it in this way. For example, if creating an AWS package then the 64px PNG icons from the AWS asset pack must be used to generate the icon.
Provide the link in either of the
data:url: formats. The icon should be 64x64 px PNG or SVG on a transparent background.
It is highly recommended to encode the icon as a
data:url, and there are various assets online that make this conversion straightforward, search “png to data uri conversion” for options.
For inclusion in the marketplace, Humio requires the package to be usable under Apache 2.0 license. This is the default choice when creating a package using the Humio UI.
For inclusion in the marketplace, Humio requires the
README part of the package to follow the below format as consistently as possible. This is to ensure simplicity for customers. Humio intends to incorporate some of these points as top level package metadata at some point in the future.
README is written in Markdown, and should incorporate the following headlines. The comments provide guidance, but should be removed from the final README.
# <!--Title The README should start with a title on it's own line that summarises the package, this should be less than 40 characters long. Underneath the title there should be a description of what it does and why. Focus on what log sources does it apply to, what data does it use and what does it show from that data. Make it really clear as to what the purpose of the package is. You can include links to external websites as long as they are directly relevant and help explain the package or add context. You can include links to Humio's documentation library to explain concepts and provide technical details or helpful tips.--> ## Package Contents <!--List which of below are included. (you can include a description of each component if you wish but limited to 100 characters each. All packages need to include the required parsers. This may change in the future but for now a working parser must be included (it can be subject to dependencies as listed in the dependencies section below).--> Parser Queries Alerts Dashboards ## Use Case <!-- (list all that apply from the selection of DevOps, SecOps, ITOps, IoT/OT) --> - DevOps <!-- used by devops teams to provide observability of applications and the platforms the code runs on, including containerised and micro-services type environments --> - SecOps <!-- useful for security operations teams, typically to provide security monitoring and SIEM like functionality. --> - ITOps <!-- used by IT operations teams to monitor IT systems and to provide investigations and root cause analysis --> - IoT/OT <!-- used to provide security and/or operational management of Internet of Things devices or Operational Technology such as Scada and industrial process control and monitoring.--> ## Technology Vendor <!-- (list of vendors supported by package) e.g. AWS (Amazon Web Services) or Cisco. --> ## Support <!-- list ONE of the following (Humio, Developer, None) to indicate who will be offering support to customers using your package. Humio support is currently only available for packages created by Humio but this may change in the future.--> Humio (Humio commit to support the package and will deal with issues as a P3. Humio will ensure it is updated and continues to work with Humio until EOL and then EOS.) Developer <!-- start with the word 'Developer' and then email or contact details for support. Can also include a URL to a site explaining what support is offered including any SLAs or relevant policies. When selecting Developer you are committing to offer a reasonable level of support for the package. We understand that this may not include any SLAs or promises but we expect that it will meet the expectations of a typical customer and that includes a timely response to questions and issues. Humio will monitor feedback from marketplace customers and if it is clear that Support is not meeting expectations Humio reserve the right to change the Support option to be listed as ‘None'.--> None. Community support may be available through our Humio slack channel but with no commitments. ## Dependencies <!--List dependencies for your package to work. These will typically include versions or configurations of the log sources and often the log collection method and formatting of logs. You can include a sample line of log data to illustrate the required format.--> ## Installation <!--Instructions on how to install and use the package. Must include any required configuration in external systems such as log sources as well as explaining any optional configurations or choices that need to be made.-->
# Amazon VPC Flow Logs This Amazon VPC package can be used to parse incoming default VPC Flow Logs from AWS. VPC Flow Logs is a feature of AWS that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. Flow log data can be published to Amazon CloudWatch Logs or Amazon S3. After you've created a flow log, you can retrieve and view its data in the chosen destination. The parser included is based on data collected from S3 using the default format, but can be very easily adapted to a custom format. For more details on why your VPC Flow Logs are important, and the implications of what you can do with them, please refer to the AWS documentation here. ## Package Contents - Parser for default VPC Flow logs in RAW format - Sample Queries - VPC Overview Dashboard ## Use Case - DevOps - SecOps - ITOps - IoT/OT ## Technology Vendor AWS (Amazon Web Services) ## Support Humio (Humio commit to support the package and will deal with issues as a P3. Humio will ensure it is updated and continues to work with Humio until EOL and then EOS). Please contact email@example.com with any issues. ## Dependencies This package relies on ingesting Amazon VPC Flow Logs in a RAW format. These logs are typically routed to an S3 bucket and stored in a compressed format. When configuring collection of these logs this package expects the log's @rawstring to be set to a values as follows: 2 051856882135 eni-00770bc8b36e4698a 172.24.54.150 172.24.24.102 9094 46990 6 770 470591 1611840313 1611840325 ACCEPT OK 2 051856882135 eni-00770bc8b36e4698a 172.24.24.102 172.24.54.150 46990 9094 6 622 62217 1611840313 1611840325 ACCEPT OK 2 051856882135 eni-00770bc8b36e4698a 172.24.54.150 172.24.24.102 9094 48864 6 20 9750 1611840313 1611840325 ACCEPT OK If your data collection wraps the message with additional information, or the format for the VPC Flow Logs is not the default, you can adapt the parser to match your data format. ## Installation Installation of the package is straightforward, the installation will deploy the parser, saved queries, and dashboards directly into the repository you install the package into. You will need to configure your ingest via an ingest token to use the new vpcflow_raw parser. - This package has a dependency on two lookup files. These files are described below, and they require some of the additional columns to be removed before uploading to Humio. - protocol-numbers-1.csv This file can be downloaded from here, and can be directly uploaded to the Files of the repository using the default filename. - service-names-port-numbers.csv This file can be downloaded here. It is necessary to remove the columns to the right of Description (i.e. you should only have the columns: Service Name, Port Number, Transport Protocol, Description).
Note that the README provides details ensuring that a user knows at least how their ingested data should arrive to ensure it is compatible with the parsers in the package. It is highly desirable that the README provides details on the exact procedures for data collection (i.e. a howto on getting the specific data source(s) into Humio).
Once you have completed the
README, you can select the components to include in the package. Tick the relevant components and when finished select Download Package. Humio will create a zip file. This file can be manually uploaded into Humio for sharing within your own Humio instance.
It is recommended that you unzip the contents of this package and commit it to version control (e.g. git). This makes the process of modifying and bug fixing the package far simpler, and at a very minimum, gives you a mechanism by which you can review the diff between current and previous versions of your package.
We would love you to share your packages with other Humio customers via our marketplace. Currently the marketplace is only available from within the Humio product UI but in future, the marketplace will also be available from the Humio public website.
After receiving the package and the required additional information detailed below, Humio will check the package by following a simple certification process which is explained here. Once complete (following any required modifications) the package will be made available on the marketplace. To ask for help or advice please get in touch with us via our public Slack channel.
To send a package to Humio, email the zip file containing the package to <firstname.lastname@example.org>. Please be sure to include the following information within the email:
You can include contributors in the package metadata. This will be displayed in the marketplace next to the package. Contributors can list the names of developers and can include email addresses if you are happy for this to be visible in the marketplace.
To ensure that Humio can test the package functionality we need to have some available log files which include all the events that are relevant to the package. Ideally Humio would like log data over a 24 hour period, but if all event types can be covered in a shorter period of time that may be acceptable. Depending on the size of the log files either attach the log files to the email or provide a link to the files and any required credentials to access these. The sample logs are mandatory for package review, and must:
Exercise all panels, alerts, and queries in the package
Be a representative sample of real event distribution for the package
Not contain any PII or other sensitive information
The log file samples are covered by the same Apache 2.0 license as your package submission. Humio may use these sample logs to generate demo environments, screenshots, marketing materials, etc. When developing or extending an existing package Humio may make these sample logs available to other developers.
On receipt of the package and required information detailed above Humio will check that the package meets our guidelines and requirements as follows. This is a simple process and we are here to help so if you are unsure of anything please get in touch with us!
The package works with the provided log samples. Testing will include, but is not limited to: Data is parsed correctly and all dashboards and alerts work as expected and provide a good overall user experience.
The provided descriptions and metadata included within the package, including the README and the additional information provided via the email, correctly describe the package and are clear and unambiguous and comply with the requirements in this document. The available sample log files cover all relevant events.
Humio components of the package (parsers, queries, dashboard, alerts) are implemented in an efficient manner using the optimal implementation and configurations. Queries are constructed efficiently and with the optimal query language, dashboards display clearly with appropriate labels etc. For any guidance on Humio’s recommendations please consult our documentation on this site, and if in doubt please contact us for assistance via <email@example.com> or on our public Slack channel.
All links within the package descriptions and metadata are checked for malware.
All company names and logos are used appropriately.
The package does not duplicate an existing package or overlap in functionality so significantly that it would make more sense to make a change to an existing package.
If any aspect of the package requires changes, then Humio will identify these to the package creator and offer advice on how to make the changes needed to be included in the marketplace. Humio may also make the changes directly during the package review.