Humio’s Internal Logging

Humio has a repository where Humio’s internal logs are sent. Humio’s logs are by default also written to files. It’s possible to configure logging to standard.out, as well. This is described on this page.

Humio’s logging is divided into six types. These types are listed below, with the names of their respective logs file in parentheses:

  • logs (humio-debug.log) — Internal debug logs;

  • activity (humio-activity.log) — Logs that are relevant to users;

  • metrics (humio-metrics.log) — Metric logs;

  • requests (humio-requests.log) — All HTTP requests. Like an accesslog in Humio’s own format;

  • nonsensitive (humio-non-sensitive.log) — Selected log lines where no searches or user data will be present. This can be shipped to Humio support or other parties; and

  • threaddumps (humio-threaddumps.log) — Humio regular logs threaddumps

The above logs are automatically rotated by Humio when it reaches 50 megabytes in size. Humio will retain up to five files of each.

All of the above logs are available for search in Humio’s internal repository. When searching Humio’s logs in the Humio repository, the tag #type #kind and #vhost can be used. All the logs will have #type=humio. They will have a #kind tag for each in the list above.

Log events will also have a vhost tag. Each node in a Humio cluster has a node number. The #vhost value indicates which node in the Humio cluster wrote the log event. Below is a couple of standard searches using the above tags:

ini
  #type=humio  //will find all events from all hosts of all kinds
  #type=humio #kind=metrics //search all metrics across all hosts in the cluster
  #type=humio #kind=metrics #vhost=1 //find all metrics for
the node number one

Observe & Monitor Humio with Built-In App

Humio comes with a built in application named, humio/insights. It’s present in the Humio repository. The application is a collection of dashboards and saved searches making it possible to monitor and observe the Humio cluster.

If a Humio cluster is having problems, refer to these dashboards. The application also serves as good examples on which to build. The application is also available in the Humio Marketplace and is continuously updated. It can be installed in any given repository from the Marketplace.

Shipping Humio Logs to another Cluster

When running a Humio cluster in production, we highly recommend shipping the logs to another Humio cluster. To do this, install the humio/insights application in the cluster to get out-of-the-box insights.

If a cluster is having problems, it will often not be possible to do searches and debug it. As a last resort, you could grep through files on multiple machines.

It is possible to setup an agent to collect Humio log files and ship them to another Humio cluster. Read the Shipping Humio Logs to another Cluster for more information on this.

It is also possible to send the logs to Humio’s cloud service. This is convenient in that you won’t have to run and maintain another cluster. It also makes it possible to share the logs with Humio support. Please note that Humio cannot guarantee what data is in the humio-debug.log file. We strive not to log any data ingested in Humio. Search strings are logged to the debug log.

Configuring Humio Logging

As described above, Humio sends its logs to the internal Humio repository and to files on the file system. It is possible to log to standard out instead. This is often preferred in Docker and Kubernetes environments. It can be configured as shown below:

ini
# HUMIO_LOG4J_CONFIGURATION=log4j2.xml
# HUMIO_LOG4J_CONFIGURATION=log4j2-json-stdout.xml
# HUMIO_LOG4J_CONFIGURATION=log4j2-stdout.xml

HUMIO_LOG4J_CONFIGURATION=file:///path/to/your/log4j2-custom-config.xml

In this example, since only the last line is not commented out, we’re suggesting you provide you own configuration file for log4j2 outside the jar file. Be aware that the file needs to be similar to the one within the jar as the internal logging into the humio repository depends on some of the configuration in that. You can extract the internal version as a starting point using unzip humio-assembly-0.1.jar log4j2.xml.

Changes to Humio Log Format in 1.19

We have made changes to how Humio logs internally in version 1.19. We did this to better support the new humio/insights application and in general improve the logging. These changes affect events in Humio’s own internal repository humio. The changes also result in Humio logging to more files in the file system than it did before.

Humio’s internal logging has been divided into five kinds. logs, activity, metrics, requests and nonsensitive. A tag #kind has been introduced for these logs in Humio’s internal repository. Querying using the new tag will make many searches much faster. Humio will log each kind to its own file.

  • kind=logs -> humio-debug.log

  • kind=metrics -> humio-metrics.log

  • kind=requests -> humio-requests.log

  • kind=nonsensitive -> humio-nonsensitive.log

The log file humio-requests.log is new in this release. The file humio-debug.log has the same format as it has always had. The three other files have a new format. The format is the same for each of these files. The format consists of a timestamp, followed by the log kind, followed by the host number, followed by the log message.

This new format is also used when logging events of these kinds to the internal humio repository. The internal Humio parser has been updated to handle these changes. The built-in dashboards in the Humio repository has been deleted. Use the new humio/insights application instead.

A vhost tag has also been introduced. vhost represents the node number for a node in a Humio cluster. Often a host field is present in Humio events when shipped using agents like Vector and Filebeat etc. When Humio logged to its internal repository it used the nodeId field. We want to align the above and make logs more self-contained by having vhost being part of the standard logline just like loglevel and class etc.

We have tried to keep the changes as small and compatible as possible, but we have made some changes that can break existing searches in the humio repository (or other repositories receiving Humio logs). We made these changes as we think they are important in order to improve things moving forward. One of the benefits is the new Humio Insights App.

The above changes can have important impacts on the following:

  • If logs from the cluster is being shipped to another Humio cluster, for example Humio Cloud, you need to make sure you are shipping all the files. Also the parser in the receiving end should be updated. If the receiving cluster is using the built in Humio parser to parse the logs, and has been updated to at least version 1.19 no changes are needed. We recommend using the Humio parser in the humio/insights app, if you need to update the parser receiving the logs.

We have a guide for Shipping Humio Logs to another Humio Cluster.

  • Custom queries, alerts and dashboards in the humio repository searching for events of the kinds metrics, requests or nonsensitive may need to be modified. These searches are typically using class=c.h.metrics, class=c.h.l.NonSensitiveLogger, class=a.a.ActorSystemImpl or class= threaddump or are using any of the previous class names as a freetext string.

Humio events had a field nodeId, when logged to the internal humio repository. As we have introduced the new vhost tag nodeId has been removed. When searching for logs of #kind=requests the freetext string “logged request” was often used -t hat will not work anymore. Logs of #kind=metrics had a field named type that has been renamed to metrics_type. That has been changed to These queries can now be written using the new #kind tag (for example #kind=metrics).