Humio's Internal Logging
Humio has a repository where Humio's internal logs are sent. Humio's logs are by default also written to files. It's possible to configure logging to standard.out, as well. This is described on this page.
Humio's logging is divided into six types. These types are listed below, with the names of their respective logs file in parentheses:
humio-debug.log) — Internal debug logs;
humio-activity.log) — Logs that are relevant to users;
humio-metrics.log) — Metric Types;
humio-requests.log) — All HTTP requests. Like an accesslog in Humio's own format;
humio-non-sensitive.log) — Selected log lines where no searches or user data will be present. This can be shipped to Humio support or other parties; and
humio-threaddumps.log) — Humio regular logs threaddumps
The above logs are automatically rotated by Humio when it reaches 50 megabytes in size. Humio will retain up to five files of each.
All of the above logs are available for search in Humio's internal
repository. When searching Humio's logs in the Humio repository, the tag
#type #kind and
#vhost can be used. All the logs will have
#type=humio. They will have a
#kind tag for each in the list above.
Log events will also have a
vhost tag. Each node in a
Humio cluster has a node number. The #vhost value
indicates which node in the Humio cluster wrote the log event. Below is a
couple of standard searches using the above tags:
#type=humio //will find all events from all hosts of all kinds #type=humio #kind=metrics //search all metrics across all hosts in the cluster #type=humio #kind=metrics #vhost=1 //find all metrics for the node number one
Observe & Monitor Humio with Built-In App
Humio comes with a built in application named, humio/insights. It's present in the Humio repository. The application is a collection of dashboards and saved searches making it possible to monitor and observe the Humio cluster.
If a Humio cluster is having problems, refer to these dashboards. The application also serves as good examples on which to build. The application is also available in the Humio Marketplace and is continuously updated. It can be installed in any given repository from the Marketplace.
Shipping Humio Logs to another Cluster
When running a Humio cluster in production, we highly recommend shipping the logs to another Humio cluster. To do this, install the humio/insights application in the cluster to get out-of-the-box insights.
If a cluster is having problems, it will often not be possible to do searches and debug it. As a last resort, you could grep through files on multiple machines.
It is possible to setup an agent to collect Humio log files and ship them to another Humio cluster. Read the Log Humio to Humio for more information on this.
It is also possible to send the logs to Humio's cloud service. This is
convenient in that you won't have to run and maintain another cluster.
It also makes it possible to share the logs with Humio support. Please
note that Humio cannot guarantee what data is in the
humio-debug.log file. We strive not to log any data
ingested in Humio. Search strings are logged to the debug log.
Configuring Humio Logging
As described above, Humio sends its logs to the internal Humio repository and to files on the file system. It is possible to log to standard out instead. This is often preferred in Docker and Kubernetes environments. It can be configured as shown below:
# HUMIO_LOG4J_CONFIGURATION=log4j2.xml # HUMIO_LOG4J_CONFIGURATION=log4j2-json-stdout.xml # HUMIO_LOG4J_CONFIGURATION=log4j2-stdout.xml HUMIO_LOG4J_CONFIGURATION=file:///path/to/your/log4j2-custom-config.xml
In this example, since only the last line is not commented out, we're
suggesting you provide you own configuration file for log4j2 outside the
jar file. Be aware that the file needs to be similar to the one within
the jar as the internal logging into the humio repository depends on
some of the configuration in that. You can extract the internal version
as a starting point using
Changes to Humio Log Format in 1.19
We have made changes to how Humio logs internally in version 1.19. We
did this to better support the new Insights Package
application and in general improve the logging. These changes affect
events in Humio's own internal repository
changes also result in Humio logging to more files in the file system
than it did before.
Humio's internal logging has been divided into five kinds.
nonsensitive. A tag #kind has
been introduced for these logs in Humio's internal repository. Querying
using the new tag will make many searches much faster. Humio will log
each kind to its own file.
kind=logs -> humio-debug.log
kind=metrics -> humio-metrics.log
kind=requests -> humio-requests.log
kind=nonsensitive -> humio-nonsensitive.log
The log file humio-requests.log is new in this release. The file humio-debug.log has the same format as it has always had. The three other files have a new format. The format is the same for each of these files. The format consists of a timestamp, followed by the log kind, followed by the host number, followed by the log message.
This new format is also used when logging events of these kinds to the
humio repository. The internal Humio parser
has been updated to handle these changes. The built-in dashboards in the
Humio repository has been deleted. Use the new
Insights Package application instead.
vhost tag has also been introduced. vhost
represents the node number for a node in a Humio cluster. Often a host
field is present in Humio events when shipped using agents like Vector
and Filebeat etc. When Humio logged to its internal repository it used
the nodeId field. We want to align the above and make logs more
self-contained by having vhost being part of the standard logline just
like loglevel and class etc.
We have tried to keep the changes as small and compatible as possible, but we have made some changes that can break existing searches in the humio repository (or other repositories receiving Humio logs). We made these changes as we think they are important in order to improve things moving forward. One of the benefits is the new Insights Package.
The above changes can have important impacts on the following:
If logs from the cluster is being shipped to another Humio cluster, for example Humio Cloud, you need to make sure you are shipping all the files. Also the parser in the receiving end should be updated. If the receiving cluster is using the built in Humio parser to parse the logs, and has been updated to at least version 1.19 no changes are needed. We recommend using the Humio parser in the humio/insights app, if you need to update the parser receiving the logs.
We have a guide for Log Humio to Humio.
Custom queries, alerts and dashboards in the humio repository searching for events of the kinds metrics, requests or nonsensitive may need to be modified. These searches are typically using
class= threaddumpor are using any of the previous class names as a freetext string.
Humio events had a field
nodeId, when logged to the
internal humio repository. As we have introduced the new
nodeId has been removed.
When searching for logs of #kind=requests the freetext string "logged
request” was often used -t hat will not work anymore. Logs of
#kind=metrics had a field named type that has been renamed to
metrics_type. That has been changed to These queries can now be written
using the new #kind tag (for example