Internal Logging

sends its own internal logs to the humio repository. logs are also written to files by default. It is possible to configure logging to standard.out, as well. This is described on this page. For specific information about logging to , see Log to .

internal logs are divided into five kinds: logs, activity, metrics, requests, and nonsensitive. These logs use a tag, #kind, and each #kind is logged to its own file. Querying using the tag improves speed for many searches.

logging types are listed below with the names of their respective log files in parentheses:

  • logs (humio-debug.log) — humio debug logs

  • activity (humio-activity.log) — humio-activity that is relevant to users

  • metrics (humio-metrics.log) — humio-metrics

  • requests (humio-requests.log) — All HTTP requests. Like an accesslog in 's own format;

  • nonsensitive (humio-non-sensitive.log) — Selected log lines where no searches or user data will be present. This can be shipped to support or other parties; and

  • threaddumps (humio-threaddumps.log) — regular logs threaddumps

The above logs are automatically rotated by when it reaches 50 megabytes in size. will retain up to five files of each.

All of the above logs are available for search in the humio repository. When searching logs in the humio repository, the tag #type, #kind, and #vhost can be used. All the logs will have #type=humio. They will have a #kind tag for each in the list above.

Log events will also have a vhost tag. Each node in a cluster has a node number. The #vhost value indicates which node in the cluster wrote the log event. Below is a couple of standard searches using the above tags:

logscale
#type=humio  //will find all events from all hosts of all kinds
#type=humio #kind=metrics //search all metrics across all hosts in the cluster
#type=humio #kind=metrics #vhost=1 //find all metrics for the node number one

Observe & Monitor with Insights

comes with a built in application, humio/insights. It's present in the humio repository. The application is a collection of dashboards and saved searches making it possible to monitor and observe the cluster. For more information about the package and its dashboards, see Insights Package.

If a cluster is having problems, the dashboards in the humio/insights package are the best place to start troubleshooting. They also serves as good examples on which to build and customize your own dashboards. The application is also available in the Marketplace and is continuously updated. It can be installed in any given repository from the Marketplace.

Ship logs to another cluster

When running a cluster in production, we highly recommend shipping the logs to another cluster.

To do this, install the humio/insights application in the cluster to get out-of-the-box insights.

If a cluster is having problems, it will often not be possible to do searches and debug it. As a last resort, you could grep through files on multiple machines.

It is possible to setup an agent to collect log files and ship them to another cluster. Read the Log to for more information on this.

Note

may allow customers to ship logs to so that Support can assist with troubleshooting. This is only available for the purposes of enabling diagnoses of specific issues. It is recommend that you deploy your own cluster to support monitoring and troubleshooting.

Please note that cannot guarantee what data is in the humio-debug.log file. strives not to log any data ingested in . Search strings are logged to the debug log.

Configure Logging

As described above, sends its logs to the internal repository and to files on the file system. It is possible to log to standard.out instead. For more information, see HUMIO_LOG4J_CONFIGURATION. This is often preferred in Docker and Kubernetes environments.

You can set custom out if the standard configurations do not meet your requirements using:

ini
HUMIO_LOG4J_CONFIGURATION=file:///path/to/your/log4j2-custom-config.xml
Using a Custom Log Configuration on Bare-Metal Installation

The following procedure can be used to create and use a custom configuration file in bare-metal installation.

  1. Extract the Humio Tarball as described in LogScale Software Installation.

  2. Open a terminal in /opt/humio/humio.x.y.z where x.y.z is the version of you have installed.

  3. Run the following to unzip the jar and create a folder, note that this procedure varies for docker and Kubernetes:

    bash
    $ cd libs
$ unzip context.jar log4j2.xml
$ mkdir ../../my-humio-log4j-config
$ mv log4j2.xml ../../my-humio-log4j-config/log4j2.xml

  4. Edit the file as required and add HUMIO_LOG4J_CONFIGURATION=file:///opt/humio/my-humio-log4j-config/log4j2.xml to your LogScale Configuration Parameters.

Using log4j2.xml to Increase Rollover Rotations

Increasing the amount of rotations between each rollover can be particularly useful for support purposes, as logs rolling-over too quickly may make it difficult to source information easily without continuously sharing the logs.

This example configuration allows you to enable more log files, such as humio-debug.log, to be kept per rotation.

  1. Unzip and open your log4j2.xml file as described above and go to section:

    xml
    <Appenders>
   
   <RollingFile name="humio-file"
   filename="${humio.debuglog.dir}/humio-debug.log"

  2. Under that section for the humio-debug.log file, modify:

    xml
    <DefaultRolloverStrategy max="5" fileIndex="min"/>

    from 5 to 10 which will keep the log files for 10 rotations instead of 5.

    xml
    <DefaultRolloverStrategy max="10" fileIndex="min"/>

  3. Once you have completed your modifications you must restart in order for the changes to take effect.