LogScale Internal Logging

LogScale has a repository where LogScale internal logs are sent. LogScale logs are by default also written to files. It's possible to configure logging to standard.out, as well. This is described on this page.

LogScale logging types are listed below, with the names of their respective log files in parentheses:

  • logs (humio-debug.log) — humio debug logs

  • activity (humio-activity.log) — humio-activity that is relevant to users

  • metrics (humio-metrics.log) — humio-metrics

  • requests (humio-requests.log) — All HTTP requests. Like an accesslog in LogScale own format;

  • nonsensitive (humio-non-sensitive.log) — Selected log lines where no searches or user data will be present. This can be shipped to LogScale support or other parties; and

  • threaddumps (humio-threaddumps.log) — LogScale regular logs threaddumps

The above logs are automatically rotated by LogScale when it reaches 50 megabytes in size. LogScale will retain up to five files of each.

All of the above logs are available for search in LogScale internal repository. When searching LogScale logs in the humio repository, the tag #type #kind and #vhost can be used. All the logs will have #type=humio. They will have a #kind tag for each in the list above.

Log events will also have a vhost tag. Each node in a LogScale cluster has a node number. The #vhost value indicates which node in the LogScale cluster wrote the log event. Below is a couple of standard searches using the above tags:

logscale
#type=humio  //will find all events from all hosts of all kinds
#type=humio #kind=metrics //search all metrics across all hosts in the cluster
#type=humio #kind=metrics #vhost=1 //find all metrics for the node number one

Observe & Monitor LogScale with Insights

LogScale comes with a built in application named, humio/insights. It's present in the humio repository. The application is a collection of dashboards and saved searches making it possible to monitor and observe the LogScale cluster.

If a LogScale cluster is having problems, refer to these dashboards. The application also serves as good examples on which to build. The application is also available in the LogScale Marketplace and is continuously updated. It can be installed in any given repository from the Marketplace.

Shipping LogScale Logs to another Cluster

When running a LogScale cluster in production, we highly recommend shipping the logs to another LogScale cluster.

To do this, install the humio/insights application in the cluster to get out-of-the-box insights.

If a cluster is having problems, it will often not be possible to do searches and debug it. As a last resort, you could grep through files on multiple machines.

It is possible to setup an agent to collect LogScale log files and ship them to another LogScale cluster. Read the Log LogScale to LogScale for more information on this.

Note

LogScale may allow customers to ship logs to LogScale so that LogScale Support can assist with troubleshooting. This is only available for the purposes of enabling diagnoses of specific issues. It is recommend that you deploy your own cluster to support monitoring and troubleshooting.

Please note that LogScale cannot guarantee what data is in the humio-debug.log file. We strive not to log any data ingested in LogScale. Search strings are logged to the debug log.

Configuring LogScale Logging

As described above, LogScale sends its logs to the internal LogScale repository and to files on the file system. It is possible to log to standard out instead. See for more information HUMIO_LOG4J_CONFIGURATION, this is often preferred in Docker and Kubernetes environments.

You can set custom out if the standard configurations do not meet your requirements using:

ini
HUMIO_LOG4J_CONFIGURATION=file:///path/to/your/log4j2-custom-config.xml
Using a Custom Log Configuration on Bare-Metal Installation

The following procedure can be used to create and use a custom configuration file in bare-metal installation.

  1. Extract the Humio Tarball as described in LogScale Software Installation.

  2. Open a terminal in /opt/humio/humio.x.y.z where x.y.z is the version of LogScale you have installed.

  3. Run the following to unzip the jar and create a folder, note that this procedure varies for docker and Kubernetes:

    bash
    $ cd libs
$ unzip context.jar log4j2.xml
$ mkdir ../../my-humio-log4j-config
$ mv log4j2.xml ../../my-humio-log4j-config/log4j2.xml

  4. Edit the file as required and add HUMIO_LOG4J_CONFIGURATION=file:///opt/humio/my-humio-log4j-config/log4j2.xml to your LogScale Configuration Parameters.

Using log4j2.xml to Increase Rollover Rotations

Increasing the amount of rotations between each roll over can particularly useful for support purposes, as logs rolling-over too quickly may make it difficult to easily source information without continuously sharing the logs.

This example configuration allows you to enable more log files such as humio-debug.log to be kept per rotation.

  1. Unzip and open your log4j2.xml file as described above and go to section:

    xml
    <Appenders>
   
   <RollingFile name="humio-file"
   filename="${humio.debuglog.dir}/humio-debug.log"

  2. Under that section, which is made for the humio-debug.log file, modify this:

    xml
    <DefaultRolloverStrategy max="5" fileIndex="min"/>

    from 5 to 10 which will keep the log files for 10 rotations instead of 5.

    xml
    <DefaultRolloverStrategy max="10" fileIndex="min"/>

  3. Once you have completed your modifications you must restart LogScale in order for the changes to take effect.

Internal Logging Format

LogScale internal logging has been divided into five kinds. logs, activity, metrics, requests and nonsensitive. A tag #kind has been introduced for these logs in LogScale internal repository. Querying using the new tag will make many searches much faster. LogScale will log each kind to its own file.

  • kind=logs -> humio-debug.log

  • kind=metrics -> humio-metrics.log

  • kind=requests -> humio-requests.log

  • kind=nonsensitive -> humio-nonsensitive.log

The log file humio-requests.log is new in this release. The file humio-debug.log has the same format as it has always had. The three other files have a new format. The format is the same for each of these files. The format consists of a timestamp, followed by the log kind, followed by the host number, followed by the log message.

This new format is also used when logging events of these kinds to the internal humio repository. The internal LogScale parser has been updated to handle these changes. The built-in dashboards in the LogScale repository has been deleted. Use the new Insights Package application instead.

A vhost tag has also been introduced. vhost represents the node number for a node in a LogScale cluster. Often a host field is present in LogScale events when shipped using agents like Vector and Filebeat etc. When LogScale logged to its internal repository it used the nodeId field. We want to align the above and make logs more self-contained by having vhost being part of the standard logline just like loglevel and class etc.

We have tried to keep the changes as small and compatible as possible, but we have made some changes that can break existing searches in the humio repository (or other repositories receiving LogScale logs). We made these changes as we think they are important in order to improve things moving forward. One of the benefits is the new Insights Package.

The above changes can have important impacts on the following:

  • If logs from the cluster is being shipped to another LogScale cluster, for example LogScale Cloud, you need to make sure you are shipping all the files. Also the parser in the receiving end should be updated. If the receiving cluster is using the built in LogScale parser to parse the logs, and has been updated to at least version 1.19 no changes are needed. We recommend using the LogScale parser in the humio/insights app, if you need to update the parser receiving the logs.

We have a guide for Log LogScale to LogScale.

  • Custom queries, alerts and dashboards in the humio repository searching for events of the kinds metrics, requests or nonsensitive may need to be modified. These searches are typically using class=c.h.metrics, class=c.h.l.NonSensitiveLogger, class=a.a.ActorSystemImpl or class= threaddump or are using any of the previous class names as a freetext string.

LogScale events had a field nodeId, when logged to the internal humio repository. As we have introduced the new vhost tag nodeId has been removed. When searching for logs of #kind=requests the freetext string "logged request" was often used -t hat will not work anymore. Logs of #kind=metrics had a field named type that has been renamed to metrics_type. That has been changed to These queries can now be written using the new #kind tag (for example #kind=metrics).