LogScale's Internal Logging
LogScale has a repository where LogScale's internal logs are sent. LogScale's logs are by default also written to files. It's possible to configure logging to standard.out, as well. This is described on this page.
LogScale's logging is divided into six types. These types are listed below, with the names of their respective logs file in parentheses:
humio-debug.log) — Internal debug logs;
humio-activity.log) — Logs that are relevant to users;
humio-metrics.log) — Metric Types;
humio-requests.log) — All HTTP requests. Like an accesslog in LogScale's own format;
humio-non-sensitive.log) — Selected log lines where no searches or user data will be present. This can be shipped to LogScale support or other parties; and
humio-threaddumps.log) — LogScale regular logs threaddumps
The above logs are automatically rotated by LogScale when it reaches 50 megabytes in size. LogScale will retain up to five files of each.
All of the above logs are available for search in LogScale's internal repository. When searching LogScale's logs in the humio repository, the tag #type #kind and #vhost can be used. All the logs will have #type=humio. They will have a #kind tag for each in the list above.
Log events will also have a vhost tag. Each node in a LogScale cluster has a node number. The #vhost value indicates which node in the LogScale cluster wrote the log event. Below is a couple of standard searches using the above tags:
#type=humio //will find all events from all hosts of all kinds #type=humio #kind=metrics //search all metrics across all hosts in the cluster #type=humio #kind=metrics #vhost=1 //find all metrics for the node number one
Observe & Monitor LogScale with Built-In App
LogScale comes with a built in application named, humio/insights. It's present in the humio repository. The application is a collection of dashboards and saved searches making it possible to monitor and observe the LogScale cluster.
If a LogScale cluster is having problems, refer to these dashboards. The application also serves as good examples on which to build. The application is also available in the LogScale Marketplace and is continuously updated. It can be installed in any given repository from the Marketplace.
Shipping LogScale Logs to another Cluster
When running a LogScale cluster in production, we highly recommend shipping the logs to another LogScale cluster. To do this, install the humio/insights application in the cluster to get out-of-the-box insights.
If a cluster is having problems, it will often not be possible to do searches and debug it. As a last resort, you could grep through files on multiple machines.
It is possible to setup an agent to collect LogScale log files and ship them to another LogScale cluster. Read the Log LogScale to LogScale for more information on this.
It is also possible to send the logs to LogScale's cloud service. This
is convenient in that you won't have to run and maintain another
cluster. It also makes it possible to share the logs with LogScale
support. Please note that LogScale cannot guarantee what data is in the
humio-debug.log file. We strive not to log any data
ingested in LogScale. Search strings are logged to the debug log.
Configuring LogScale Logging
As described above, LogScale sends its logs to the internal LogScale repository and to files on the file system. It is possible to log to standard out instead. This is often preferred in Docker and Kubernetes environments. It can be configured as shown below:
# HUMIO_LOG4J_CONFIGURATION=log4j2.xml # HUMIO_LOG4J_CONFIGURATION=log4j2-json-stdout.xml # HUMIO_LOG4J_CONFIGURATION=log4j2-stdout.xml HUMIO_LOG4J_CONFIGURATION=file:///path/to/your/log4j2-custom-config.xml
In this example, since only the last line is not commented out, we're suggesting you provide you own configuration file for log4j2 outside the jar file. Be aware that the file needs to be similar to the one within the jar as the internal logging into the humio repository depends on some of the configuration in that. You can extract the internal version as a starting point using:
$ unzip humio-assembly-0.1.jar log4j2.xm
Changes to LogScale Log Format in 1.19
We have made changes to how LogScale logs internally in version 1.19. We did this to better support the new Insights Package application and in general improve the logging. These changes affect events in LogScale's own internal repository humio. The changes also result in LogScale logging to more files in the file system than it did before.
LogScale's internal logging has been divided into five kinds. logs, activity, metrics, requests and nonsensitive. A tag #kind has been introduced for these logs in LogScale's internal repository. Querying using the new tag will make many searches much faster. LogScale will log each kind to its own file.
kind=logs -> humio-debug.log
kind=metrics -> humio-metrics.log
kind=requests -> humio-requests.log
kind=nonsensitive -> humio-nonsensitive.log
The log file humio-requests.log is new in this release. The file humio-debug.log has the same format as it has always had. The three other files have a new format. The format is the same for each of these files. The format consists of a timestamp, followed by the log kind, followed by the host number, followed by the log message.
This new format is also used when logging events of these kinds to the internal humio repository. The internal LogScale parser has been updated to handle these changes. The built-in dashboards in the LogScale repository has been deleted. Use the new Insights Package application instead.
A vhost tag has also been introduced. vhost represents the node number for a node in a LogScale cluster. Often a host field is present in LogScale events when shipped using agents like Vector and Filebeat etc. When LogScale logged to its internal repository it used the nodeId field. We want to align the above and make logs more self-contained by having vhost being part of the standard logline just like loglevel and class etc.
We have tried to keep the changes as small and compatible as possible, but we have made some changes that can break existing searches in the humio repository (or other repositories receiving LogScale logs). We made these changes as we think they are important in order to improve things moving forward. One of the benefits is the new Insights Package.
The above changes can have important impacts on the following:
If logs from the cluster is being shipped to another LogScale cluster, for example LogScale Cloud, you need to make sure you are shipping all the files. Also the parser in the receiving end should be updated. If the receiving cluster is using the built in LogScale parser to parse the logs, and has been updated to at least version 1.19 no changes are needed. We recommend using the LogScale parser in the humio/insights app, if you need to update the parser receiving the logs.
We have a guide for Log LogScale to LogScale.
Custom queries, alerts and dashboards in the humio repository searching for events of the kinds metrics, requests or nonsensitive may need to be modified. These searches are typically using class=c.h.metrics, class=c.h.l.NonSensitiveLogger, class=a.a.ActorSystemImpl or class= threaddump or are using any of the previous class names as a freetext string.
LogScale events had a field nodeId, when logged to the internal humio repository. As we have introduced the new vhost tag nodeId has been removed. When searching for logs of #kind=requests the freetext string "logged request” was often used -t hat will not work anymore. Logs of #kind=metrics had a field named type that has been renamed to metrics_type. That has been changed to These queries can now be written using the new #kind tag (for example #kind=metrics).