LogScale Internal Logging

LogScale sends its own internal logs to the humio repository. LogScale logs are also written to files by default. You can configure logging to standard.out, as well. This guide describes these logging options. For specific information about logging LogScale to LogScale, see Log LogScale to LogScale.

LogScale internal logs are divided into five kinds: logs, activity, metrics, requests, and nonsensitive. These logs use a tag, #kind, and each #kind is logged to its own file. Querying using the tag improves speed for many searches.

LogScale logging types are listed below with the names of their respective log files in parentheses:

  • logs (humio-debug.log) โ€” humio debug logs.

  • activity (humio-activity.log) โ€” humio-activity that is relevant to users.

  • metrics (humio-metrics.log) โ€” humio-metrics.

  • requests (humio-requests.log) โ€” All HTTP requests. Like an accesslog in LogScale's own format.

  • nonsensitive (humio-non-sensitive.log) โ€” Selected log lines where no searches or user data will be present. This can be shipped to LogScale support or other parties.

  • threaddumps (humio-threaddumps.log) โ€” LogScale regularly logs threaddumps.

LogScale automatically rotates each log when it reaches 50 megabytes in size. LogScale retains up to five files of each log.

All of the logs in the preceding list are available for search in the humio repository. When searching LogScale logs, you can use these tags: #type, #kind, and #vhost. All logs have #type=humio. They have a #kind tag for each type.

Log events will also have a vhost tag. Each node in a LogScale cluster has a node number. The #vhost value indicates which node in the LogScale cluster wrote the log event. Below are a few standard searches using the above tags:

logscale
#type=humio  //will find all events from all hosts of all kinds
#type=humio #kind=metrics //search all metrics across all hosts in the cluster
#type=humio #kind=metrics #vhost=1 //find all metrics for the node number one

Observe and monitor LogScale with Insights

LogScale comes with a built in package, humio/insights. It's present in the humio repository and can help you to analyze logs in the humio- system repositories. The package provides dashboards and saved searches. These help you monitor and observe the LogScale cluster. For more information about the package and its dashboards, see Insights Package.

If a LogScale cluster is having problems, the dashboards in the humio/insights package are the best place to start troubleshooting. They serve as examples for building and customizing your own dashboards. The package is also available in the LogScale Marketplace and is continuously updated. It can be installed in any given repository from the Marketplace.

Important

Be aware that the package is written for self-hosted LogScale instances, and not all assets will apply to a cloud organization's logs.

Ship LogScale logs to another cluster

When running a LogScale cluster in production, you should ship logs to another LogScale cluster.

For information about how to do this, see Send LogScale logs to LogScale Cloud.

Configure LogScale logging

LogScale sends its logs to the internal LogScale repository and to files on the file system. You can log to standard.out instead. For more information, see HUMIO_LOG4J_CONFIGURATION. This is often preferred in Docker and Kubernetes environments.

If the standard configurations do not meet your requirements, you can set custom output using:

ini
HUMIO_LOG4J_CONFIGURATION=file:///path/to/your/log4j2-custom-config.xml
Using a Custom Log Configuration on Bare-Metal Installation

Use the following procedure to create and use a custom configuration file in bare-metal installation.

  1. Extract the tar file as described in LogScale Software Installation.

  2. Open a terminal in /opt/humio/humio.x.y.z. Replace x.y.z with your installed LogScale version.

  3. Run the following commands to unzip the jar and create a folder. This procedure varies for Docker and Kubernetes:

    bash
    $ cd libs
$ unzip context.jar log4j2.xml
$ mkdir ../../my-humio-log4j-config
$ mv log4j2.xml ../../my-humio-log4j-config/log4j2.xml

  4. Edit the file as required and add HUMIO_LOG4J_CONFIGURATION=file:///opt/humio/my-humio-log4j-config/log4j2.xml to your Configuration Variables.

Use log4j2.xml to increase rollover rotations

Increasing the amount of rotations between each rollover is useful for support purposes. Logs that roll over too quickly make it difficult to find information. You might need to share logs continuously.

This example configuration keeps more log files, such as humio-debug.log, per rotation.

  1. Unzip and open your log4j2.xml file and find this section:

    xml
    <Appenders>
   
   <RollingFile name="humio-file"
   filename="${humio.debuglog.dir}/humio-debug.log"

  2. Under that section for the humio-debug.log file, modify:

    xml
    <DefaultRolloverStrategy max="5" fileIndex="min"/>

    from 5 to 10. This change keeps the log files for 10 rotations instead of 5.

    xml
    <DefaultRolloverStrategy max="10" fileIndex="min"/>

  3. After you complete your modifications, restart LogScale for the changes to take effect.