LogScale Internal Logging
LogScale has a repository where LogScale internal logs are sent. LogScale logs are by default also written to files. It's possible to configure logging to standard.out, as well. This is described on this page.
LogScale logging is divided into six types. These types are listed below, with the names of their respective logs file in parentheses:
humio-debug.log) — Internal debug logs;
humio-activity.log) — Logs that are relevant to users;
humio-metrics.log) — Metric Types;
humio-requests.log) — All HTTP requests. Like an accesslog in LogScale own format;
humio-non-sensitive.log) — Selected log lines where no searches or user data will be present. This can be shipped to LogScale support or other parties; and
humio-threaddumps.log) — LogScale regular logs threaddumps
The above logs are automatically rotated by LogScale when it reaches 50 megabytes in size. LogScale will retain up to five files of each.
All of the above logs are available for search in LogScale internal repository. When searching LogScale logs in the humio repository, the tag #type #kind and #vhost can be used. All the logs will have #type=humio. They will have a #kind tag for each in the list above.
Log events will also have a vhost tag. Each node in a LogScale cluster has a node number. The #vhost value indicates which node in the LogScale cluster wrote the log event. Below is a couple of standard searches using the above tags:
#type=humio //will find all events from all hosts of all kinds
#type=humio #kind=metrics //search all metrics across all hosts in the cluster
#type=humio #kind=metrics #vhost=1 //find all metrics for the node number one
Observe & Monitor LogScale with Insights
LogScale comes with a built in application named, humio/insights. It's present in the humio repository. The application is a collection of dashboards and saved searches making it possible to monitor and observe the LogScale cluster.
If a LogScale cluster is having problems, refer to these dashboards. The application also serves as good examples on which to build. The application is also available in the LogScale Marketplace and is continuously updated. It can be installed in any given repository from the Marketplace.
Shipping LogScale Logs to another Cluster
When running a LogScale cluster in production, we highly recommend shipping the logs to another LogScale cluster. To do this, install the humio/insights application in the cluster to get out-of-the-box insights.
If a cluster is having problems, it will often not be possible to do searches and debug it. As a last resort, you could grep through files on multiple machines.
It is possible to setup an agent to collect LogScale log files and ship them to another LogScale cluster. Read the Log LogScale to LogScale for more information on this.
It is also possible to send the logs to LogScale cloud service. This is convenient in that you won't have to run and maintain another cluster. It also makes it possible to share the logs with LogScale support.
Please note that LogScale cannot guarantee what data is in the
humio-debug.log file. We strive not to log any data
ingested in LogScale. Search strings are logged to the debug log.