LogScale recommends shipping LogScale logs using the Falcon LogScale Collector. To install it, see the Falcon LogScale Collector documentation.

After you have it installed, you'll need to edit your configuration file to look like this:

sources:
  file:
    type: file
    sink: humio
    include:
      - $LOGSCALE_LOGS_DIR/*.log
    exclude:
      # humio-audit.log is included in humio-debug.log
      - $LOGSCALE_LOGS_DIR/humio-audit.log
    multiLineBeginsWith: ^[0-9]{4}-[0-9]{2}-[0-9]{2}

sinks:
  humio:
    type: humio
    token: $INGEST_TOKEN
    url: $YOUR_LOGSCALE_URL

In the above configuration you need to replace the following:

Vector is a lightweight agent that may be used to send logs. It has built-in support for shipping logs to LogScale via the humio_logs sink.

To use Vector, you'll have to install it on all LogScale nodes within the cluster you want to monitor. See the Vector documentation on how to install vector.

After you've finished installing it, edit the vector.toml configuration file to look like the following:

[sources.logs]
type = "file"
include = ["$LOGSCALE_LOGS_DIR/humio*.log"]

[sources.logs.multiline]
start_pattern = "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
mode = "halt_before"
condition_pattern = "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
timeout_ms = 2000

# Humio Sink: https://vector.dev/docs/reference/sinks/humio_logs/
[sinks.humio_cluster]
type = "humio_logs"
inputs = ["logs"]
compression = "gzip"
host = "$YOUR_LOGSCALE_URL"
token = "$INGEST_TOKEN"
encoding.codec = "text"

In the above configuration example, you'll need to replace the following placeholders:

Once you've made those changes to the configuration file, start Vector and then check the repository for LogScale internal logs.

You can also send LogScale internal logs via Filebeat. To do so, install Filebeat on all LogScale nodes within the cluster you are going to monitor.

After you've done so, edit your filebeat.yml configuration file to like the example below:

filebeat.inputs:
- paths:
  - $LOGSCALE_LOGS_DIR/humio-*.log
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

 queue.mem:
   events: 8000
   flush.min_events: 1000
   flush.timeout: 1s

 output:
   elasticsearch:
     hosts: ["$YOUR_LOGSCALE_URL"]
     password: "$INGEST_TOKEN"
     compression_level: 5
     bulk_max_size: 200

In the above configuration example, you'll need to replace the following placeholders:

Once you've made those changes to the configuration file, start Filebeat and then check the repository to see if LogScale internal logs are being received.

To assist in troubleshooting your self-hosted instance of LogScale, LogScale recommends you stream your LogScale logs into a LogScale Cloud instance. This allows LogScale Support to immediately investigate your cluster logs when you submit any issues to our helpdesk, expediting the investigation. If you did not set this up during onboarding, contact LogScale Support, and they will get an instance configured for you.

Support will setup a repository for you to ship to, and share an ingest token and URL for you to use when shipping logs. You will then need to configure a shipper following the instructions.

The $YOUR_LOGSCALE_URL value will depend on your region. Support will provide you the URL to use. It will generally be either https://cloud.humio.com:443/api/v1/ingest/elastic-bulk or https://cloud.us.humio.com/api/v1/ingest/elastic-bulk.

By default, Support sets a 30 day time limit on the repository to provide adequate amount of time for us to assist and troubleshoot. After 30 days, the data will be removed following the platform's removal process.

You can request a shorter retention period set on your data, if desired. This will limit the timeframe that Support can investigate any issues.

If you are not able to ship logs to LogScale Cloud continuously, Support will need to receive logs from your system to troubleshoot any issues you raise in a Support case.

To receive these logs, Support will setup a repository and provide you the necessary details to ship logs into the repository. Following the log shipper configuration instructions, set the file to your static file containing the excerpted logs.

If you are sending data that is more than 30 days old, make sure Support is aware so that the data will be retained on ingest.