File Source

Example
yaml
sinks:
   logscale_sink:
    type: logscale 
    url: "https://cloud.humio.com/" 
    token: "${LOGSCALE_TOKEN}"
queue:
 type: memory
 maxLimitInMB: 64
sources:
 apache_access_logs:
  type: file  
  include:
   - "/var/log/apache2/access.log"
  exclude:
   - "/var/log/apache2/access.log.1"
   - "/var/log/apache2/excluded_access.log"
  excludeExtensions:
   - "gz"
   - "zip"
  multiLineBeginsWith: '^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
  sink: logscale_sink
  encoding: UTF-8
  transforms:
   - type: static_fields
    fields:
     log_type: "apache_access"
     environment: "${ENV}"
Introduction

This is an example configuration file for a basic Falcon LogScale Collector file-based service.

Step-by-Step
  1. yaml
    sinks:
       logscale_sink:
        type: logscale 
        url: "https://cloud.humio.com/" 
        token: "${LOGSCALE_TOKEN}"

    Here we create a new sink named logscale_sink which sends data to a LogScale with the URL in url using the ingest token ${LOGSCALE_TOKEN}.

  2. yaml
    queue:
     type: memory
     maxLimitInMB: 64

    Here, the queue is configured as a memory queue with a limit of 64MB.

  3. yaml
    sources:
     apache_access_logs:
      type: file  
      include:
       - "/var/log/apache2/access.log"

    Here, we define a file-based source for Apache access logs. The configuration incudes the path to the log file using the include field.

  4. yaml
    exclude:
       - "/var/log/apache2/access.log.1"
       - "/var/log/apache2/excluded_access.log"
      excludeExtensions:
       - "gz"
       - "zip"

    This fragment defines how can we exclude apache log files and extensions.

  5. yaml
    multiLineBeginsWith: '^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'

    It enables multiline log parsing using a regular expression that detects lines starting with an IP address.

  6. yaml
    sink: logscale_sink
      encoding: UTF-8

    Here, we assign the logscale_sink for output and explicitly sets the file encoding to UTF-8.

  7. yaml
    transforms:
       - type: static_fields
        fields:
         log_type: "apache_access"
         environment: "${ENV}"

    Here, we add static metadata fields to each log event, including the log type and the deployment environment.

  8. Event Result set.

Summary and Results

This example demonstrates a complete configuration for collecting and processing Apache access logs using file-based source. It includes log collection setup, multiline parsing, output sink configuration, and felt enrichment using static values.