File Source with Transforms

Example
yaml
sources:
  demo:
   type: file
   include: /home/me/demo_log_file.log
   sink: logscale
  transforms:
   - type: regex_filter
    mode: include
    pattern: ^category=(error|warning|info|debug)
   - type: static_fields
    fields:
     mykey: myvalue
     myenvvar: $MY_ENV_VAR
   - type: regex_filter
    mode: exclude
    pattern: (some pattern|another pattern)
   # Alternative approach for multiple patterns:
   - type: regex_filter
    mode: exclude
    pattern: some pattern
   - type: regex_filter
    mode: exclude
    pattern: another pattern
Introduction

This configuration example demonstrates how to define a file source and apply different transforms to preprocess events before sending them to LogScale.

Step-by-Step
  1. yaml
    sources:
  demo:
   type: file
   include: /home/me/demo_log_file.log
   sink: logscale

    Here, it defines source file without transforms. We will add transforms in following fragments.

  2. yaml
    transforms:
   - type: regex_filter
    mode: include
    pattern: ^category=(error|warning|info|debug)

    Here, we defined regex_filter transform to include categories.

  3. yaml
    - type: static_fields
    fields:
     mykey: myvalue
     myenvvar: $MY_ENV_VAR

    Here, we defined static_fields transform to add key-value pairs.

  4. yaml
    - type: regex_filter
    mode: exclude
    pattern: (some pattern|another pattern)

    Here, we defined regex_filter transform using a single-line pattern (OR logic).

  5. yaml
    # Alternative approach for multiple patterns:
   - type: regex_filter
    mode: exclude
    pattern: some pattern
   - type: regex_filter
    mode: exclude
    pattern: another pattern

    Here, we defined regex_filter transform using multiple lines (AND logic).

  6. Event Result set.

Summary and Results

This example is divided into smaller fragments to highligh each component: base file source, static_fields transform, regex_filter in single and multiline patterns. This modular approach makes it easier to understand, reuse, and extend each configuration block for different use cases.