All Sources: How to Use Transforms
Use Transforms
Example Config
yaml
sources:
  demo:
    type: file
    include: /home/me/demo_log_file.log
    sink: logscale
    transforms:
      # regex_filter transform can filter events by either including or excluding events matching a regex pattern.
      - type: regex_filter
        mode: include
        pattern: ^category=(error|warning|info|debug)

      # static_fields transform adds configured key, value pairs as fields
      - type: static_fields
        fields:
          mykey: myvalue
          #fields to add must be indented 2 spaces under the fields: label
          # Passing environment variables is supported
          myenvvar: $MY_ENV_VAR

      # regex_redact transform replaces sensitive data matching a pattern
      # with a repeated character. See below for full examples.

You can use multiple events on one single line (recommended) or on multiple lines:

Single Line Transforms (AND or OR can be used in the regex statement) 

transforms:
  - type: regex_filter
    mode: exclude
    pattern: (some pattern|another pattern)

Multi-Line Transforms (the statements are treat as AND) 

transforms:
  - type: regex_filter
    mode: exclude
    pattern: some pattern

  - type: regex_filter
    mode: exclude
    pattern: another pattern
Common Filtering Patterns

The following examples demonstrate common use cases for filtering data across different source types.

Exclude Health Check and Monitoring Traffic

Filter out routine health checks, readiness probes, and monitoring requests that create noise in logs:

yaml
sources:
  app_logs:
    type: file
    include: /var/log/application/*.log
    sink: logscale
    transforms:
      - type: regex_filter
        mode: exclude
        pattern: '^.*(healthcheck|kube-probe|readiness|liveness).*$'

Include Only Critical Log Levels

Ingest only error, warning, and critical events to reduce volume and focus on important issues:

yaml
sources:
  filtered_logs:
    type: file
    include: /var/log/app.log
    sink: logscale
    transforms:
      - type: regex_filter
        mode: include
        pattern: '^category=(error|warning|critical)$'

Filter Syslog by Application Name

Use the optional field parameter to filter on specific parsed fields. This example excludes noisy applications from syslog:

yaml
sources:
  syslog_filtered:
    type: syslog
    mode: tcp
    port: 1514
    sink: logscale
    parser: syslog_rfc5424
    transforms:
      - type: regex_filter
        mode: exclude
        field: log.syslog.appname
        pattern: '^(Vpxa|envoy-access|sandboxd|crond|snmpd|heartbeat)$'

Filter Windows Event Logs by Event ID

Exclude verbose or routine Windows events while preserving important security and system events:

yaml
sources:
  windows_security:
    type: wineventlog
    sink: logscale
    channels:
      - name: Security
    transforms:
      - type: regex_filter
        mode: exclude
        pattern: 'EventID=(4662|4688|5156)'
      - type: static_fields
        fields:
          host_role: domain_controller

Combine Multiple Filters with Static Fields

Chain multiple transforms to filter events and add contextual metadata:

yaml
sources:
  production_api:
    type: file
    include: /var/log/api/*.log
    sink: logscale
    transforms:
      # Exclude health checks
      - type: regex_filter
        mode: exclude
        pattern: 'GET /health'
      # Exclude debug logs
      - type: regex_filter
        mode: exclude
        pattern: 'level=debug'
      # Add environment context
      - type: static_fields
        fields:
          environment: production
          service: api-gateway

Note

The field parameter for regex_filter is optional. When omitted, the pattern matches against the entire raw event. When specified, the pattern matches only the value of the named field after parsing.

regex_redact Transform

Replaces text matching a configured pattern with a repeated single character. Use this to prevent sensitive data from leaving the host while keeping logs operationally useful.

Key behaviors:

  • Masks every match on an event

  • Preserves the event's original length and structure (existing parsers and dashboards are unaffected)

  • Without a capture group, masks the entire match

  • With capture groups, masks only the captured text; the rest of the match remains visible

  • Multiple capture groups are supported; each is masked individually

Example with no capture group (entire match is masked):

yaml
transforms:
  - type: regex_redact
    pattern: 'mrn=\S+'
before: admit mrn=MR0092831 ward=4B status=stable
after:  admit ************* ward=4B status=stable

Example with capture groups (only captured values are masked):

yaml
transforms:
  - type: regex_redact
    pattern: 'dob=(\S+) mrn=(\S+)'
before: patient dob=1974-03-11 mrn=MR0092831 ward=4B
after:  patient dob=********** mrn=********* ward=4B

Note

Configuring regex_redact via Fleet Management requires LogScale 1.261 or later. Local configuration files support it immediately.

Table: Transforms

ParameterTypeRequiredDefault ValueDescription
typetransformtyperequired   Must be set to "regex_filter", "static_fields", or "regex_redact".
   Values
    regex_filter
    regex_redact
    static_fields

Table: Static Fields

ParameterTypeRequiredDefault ValueDescription
fieldsstringrequired   An object containing field-value pairs.
typeconst:static_fieldsrequired   Must be set to "static_fields".

Table: Regex-filter

ParameterTypeRequiredDefault ValueDescription
fieldstringoptional[a]   The field to apply the regex pattern to.
moderegexfiltermoderequired   Determines whether to include or exclude events that match the pattern. Must be set to "include" or "exclude".
   Values
    exclude
    include
patternstringrequired   The regex pattern to match.
typeconst:regex_filterrequired   Must be set to "regex_filter".

[a] Optional parameters use their default value unless explicitly set.


Table: Regex-redact

ParameterTypeRequiredDefault ValueDescription
patternstringrequired   The regex pattern to match. Supports capture groups.
typeconst:regex_redactrequired   Must be set to "regex_redact".