Sources (sources)

The Sources configuration block in Log Collector allows you to define and manage data sources that will be sent to LogScale, with options to specify parsers, sinks, and transformations for each source. The configuration supports multiple source types including files, command-line, journal, syslog, macOS unified logs, and Windows Event logs, while also providing capabilities for filtering data and adding custom fields through transform operations.

The sources block configures the sources of data that the Log Collector will send to LogScale.

The MySourceName is a top level element which contains each of your source configurations.

Table: Sources

ParameterTypeRequiredDefault ValueDescription
MySourceNamestringoptional[a]   MySourceName specify a name for each source you are collecting, you can have one or more sources and specify a sink out of those specified in the sinks section for each source.

[a] Optional parameters use their default value unless explicitly set.


The elements listed in this table apply to all source types, for information on specific sources see the relative sections in Sources & Examples

Table: MySourceName

ParameterTypeRequiredDefault ValueDescription
parserstringoptional[a]   Specify the parser within LogScale to use to parse the logs, if you install the parser through a package you must specify the type and name as displayed on the parsers page for example linux/system-logs:linux-filebeat. If a parser has been assigned to the ingest token you are using this will be ignored.
sinkstringoptional[a]   Name of the configured sink that which will receive the collected events.
transformsstringoptional[a]   The transforms parameter allows you to add fields or to filter data before it is ingested. See How to Use Transforms for more information.
typestringoptional[a]   The sources block configures the sources of data that the log collector will send to LogScale.
   Values
   cmdCommand-line log source, for more information see Sources & Examples.
   fileFile based log source, for more information see Sources & Examples.
   journalJournal source, for more information see Sources & Examples.
   syslogSyslog event source, for more information see Sources & Examples.
   unifiedlogmacOS log source, for more information see Sources & Examples.
   wineventlogWindows Event log source, for more information see Sources & Examples.

[a] Optional parameters use their default value unless explicitly set.


How to Use Transforms
Example Config
yaml
sources:
  demo:
    type: file
    include: /home/me/demo_log_file.log
    sink: logscale
    transforms:
      # regex_filter transform can filter events by either including or excluding events matching a regex pattern.
      - type: regex_filter
        mode: include
        pattern: ^category=(error|warning|info|debug)

      # static_fields transform adds configured key, value pairs as fields
      - type: static_fields
        fields:
          mykey: myvalue
          #fields to add must be indented 2 spaces under the fields: label
          # Passing environment variables is supported
          myenvvar: $MY_ENV_VAR

You can use multiple events on one single line (recommended) or on multiple lines:

Single Line Transforms( AND or OR can be used in the regex statement) 

transforms:
  - type: regex_filter
    mode: exclude
    pattern: (some pattern|another pattern)

Multi-Line Transforms (the statements are treat as AND) 

transforms:
  - type: regex_filter
    mode: exclude
    pattern: some pattern

  - type: regex_filter
    mode: exclude
    pattern: another pattern

Table: Static Fields

ParameterTypeRequiredDefault ValueDescription
fieldsstringoptional[a]   Adds a field to data as it is ingested. If static_field is specified you must specify a key and a value which can be an environment variable for example myenvvar:$MY_ENV_VAR
   Values
   A key and value.

[a] Optional parameters use their default value unless explicitly set.


The regex_filter transform uses RE2 Regex syntax. While powerful, this engine has some limitations that users should be aware of:

  • No Look-around Assertions:

    The engine does not support look-ahead or look-behind assertions ((?=), (?!), (?<=), (?<!)).

  • No Backreferences:

    Backreferences (\1, \2, etc.) are not supported in the regex patterns.

  • No Atomic Grouping:

    The engine does not support atomic grouping (?>...).

  • No Recursive Patterns:

    The engine does not support recursive Regex patterns.

Table: Kubernetes

ParameterTypeRequiredDefault ValueDescription
containersstringoptional[a]   See Collect Kubernetes Pod Logs for more information.

[a] Optional parameters use their default value unless explicitly set.