Sources (sources)

The sources block configures the sources of data that the LogScale Collector will send to LogScale.

The MySourceName is a top level element which contains each of your source configurations.

Table: Sources

ParameterTypeRequiredDefault ValueDescription
MySourceNamestringoptional[a]  MySourceName specify a name for each source you are collecting, you can have one or more sources and specify a sink out of those specified in the sinks section for each source.

[a] Optional parameters use their default value unless explicitly set.


The elements listed in this table apply to all source types, for information on specific sources see the relative sections in Sources & Examples

Table: MySourceName

ParameterTypeRequiredDefault ValueDescription
parserstringoptional[a]  Specify the parser within LogScale to use to parse the logs, if you install the parser through a package you must specify the type and name as displayed on the parsers page for example linux/system-logs:linux-filebeat. If a parser has been assigned to the ingest token you are using this will be ignored.
sinkstringoptional[a]  Name of the configured sink that which will receive the collected events.
transformsstringoptional[a]  The transforms parameter allows you to add fields or to filter data before it is ingested. See How to Use Transforms for more information.
typestringoptional[a]  The sources block configures the sources of data that the log collector will send to LogScale.
   Valid Values
   cmdCommand-line log source, for more information see Sources & Examples.
   fileFile based log source, for more information see Sources & Examples.
   journalJournal source, for more information see Sources & Examples.
   syslogSyslog event source, for more information see Sources & Examples.
   unifiedlogmacOS log source, for more information see Sources & Examples.
   wineventlogWindows Event log source, for more information see Sources & Examples.

[a] Optional parameters use their default value unless explicitly set.


How to Use Transforms
Example Config
yaml
sources:
  demo:
    type: file
    include: /home/me/demo_log_file.log
    sink: logscale
    transforms:
      # regex_filter transform can filter events by either including or excluding events matching a regex pattern.
      - type: regex_filter
        mode: include
        pattern: ^category=(error|warning|info|debug)

      # static_fields transform adds configured key, value pairs as fields
      - type: static_fields
        fields:
          mykey: myvalue
          # Passing environment variables is supported
          myenvvar: $MY_ENV_VAR

You can use multiple events on one single line (recommended) or on multiple lines: 

transforms:
  - type: regex_filter
    mode: exclude
    pattern: (some pattern|another pattern)
transforms:
  - type: regex_filter
    mode: exclude
    pattern: some pattern

  - type: regex_filter
    mode: exclude
    pattern: another pattern

Table: Transforms

ParameterTypeRequiredDefault ValueDescription
typestringoptional[a]  If static_field is specified you must specify a key and a value which can be an environment variable for example myenvvar:$MY_ENV_VAR. If you specify regex_filter you must specify a mode and the regular expression to use in pattern.
   Valid Values
   kubernetesAdds kubernetes pod metadata, see Collect Kubernetes Pod Logs for more information.
   regex_filterThe regex_filter allows you to filter which events to include or exclude using the mode, before data is ingested based on a pattern see, the regex-filter table for details.
   static_fieldsAdd a field to data as it is ingested.

[a] Optional parameters use their default value unless explicitly set.


Table: Static Fields

ParameterTypeRequiredDefault ValueDescription
fieldsstringoptional[a]  Adds a field to data as it is ingested. If static_field is specified you must specify a key and a value which can be an environment variable for example myenvvar:$MY_ENV_VAR
   Valid Values
   A key and value.

[a] Optional parameters use their default value unless explicitly set.


Table: Regex-filter

ParameterTypeRequiredDefault ValueDescription
modestringoptional[a]  If you specify regex_filter you must specify a mode, which can be include or exclude, and the regular expression to use in pattern.
   Valid Values
   include or exclude
patternstringoptional[a]  The regular expression to include or exclude.
   Valid Values
   a regular expression

[a] Optional parameters use their default value unless explicitly set.


Table: Kubernetes

ParameterTypeRequiredDefault ValueDescription
containersstringoptional[a]  See Collect Kubernetes Pod Logs for more information.

[a] Optional parameters use their default value unless explicitly set.