Sources (sources)

Sources (`sources`)

The sources block configures the sources of data that the Log Collector will send to LogScale.

The MySourceName is a top level element which contains each of your source configurations.

Table: Sources

Parameter	Type	Required	Default Value	Description
`MySourceName`	string	optional^[a]		`MySourceName` specify a name for each source you are collecting, you can have one or more sources and specify a sink out of those specified in the `sinks` section for each source.
^[a]Optional parameters use their default value unless explicitly set.

The elements listed in this table apply to all source types, for information on specific sources see the relative sections in Sources & Examples

Table: MySourceName

Parameter	Type	Required	Default Value	Description
`parser`	string	optional^[a]		Specify the parser within LogScale to use to parse the logs, if you install the parser through a package you must specify the type and name as displayed on the parsers page for example linux/system-logs:linux-filebeat. If a parser has been assigned to the ingest token you are using this will be ignored.
`sink`	string	optional^[a]		Name of the configured sink that which will receive the collected events.
`transforms`	string	optional^[a]		The transforms parameter allows you to add fields or to filter data before it is ingested. See How to Use Transforms for more information.
`type`	string	optional^[a]		The sources block configures the sources of data that the log collector will send to LogScale.
			Valid Values
			`cmd`	Command-line log source, for more information see Sources & Examples.
			`file`	File based log source, for more information see Sources & Examples.
			`journal`	Journal source, for more information see Sources & Examples.
			`syslog`	Syslog event source, for more information see Sources & Examples.
			`unifiedlog`	macOS log source, for more information see Sources & Examples.
			`wineventlog`	Windows Event log source, for more information see Sources & Examples.
^[a]Optional parameters use their default value unless explicitly set.

How to Use Transforms

Example Config

yaml

sources:
  demo:
    type: file
    include: /home/me/demo_log_file.log
    sink: logscale
    transforms:
      # regex_filter transform can filter events by either including or excluding events matching a regex pattern.
      - type: regex_filter
        mode: include
        pattern: ^category=(error|warning|info|debug)

      # static_fields transform adds configured key, value pairs as fields
      - type: static_fields
        fields:
          mykey: myvalue
          #fields to add must be indented 2 spaces under the fields: label
          # Passing environment variables is supported
          myenvvar: $MY_ENV_VAR

You can use multiple events on one single line (recommended) or on multiple lines:

Single Line Transforms( AND or OR can be used in the regex statement)

transforms:
  - type: regex_filter
    mode: exclude
    pattern: (some pattern|another pattern)

Multi-Line Transforms (the statements are treat as AND)

transforms:
  - type: regex_filter
    mode: exclude
    pattern: some pattern

  - type: regex_filter
    mode: exclude
    pattern: another pattern

Table: Transforms

Parameter	Type	Required	Default Value	Description
`type`	string	optional^[a]		If `static_field` is specified you must specify a key and a value which can be an environment variable for example `myenvvar:$MY_ENV_VAR`. If you specify `regex_filter` you must specify a `mode` and the regular expression to use in `pattern`.
			Valid Values
			`kubernetes`	Adds kubernetes pod metadata, see Collect Kubernetes Pod Logs for more information.
			`regex_filter`	The `regex_filter` allows you to filter which events to include or exclude using the mode, before data is ingested based on a `pattern` see, the regex-filter table for details.
			`static_fields`	Add a field to data as it is ingested.
^[a]Optional parameters use their default value unless explicitly set.

Table: Static Fields

Parameter	Type	Required	Default Value	Description
`fields`	string	optional^[a]		Adds a field to data as it is ingested. If `static_field` is specified you must specify a key and a value which can be an environment variable for example `myenvvar:$MY_ENV_VAR`
			Valid Values
			`A key and value.`
^[a]Optional parameters use their default value unless explicitly set.

Table: Regex-filter

Parameter	Type	Required	Default Value	Description
`mode`	string	optional^[a]		If you specify `regex_filter` you must specify a `mode`, which can be include or exclude, and the regular expression to use in `pattern`.
			Valid Values
			`include or exclude`
`pattern`	string	optional^[a]		The regular expression to include or exclude.
			Valid Values
			`a regular expression`
^[a]Optional parameters use their default value unless explicitly set.

Table: Kubernetes

Parameter	Type	Required	Default Value	Description
`containers`	string	optional^[a]		See Collect Kubernetes Pod Logs for more information.
^[a]Optional parameters use their default value unless explicitly set.

How to Use Transforms

Enter search term

Sources (`sources`)