Logscale Collector Configuration Elements

The LogScale Collector is configured via .yaml file either manually on a local file or in centralized way via the Config Editor.

The file is nested and the indention of the file is essential to the correct function of the LogScale collector. The first level of the file is as follows, however the elements fleetManagement and datadirectory are not to be used in the case of remote configuration.

Element	Description	Contents
`flags`	Optional configuration flags which allow certain additional behaviors like communication over HTTP.	See Optional Flags(`flags`).
`fleetManagement`	The set of details required of the instance to work with fleet management, see LogScale Collector Fleet Managementfor more on fleet management. Note that this section is not applicable if you are using remote configuration management.	See Fleet Management (`fleetManagement`).
`dataDirectory`	Defines the where the LogScale Collector will create its "database", e.g. checkpoints.json file.	The name of the file or path to the folder.
`sources`	This element allows you to define one or more data sources and a configuration for each data source including a sink for each source.	See Sources and Examples for more information and examples for different source types.
`sinks`	Defines where the data will be sent and specifications on the Queue (`queue`), memory, compression, proxy configuration	See Sinks (`sinks`).

Data Directory `(dataDirectory)`

The top level element defines the where the LogScale Collector will create its "database", e.g. checkpoints.json file.

yaml

dataDirectory: data
flags:
  ...
sources:
  apache_logs:
    ...

sinks:
  my_humio_instance:
    ...

Table: dataDirectory table

Parameter	Type	Required	Default	Description
`data-directory`	string			Data directory is a top level element which is required to define where the LogScale Collector will create its "database", e.g. checkpoints.json file. Note this is not required for remote configurations.

Optional Flags(`flags`)

This set of flags allows you to enable and disable a set of configurations which are either optional or only for testing purposes.

yaml

flags:
  # enable to allow non-https addresses in fleet management and sinks. defaults to false 
  allowInsecureHTTP: false
  # enable to allow the use of 'cmd' sources when using remote configuration.
  allowRemoteCMD: false
sources:
...
sinks:
...

Table: flags

Parameter	Type	Required	Default	Description
`allowInsecureHTTP`	string			Set to false by default, is true allows non-https addresses in fleet management and sinks.
`allowRemoteCMD`	string			Set to false by default, if true allows the use of CMD sources when using remote configuration

Fleet Management (`fleetManagement`)

The fleet management block configures instances of the log collector to work on the LogScale Collector Fleet Management, You can retrieve this token on the LogScale Collector Fleet Management Overview this section is not required in the case of Managing Remote Configurations.

yaml

fleetManagement:
         token: 4b09c4f7-2364-605t-a55f-d5d2fg881d66
         url: https://cloud.us.humio.com

Table: fleetManagement

Parameter	Type	Required	Default	Description
`token`	string			This key specifies the token which instances of the log collector to be visualized on the LogScale Collector Fleet Management page.
`URL`	string			URL of the humio installation where the fleet management page is hosted.

Note

Proxy must be set to none except for Linux use cases.

Sources (`sources`)

The sources block configures the sources of data that the log collector will send to LogScale.

The MySourceName is a top level element which contains each of your source configurations.

Table: Sources

Parameter	Type	Required	Default	Description
`MySourceName`	string			`MySourceName` specify a name for each source you are collecting, you can have one or more sources and specify a sink out of those specified in the `sinks` section for each source.

The elements listed in this table apply to all source types, for information on specific sources see the relative sections in Sources and Examples

Table: MySourceName

Parameter	Type	Required	Default	Description
`parser`	string			Specify the parser within LogScale to use to parse the logs, if you install the parser through a package you must specify the type and name as displayed on the parsers page for example linux/system-logs:linux-filebeat.
`sink`	string			Name of the configured sink that which will receive the collected events.
`type`	string			The sources block configures the sources of data that the log collector will send to Humio.
		Valid Values	`cmd`	Command-line log source, for more information see Sources and Examples.
			`file`	File based log source, for more information see Sources and Examples.
			`journal`	Journal source, for more information see Sources and Examples.
			`syslog`	Syslog event source, for more information see Sources and Examples.
			`wineventlog`	Windows Event log source, for more information see Sources and Examples.

Sinks (`sinks`)

The sinks block configures the sinks (where the data will be sent) that are used by the source or sources.

yaml

sinks:
  my_other_humio_instance:
    type: humio
    token: <ingest-token_repo1>
    url: https://cloud.us.humio.com
  my_humio_instance:
    type: humio
    token: <ingest-token-repo2> or an environment variable
    url: https://cloud.us.humio.com
  # maxEventSize (default 1MB) sets the limit for a single event in bytes, if exceeded the event will be truncated.
  maxEventSize: 1048576

  # maxBatchSize (default: 16 MB), sets the maximum size in bytes of a batch which is sent to the configured sink.
  # This includes fields as well as event data. If exceeded data will be sent in a subsequent batch.
  maxBatchSize: 16777216

    # auto, none, gzip, deflate, none. Default: auto
    compression: gzip

    # Number between: 1 ... 9.
    #   1 = highest speed
    #   9 = highest compression
    # If unspecified or 0 the default value for the compression algorithm specified in compression is used
    compressionLevel: 9

    # Override default tls configuration
    # Only one of the following options should be used at a time.
    # If multiple are given, the precedence is: 'insecure', 'caCert', 'caFile'.
    tls:
      # Specify insecure to skip certificate validation
      insecure: false

      # Specify caCert to load a PEM certificate from the config file
      caCert: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----

      # Specify caFile to load PEM certificate from an external file.
      caFile: /etc/ssl/cert.pem
    # Override proxy configuration for the sink. Must be set to 'none' for Windows Server and fleet management.
    # Accepted values: 'system', 'none' or a URL such as: http://127.0.0.1:3129 for an http proxy.
    # Defaults to system, which will try to determine the appropriate proxy or fallback to none.
    proxy: none

The MySinksName is a top level element which contains each of your sink configurations.

Table: Sinks

Parameter	Type	Required	Default	Description
`My Sink Name/s`	string			The user defined name for each sink configuration. This name will be referenced in your `sources`.

The elements listed in this table define how each sink is configured.

Table: MySinkName

Parameter	Type	Required	Default	Description
`compression`	string			Specify the type of data compression, possible values: auto, none, gzip, deflate. The default value is auto.
`compressionLevel`	integer			Specify the level of compression where 1 is best speed and 9 is best compression, if set to undefined or 0 the default value for the compression algorithm specified in compression is applied.
`maxBatchSize`	integer			Specifies the max size of batch (default 16MB) and works along with the maximum events per request. The limits are propagated to the queue and replace the `maxEventsPerRequest`. The limits are also propagated to all the sources that reference the sink.
`maxEventsPerRequest`	integer			Specify the max number of events per request by size (Default 1MB) and works with `maxBatchSize`.
`proxy`	string			Set to `none` for Windows Server or you can specify, if required, an override proxy configuration for the sink, possible values: `system`, `none` or a URL such as: http://127.0.0.1:3129 for a http proxy. The default is system, which will try to determine the appropriate proxy or fallback to none.
`tls`	string			This object contains details on the PEM certificates. this section allows you to override the defaults. Only one of the following options should be specified:
		Valid Values	`caCert`	Specify this key to load a certificate from the config file.
			`caFile`	Specify this key to load the PEM certificate from an external file.
			`insecure`	Specify if certificate validation is needed, if set to true the certificate validation is skipped.
`token`	string			Specify the Ingest Tokens for your LogScale repository or an environment variable.
`type`	string			Specify the type of sink. This must be set to humio.
`url`	string			Specify the url of your LogScale account for example https://cloud.humio.com.

Queue (`queue`)

The queue block is part of the Sinks (sinks) and configures the behaviour of the queue.

Note

The memory queue no longer supports configuration of maxEventsPerRequest, it inherits the maximum bytes per request from the sink maxBatchSize.

yaml

queue:
  # Default: 1024

  # fullAction determines queue behavior when it is full.
  #   pause = queue pauses ingesting new batches if it is full (Default if not mentioned) deleteLatest is no longer support and automatically set to pause.
  #   deleteOldest = queue deletes the oldest batch to accept new batches if it is full
  # Default: pause
  fullAction: deleteOldest

  memory:
    # Default: 1000
    flushTimeOutInMillisecond: 200
    # Default: 2048
    maxLimitInMB: 1024

Table: Queue

Parameter	Type	Default	Description
`disk`	integer	`1024`	when disk is specified the data is written in the `dataDirectory/queue/sinkName/` unless specified using `storageDir`. The `maxLimitInMB` must be set to the maximum size of the queue when set to disk, by default set to 1024.
`fullAction`	string	`pause`	Specify the action to take when the queue is full. The possible values are: `deleteOldest` accepts new batches but deletes the oldest batch and whichever is reached first triggers the data to be sent to Humio. `pause` this is the default value. The queue does not ingest new batches when it is full. Note that deleteLatest is no longer supported and automatically set to pause.
`memory`	integer	`1024`	`memory` default; the `maxLimitInMB` parameter can be set to the required value and is set to 1024mb by default.

Table: Memory

Parameter	Type	Required	Default	Description
`flushTimeOutInMillisecond`	integer		`1000`	Specify how often data is sent to LogScale Collector.
`maxLimitInMB`	integer		`1024mb`	The maximum size in MB.

Falcon LogScale Documentation

Documentation

Reference Information

Knowledge Base

LogScale Notices

Release Notes

LogScale Training