Falcon LogScale Collector 1.2.0 GA (2022-10-27)

Version?Type?Release Date?Config.Changes?
1.2.0GA2022-10-27Yes

Hide file hashes

Show file hashes

This version of the humio log collector offers the Fleet Overview functionality, which allows you to monitor the status of log collector instances and the following improvements:

  • Improved configuration file validation

  • Improved error logging

  • Reload configuration file feature

  • Using environment variables as the sink url

  • The file source now has more include and exclude patterns and uses less resources by waiting for changes to the file

  • the CMD source can now create single multiline events

  • the wineventlog can now filter events by provider and keep bookmarks of its progress

  • Performance improvements

  • improved batch handling

  • Enforces the use of HTTPS.

Improvements, new features and functionality

  • Collecting Data

    • The url option in the sinks part of the configuration can now refer to an environment variable by using the ${ENVVAR} syntax.

    • Improved batch handling

      • The sinks now have additional configuration options to change the maximum event size maxEventSize (default 1MB) and the maximum batch size maxBatchSize (default: 16 MB). The limits are propagated to the queue, where it replaces the previous maxEventsPerRequest option. The limits are also propagated to all the sources that reference the sink.

      • The memory queue no longer supports configuration of maxEventsPerRequest, it inherits the maximum bytes per request from the sink maxBatchSize.

      • The memory queue no longer waits before flushing a batch that is larger than the maximum batch size.

      • The collector now warns you when a memory queue reaches 50% and 80% of capacity.

      • The collector now sends a warning after 2 retry attempts when sending events to a http sink.

    • The file source can now have additional include and exclude patterns in the same configuration. Specifically, the options exclude and include can be either a string or an array of strings.

    • The cmd source can now create a single multiline event when running in the schedule mode. Set the option consolidateOutput to true to enable this feature.

    • The wineventlog source can now filter events based on the provider name. Set the option providers to an array of provider names that should be included to enable this feature. This source also keeps a bookmark of its progress in the Windows event log, and resumes from there when the collector is restarted.

  • Configuration

    • The collector now reloads the configuration file when it receives a SIGHUP. This does not apply to the logLevel and dataDirectory options. If the new configuration is invalid, the program will stop.

    • Improved configuration file validation - The collector is now more thorough when validating its configuration file. An example of this is that unknown options in the configuration are invalid and will prevent running the program. Upon detection of an invalid configuration, the collecter will attempt to provide a descriptive error, some examples of this are: 

      error reading config file "my_config.yaml" sources:
        name must consist of only alphanumeric characters or '.', '_'
        and '-'

            error reading config file "my_config.yaml": 
            sources.cmd_uname_scheduled.interval: invalid type string,
            wanted int` 

            error reading config file "my_config.yaml"
  : sources.dummy_logs.sink: missing value for required field`

    • The collector now enforces using https:// for URLs, this can be overridden by adding the -allow-insecure-http command line flag.

  • Debugging

    • The default log level is now set to warnings, previously only errors were logged by default.

  • Managing Data

    • Improved serialization performance in the humiosink leading to lower memory usage and faster serialization of events.

    • Improved memory usage of the memory queue component by removing an upfront buffer that caused it to store more events than specified by the maxLimitInMB option.

    • If a file monitored by the file source is inactive not written for a configurable period default: 60 seconds, the file descriptor is closed to release system resources, and watched for changes instead. Whenever the file changes, it is re-opened. This is configurable by the inactivityTimeout option in the file source.

  • Fleet Management

    • The collector now supports reporting to the fleet overview of LogScale. Configure the fleetManagement part of the configuration to enable this feature, see Fleet Management (fleetManagement) for more information.

      When the feature is enabled, the collector will periodically send metrics to LogScale, including the OS version, the collector version, how much data is ingested, and a description of the configured log sources.

Bug Fixes

  • Collecting Data

    • Events from the wineventlog source which contain fields of the type hexadecimal integer were presented as a base 10 number, they are now presented as a base 16 number.