Key Concepts

Data Sources

The Falcon LogScale Collector supports multiple data sources for comprehensive log collection, including file-based events, Windows Events, Syslog streams, executable outputs, SystemD logs on Linux, and macOS Unified Logs. Each data source offers specialized capabilities, from handling compressed files and log rotation to providing secure transmission of syslog data and rich event parsing for Windows logs, enabling flexible and robust log collection across diverse environments.

Data sources are the data points from which the data is collected. Falcon LogScale Collector currently supports the following inputs or data sources:

Collecting Events from Files

Collecting events from local files on disk is one of the most common log collection scenarios. Examples include logs produced by custom applications, web servers, and firewalls.

  • Glob pattern to specify the file(s) to collect; recursively collect files from a directory, a glob patterns allow you specify a set of file names using wildcard characters for example *.log.

  • Glob pattern to exclude files

  • Sends the entire existing content of files it finds

  • Tails existing files looking for new events

  • Multi-line logs

  • Reads gzip and bzip2 compressed files

  • Handles log rotation scenarios

Windows Events

Collecting Windows Events is simple and produces rich events. The Falcon LogScale Collector attempts to automatically detect which channels are available, or you can explicitly identify which channels you want to collect.

The Falcon LogScale Collector uses the internal Windows events templates to ensure the event is fully parsed where possible; this means that not only can you see the human readable representation of the event, you get all fields parsed automatically and the XML representation of the event.

Syslog Receiver

Collecting TCP and UDP syslog streams from within the infrastructure is an important feature in securing legacy logging scenarios. The Log Collector can listen for TCP or UDP syslog traffic on any port and will receive and buffer that data and stream it securely to LogScale.

Deploying the Falcon LogScale Collector close to the system sending syslog minimizes exposure to the unsecured traffic, and also provides maximum durability for syslog over UDP.

TLS encrypted syslog ingest is also supported in the Falcon LogScale Collector.

The Falcon LogScale Collector provides additional useful metadata on the events outside of the syslog envelope.

Exec Input

The Falcon LogScale Collector supports running a user configured sub-process to gather log data. This process is run based on a schedule and all the output produced by the sub-process on stderr and stdout is streamed to LogScale as events.

This allows the Falcon LogScale Collector to gather any information from the host that is available from the standard tools, or administrators can provide a script.

This custom input type can be used to extend the Falcon LogScale Collector to check host metrics, perform ping and HTTP based polling, or pull data from any other kind of API or service.

SystemD Logs on Linux

The journald source collects systemd logs from a local Linux journal. The structured journal has some advantages compared to plain text files, including built in filtering on specific systemd units, reading logs from the current boot only and built in log rotation.

The output of the source is similar, depending on the configuration, to what you would see with the journal viewer journalctl.

macOS Unified Logs

Unified Logs provide as of macOS 10 provide a unified source of logs which provides a range of information that can be used for forensics and to gain insight.

Log Collector Operation

Falcon LogScale Collector buffers data in memory or on disk (configurable). It offers a sub-second ingest lag between a line being written and sent to LogScale. It also supports event filtering, compression of data in transit, encryption of data in transit, and supports HTTP(S) proxies.

Log Collector Metadata

Falcon LogScale Collector creates fields with the prefix @collect.* to add metadata attached to events, including:

  • @collect.id

  • @collect.hostname

  • @collect.timestamp

    For more information on Falcon LogScale Collector Metadata see Log Collector Metadata.

Destination Sinks

The documentation explains how the Falcon LogScale Collector uses sinks as destinations for collected data, specifically optimized to send data to LogScale through proprietary ingest APIs over HTTP/HTTPS. Key features include support for multiple sinks per configuration, event buffering for optimization and durability, metadata attachment with the @collect prefix, and compatibility with custom TLS configurations and HTTP proxies.

Sinks are the destination of the data being collected, the Falcon LogScale Collector is designed to send data to LogScale only. It makes use of the Falcon LogScale proprietary ingest APIs as these have been optimized for efficient transport of event data including features like hierarchical metadata.

You can define multiple sinks for each configuration file. See Sinks (sinks) for more information.

The LogScale ingest APIs currently transport data over HTTP to the same ports that are used for the web interface for LogScale, no special ports need to be configured. By default the data is compressed and requires HTTPS, although these can be configured.

Falcon LogScale Collector can be configuration with additional transport options, including:

  • Custom TLS configuration to secure transmission.

  • Custom HTTPS security

  • Proxy support

Fleet Management

Fleet management, which is only available for enrolled instances of the Falcon LogScale Collector, provides a set of features aimed to facilitate the mangement of a large quantity of instances, for example, an overview of the status of all the instances and the configurations assigned to the instances.

Instance Enrollment

Enrolling instances essentially connects a Falcon LogScale Collector instances to the Fleet Management of Falcon LogScale so that you can view details on the status of all enrolled instances in one central location. There are multiple ways to enroll an instance in Fleet Management:

Groups

Groups allow you to manage the configuration of multiple instances of the Falcon LogScale Collector along with the possibility to combine configuration snippets to create a configuration which can be applied to all the instances in the group, these features are aimed at the bulk management of instances.

Instances can be grouped manually or by creating dynamic filters which automatically add any instances that match to the group.