Data Sources

Data sources are the data points from which the data is collected. Falcon LogScale Collector currently supports the following inputs or data sources:

Collecting Events from Files

Collecting events from local files on disk is one of the most common log collection scenarios. Examples include logs produced by custom applications, web servers, and firewalls.

  • Glob pattern to specify the file(s) to collect; recursively collect files from a directory, a glob patterns allow you specify a set of file names using wildcard characters for example *.log.

  • Glob pattern to exclude files

  • Sends the entire existing content of files it finds

  • Tails existing files looking for new events

  • Multi-line logs

  • Reads gzip and bzip2 compressed files

  • Handles log rotation scenarios

Collecting Windows Events

Collecting Windows Events is simple and produces rich events. The LogScale Collector attempts to automatically detect which channels are available, or you can explicitly identify which channels you want to collect.

The LogScale Collector uses the internal Windows events templates to ensure the event is fully parsed where possible; this means that not only can you see the human readable representation of the event, you get all fields parsed automatically and the XML representation of the event.

Syslog Receiver

Collecting TCP and UDP syslog streams from within the infrastructure is an important feature in securing legacy logging scenarios. The LogScale Collector can listen for TCP or UDP syslog traffic on any port and will receive and buffer that data and stream it securely to LogScale.

Deploying the LogScale Collector close to the system sending syslog minimizes exposure to the unsecured traffic, and also provides maximum durability for syslog over UDP.

TLS encrypted syslog ingest is also supported in the LogScale Collector.

The LogScale Collector provides additional useful metadata on the events outside of the syslog envelope.

Exec Input

The LogScale Collector supports running a user configured sub-process to gather log data. This process is run based on a schedule and all the output produced by the sub-process on stderr and stdout is streamed to LogScale as events.

This allows the LogScale Collector to gather any information from the host that is available from the standard tools, or administrators can provide a script.

This custom input type can be used to extend the LogScale Collector to check host metrics, perform ping and HTTP based polling, or pull data from any other kind of API or service.

Collecting Logs from SystemD on Linux

The journald source collects systemd logs from a local Linux journal. The structured journal has some advantages compared to plain text files, including built in filtering on specific systemd units, reading logs from the current boot only and built in log rotation.

The output of the source is similar, depending on the configuration, to what you would see with the journal viewer journalctl.

Collecting Logs from macOS Unified Logs

Unified Logs provide as of macOS 10 provide a unified source of logs which provides a range of information that can be used for forensics and to gain insight.