File Snapshot Source

Overview

The File Snapshot Source monitors directories for files that arrive complete, and ingests each file once as a snapshot.

For example, files that might be ingested include:

  • Inventory lists

  • Database dumps

  • Rotated archives

  • SAP audit files, and so on

Unlike the File Source, the File Snapshot Source does not tail growing files. It waits for a file to stop changing. Then, it reads the file from start to end. The source marks the file as ingested to avoid duplication.

How it works

The File Snapshot Source monitors directories for complete files and processes them once. Key features include:

  • Discovery and Scheduling: The source accepts glob patterns (single or list) to match files. Matching paths are polled every 100 ms. Symlinks are skipped.

  • Settle Time: Configurable wait time (default 1 second) for file modification time to remain unchanged before processing. This ensures that the source does not ingest partially written files.

  • Once-Only Processing: The source checkpoints each file under its source name and path, together with its modification time. Restarting the collector does not re-send files that were already processed.

  • Multiple Parsing Modes: Supports line-delimited text, CSV with RFC 4180 parsing, and SAP audit log fixed-width records.

Once-only processing

Each file is checkpointed under its source name and path, together with its modification time:

  • Restarting the collector does not re-send files that the source has already processed.

  • The source re-ingests in full any file whose modification time changes.

  • The source writes the checkpoint only after the last batch has been acknowledged upstream. An interrupted transfer causes the source to re-send the file rather than lose it.

Parsing modes

The parsing mode is selected with mode.type and defaults to line.

Line mode

Line mode supports newline-delimited text events. Trailing carriage returns and trailing spaces are trimmed.

CSV mode

CSV mode supports RFC 4180 parsing with lazy quotes. It supports a variable number of fields per record.

  • delimiter (default ,) is the field delimiter and must be exactly one character.

  • columns.type: header reads the column names from the first data line (after any preamble).

  • columns.type: list takes the names from an explicit names list.

  • With column names configured, the source emits each record as a JSON object. Fields missing from a record become empty strings. The source collects surplus fields into an extra array.

  • With no columns, the source passes the record through as a plain delimited line.

SAP mode

SAP mode supports SAP audit log fixed-width records. recordBytes (default 400) is the record length in bytes. The source decodes each record into a JSON object with the following fields:

  • version

  • message_id

  • timestamp

  • os_pid

  • sap_pid

  • logon_type

  • sap_pid_hex

  • username

  • transaction

  • report

  • mandant

  • session_id

  • parameters

  • terminal

Events and fields

  • Every event carries @collect.file (the file name) and @collect.modified (the modification time in epoch milliseconds).

  • addLineNumber (default false) adds @collect.line. In block and sap mode this counts records.

  • preambleLines (default 0) skips leading lines in line and csv mode.

  • encoding selects the file encoding: UTF-8 (default), UTF-16LE, or UTF-16BE.

Robustness

  • The source batches events up to the upstream batch limits and honours backpressure. Large files are streamed in chunks; small files are read in one shot.

  • In sap mode, the source checks file size. If the size is not a multiple of recordBytes, the source skips the file with a warning. If the file changes while being read, the source skips it. The source retries the file on a later scan.

  • The source drops an event larger than the maximum batch size with a warning. This prevents stalling the file.

Examples

These examples are added to the pipeline configuration file, alongside the sinks that they refer to.

SAP audit logs

yaml
sources:
  sapAudit:
    type: file_snapshot
    include: /usr/sap/*/log/*.AUD
    sink: logscale
    mode:
      type: sap
      recordBytes: 400

Semicolon-separated CSV exports with column names on the first line, ignoring files already on disk at startup

yaml
sources:
  exports:
    type: file_snapshot
    include:
      - /var/exports/*.csv
    exclude:
      - /var/exports/tmp-*.csv
    sink: logscale
    settleTime: 5
    processExisting: false
    addLineNumber: true
    mode:
      type: csv
      delimiter: ";"
      columns:
        type: header

Plain UTF-16 text drops with a two-line header to skip

yaml
sources:
  drops:
    type: file_snapshot
    include: /var/spool/drops/*.log
    sink: logscale
    encoding: UTF-16LE
    preambleLines: 2
Configuration Parameters

Table: File Snapshot source

ParameterTypeRequiredDefault ValueDescription
addLineNumberbooleanoptional[a] false adds @collect.line to each event.
encodingutf-8, utf-16be, utf-16leoptional[a] UTF-8 Unlike the File Source, this does not auto-detect from BOM.
excludegloboptional[a]   Files to exclude; env vars expanded.
includeglobrequired   Directories/files to monitor; env vars expanded; a bare scalar is accepted as a 1-element list.
modeobjectrequired   Parsing mode.
preambleLinesintegeroptional[a] 0 Lines skipped at the start of each file.
processExistingbooleanoptional[a] true Process files already present at startup.
settleTimeintegeroptional[a] 1 Time to wait in seconds for file size stability before processing.

[a] Optional parameters use their default value unless explicitly set.