Syslog Source: Multi-Destinations Sinks

Multi-destination routing allows one syslog source to feed multiple destinations, enabling content-based filtering to route logs based on specific criteria, destination-specific transformations to process data differently for each endpoint, and team-specific views to provide customized access to the same underlying data.

The multi-destination uses a loopback architecture that consists of these key components:

  1. Syslog Source: Receives syslog messages over UDP or TCP

  2. Loopback Sink: Acts as an intermediary distribution point that stores events in memory

  3. Internal Sources: Connect to the loopback sink and forward events to different destinations

  4. Transforms: Optional processing rules applied to each routing path

  5. Destination Sinks: Final endpoints where logs are delivered (LogScale, NGSIEM, etc.)

Delivery and Backpressure in Multi-Destination Systems

When routing data to multiple destinations that process at different rates, effective backpressure management becomes critical. In multi-destination architectures, each sink may consume data at varying speeds, creating potential bottlenecks.

Primary vs. Secondary Forwarders

Our system implements a priority-based approach to handle these scenarios:

  • Primary Forwarders (default): Can pause the data source when their queue fills up, preventing data loss but potentially slowing overall throughput.

  • Secondary Forwarders: When configured as secondary and their queue fills up, they will not pause the source. This allows continuous data flow to primary destinations at the cost of potential data loss at the secondary sink.

Queue Management - Multi-destination

Each destination has a configurable queue that buffers incoming data. When a destination processes data slower than it arrives the the data is:

  1. Data accumulates in the destination's queue

  2. Once full, the system's response depends on the forwarder's priority setting

  3. Primary forwarders signal backpressure to the source

  4. Secondary forwarders drop excess data to maintain system throughput

This configurable priority system allows you to tailor the behavior to your specific requirements, balancing between data completeness and system performance.

How to Use Multi Destination Sinks

To set multiple destinations for a syslog source configure in source a Internal source and in sinks a distributor with a sink for each destination.

The fields used by the loopback feature need to be configured in both the source and sinks:

  • In the Source define one or more sources of type internal for each split of the data.

  • In the Sinks define a sink of type loopback and the final sinks for the redistributed data.

yaml
sources:
  syslog_input:
    type: syslog
    port: 514
    mode: udp  # or tcp
    sink: distributor  # points to a loopback sink

  route_to_destination1:
    type: internal
    from: distributor  # references the loopback sink name
    sink: destination1  # points to the first destination sink
    transforms:
      - type: regex_filter  # optional transformation
        pattern: "^ERROR"
        mode: include

  route_to_destination2:
    type: internal
    from: distributor
    sink: destination2
    transforms:
      - type: regex_filter
        pattern: "^INFO"
        mode: include

sinks:
  distributor:
    type: loopback  # this is the distribution point

  destination1:
    type: logscale  # or any other sink type
    token: your-token-1
    url: https://cloud.logscale.com

  destination2:
    type: logscale
    token: your-token-2
    url: https://cloud.logscale.com