Event Forwarding

Event Forwarding in LogScale enables organizations to transmit ingested events to external systems for purposes like Machine Learning while maintaining centralized logging, primarily using Kafka as the forwarding mechanism. The feature includes configurable forwarders and rules for filtering and transforming events, with JSON-formatted output and built-in error handling through circuit breakers, though it operates without delivery guarantees to maintain optimal ingestion performance.

Event forwarding is used to forward events that are ingested into LogScale to other systems that require parts of the data —Machine Learning, for example— while still logging everything in a central place.

Currently, forwarding is done using Kafka.

graph LR; LS["Log Shipper"] --Ingest API--> L{LogScale} R["Repository"] --> P["Parser"] L --> P P --> EF["Event Forwarder"] P --> R EF --> S((Storage)) style LS fill:#40bd28 style L fill:#fc0000 style R fill:#f5ad42 style P stroke-dasharray: 5 5 style EF fill:#lightgrey style S fill:#lightgrey
Diagram illustrating the LogScale Event Forwarding architecture and data flow. The flowchart shows how data moves through the system: a Log Shipper sends data via the Ingest API to LogScale, which then passes the data through a Parser. The Parser simultaneously sends the processed data to both a Repository (for storage within LogScale) and to an Event Forwarder component. The Event Forwarder then transmits the selected data to external storage systems (such as Kafka). This architecture allows LogScale to maintain a complete record of ingested data internally while also forwarding specified events to external systems for additional processing or integration with other platforms.

Figure 70. Event Forwarding


When a log shipper is ingesting data, it is first parsed. Then it is stored in a repository in LogScale and sent through any Event Forwarding Rules on the repository, which can filter and transform the events before forwarding them to an external system through an Event Forwarders.

Note

If you ingest events with timestamps older than your retention policy allows, the data will be forwarded without being stored in LogScale.

Configuration

Security Requirements and Controls

Event forwarding is configured in two steps. First, an Event Forwarders is created that defines a Kafka server and topic to forward the events to. Second, one or more Event Forwarding Rules are configured on the repositories, where events should be forwarded from, and connected to the event forwarders.

You must enable event forwarding in the configuration using ENABLE_EVENT_FORWARDING.

Event Forwarding Format

Events are sent as one event per message. The format is JSON where every field is a top-level element, and all field values are returned as JSON strings, except for @timestamp, which is returned as a number representing time as Unix time in milliseconds (UTC/Zulu time).

Error Handling

There are no delivery guarantees on event forwarding. This means, that if forwarding an event fails, LogScale will neither retry later nor fail the ingest. This is done in order not to impact ingest.

Note

A failure during parsing does not necessarily mean a failure during forwarding. As described in the Parser Errors documentation, this will simply set a number of fields to error values, such as @event_parsed = false. Unless filtered out by the forwarding rule (using for example: @event_parsed != false), these events will still be forwarded.

LogScale will only log that event forwarding failed if multiple events fail to be forwarded, in order not to spam the logs. The number of events that failed to be forwarded can be seen in the metric event-forwarding-errors. See LogScale Metrics.

LogScale uses circuit breakers to prevent event forwarding errors from affecting the performance and stability of LogScale. The circuit breakers can be configured using the configuration options that have the FORWARDING_BREAKER_ prefix (for example, FORWARDING_BREAKER_FAILURES).

LogScale also enforces a limit on the number of events concurrently waiting to be forwarded. This is to prevent events exhausting memory due to a stuck/blocked forwarder. This limit can be changed by setting FORWARDING_MAX_CONCURRENCY.