Event Forwarding

Event Forwarding enables you to transmit parsed, ingested events from LogScale to external systems via Kafka, while continuing to store and manage all data centrally in LogScale.

This feature allows other systems, such as enrichment pipelines, machine learning platforms, or downstream processing tools, to consume LogScale data in real time without affecting your central logging.

Important

On-Premises Only

Event Forwarding is intended for on-premises LogScale deployments only. It is not recommended for cloud-hosted environments due to the operational complexity and risk involved in managing Kafka producer configuration at scale.

How Event Forwarding Works

When a log shipper ingests data into LogScale, the following sequence occurs:

flowchart TD A[Log Shipper sends data] --> B[LogScale Ingest] B --> C[Parser runs] C --> D[Parsed event stored in Repository] D --> E{Event Forwarding Rules evaluated} E -->|Rule matches| F[Event forwarded to Kafka topic] E -->|No match| G[Event not forwarded]
flowchart TD A[Log Shipper sends data] --> B[LogScale Ingest] B --> C[Parser runs] C --> D[Parsed event stored in Repository] D --> E{Event Forwarding Rules evaluated} E -->|Rule matches| F[Event forwarded to Kafka topic] E -->|No match| G[Event not forwarded]
Diagram illustrating the LogScale Event Forwarding architecture and data flow. The flowchart shows how data moves through the system: a Log Shipper sends data via the Ingest API to LogScale, which then passes the data through a Parser. The Parsed event is stored in the Repository. The Parser then sends the data to an Event Forwarder which then transmits the selected data to external storage systems (such as Kafka) if the rule matches. This architecture allows LogScale to maintain a complete record of ingested data internally while also forwarding specified events to external systems for additional processing or integration with other platforms.

Figureย 7.ย Event Forwarding Pipeline


Key points about this pipeline:

  • Forwarding occurs after parsing. Rules operate on fully parsed events, not raw log data

  • Forwarding is non-destructive. The event stored in LogScale is always the parsed version from the parser, regardless of any transformations applied in forwarding rules

  • If you ingest events with timestamps older than your retention policy allows, the data will be forwarded without being stored in LogScale

Important

Cross-view references (Resolving Resources to a Different Named View or Repository) are actively blocked for parsers and saved queries during event forwarding.

Use Cases

Event Forwarding is best suited for scenarios where you need to route live data to an external system for further processing, while retaining full data in LogScale. Common use cases include:

Use Case Description
Enrichment Forward events to an external system that adds contextual data (e.g. threat intelligence, asset information) before re-ingesting or storing elsewhere
Machine Learning (ML) Stream events to an ML platform for real-time model training or inference
Downstream processing Feed events into a broader data pipeline for transformation or aggregation
Integration Supply data to other tools in your ecosystem that consume from Kafka

Event Forwarding vs. Archiving

Event Forwarding is not a backup or archival mechanism. If your goal is durable data backup or long-term retention, use Archiving, which is more robust for that purpose. Event Forwarding is designed for live routing of data to external systems, and operates without delivery guarantees. See Archive Data for more details.

Configuration overview

Security Requirements and Controls

Event forwarding is configured in two steps:

  1. Create an Event Forwarder. Event Forwarders define the Kafka server and topic to forward events to.

  2. Create one or more Event Forwarding Rules. These rules are configured per repository, and define which events to forward, and how.

You must first enable Event Forwarding in your LogScale configuration using ENABLE_EVENT_FORWARDING=true.

This flag must be set in your LogScale server configuration before Event Forwarding can be used. By default, Event Forwarding is disabled.

Note

Permission Required:

You must have the Change event forwarding permission to configure Event Forwarding. Root access is required to manage Event Forwarders. See Manage root access for more details.

Event Forwarding Format

Events are forwarded as one event per Kafka message in JSON format. All field values are returned as JSON strings, with the following exception:

  • @timestamp is returned as a number representing Unix time in milliseconds (UTC/Zulu time).

Every field is a top-level element in the JSON object.

Error Handling

Delivery Guarantees

There are no delivery guarantees on Event Forwarding. If forwarding an event fails, LogScale will neither retry the delivery nor fail the ingest. This design decision ensures that forwarding failures cannot impact ingest performance.

Note

A failure during parsing does not necessarily mean a failure during forwarding. As described in the Parser Errors documentation, this type of failure sets a number of fields to error values, such as @event_parsed = false. Unless filtered out by the forwarding rule (using for example: @event_parsed != false), these events are still forwarded.

LogScale only logs that Event Forwarding failed if multiple events fail to be forwarded, in order not to overwhelm the logs. The number of events that failed to be forwarded are shown in the metric event-forwarding-errors. See LogScale Metrics.

Circuit Breakers

LogScale uses circuit breakers to prevent Event Forwarding errors from affecting the performance and stability of the platform. Circuit breaker behaviour can be configured using options with FORWARDING_BREAKER_ prefix (for example, FORWARDING_BREAKER_FAILURES=10).

Concurrency Limits

LogScale enforces a limit on the number of events concurrently waiting to be forwarded. This prevents a stuck or blocked forwarder from exhausting available memory. This limit can be adjusted using FORWARDING_MAX_CONCURRENCY=value.

Performance and Risk Considerations

Event Forwarding introduces operational risk that must be carefully managed. Before deploying Event Forwarding in a production environment, consider the following:

Kafka Producer Misconfiguration

The most significant risk with Event Forwarding is incorrect Kafka producer configuration. While a forwarding failure itself will not directly impact LogScale ingest (due to circuit breakers and the no-retry design), a misconfigured Kafka producer can consume excessive CPU resources on the LogScale node. This CPU exhaustion can indirectly degrade LogScale performance, including ingest throughput.

Ensure your Kafka producer properties (particularly batch.size and buffer-related settings) are appropriately sized for your expected data volume.

Capacity Planning Checklist

Before going live with Event Forwarding, verify the following:

Check Detail
Kafka producer properties sized correctly Match batch and buffer settings to your data volume
Kafka cluster has sufficient resources Especially if sharing with LogScale's internal Kafka
Receiving system is scaled appropriately The downstream Kafka topic consumer must handle the forwarded volume
Forwarding rules are selective Avoid forwarding more data than necessary. Use rules to filter
Monitoring is in place Track event-forwarding-errors metric and set up alerting

Monitoring

Check Detail
event-forwarding-errors Count of events that failed to be forwarded
FORWARDING_BREAKER_ options Configure circuit breaker thresholds
FORWARDING_MAX_CONCURRENCY Limit concurrent forwarding to protect memory