Event Forwarding
Event forwarding is used to forward events that are ingested into Humio to other systems that require parts of the data for e.g. Machine Learning, while still logging everything in a central place. Currently, forwarding is done using Kafka.
Figure 26. Event Forwarding>
When data is Ingesting Data, it is first parsed. Then it is stored in a repository in Humio and sent through any Event Forwarding Rules on the repository, which can filter and transform the events before forwarding them to an external system through an Event Forwarders.
If you ingest events with timestamps older than your retention policy allows, the data will be forwarded without being stored in Humio.
Configuration
Event forwarding is configured in two steps. First, an Event Forwarders is created that defines a Kafka server and topic to forward the events to. Second, one or more Event Forwarding Rules are configured on the repositories, where events should be forwarded from, and connected to the event forwarders.
You need to enable event forwarding in the configuration using
ENABLE_EVENT_FORWARDING.
Format
Events are sent as one event per message. The format is JSON where every field is a top-level element, and all field values are returned as JSON strings, except for @timestamp, which is returned as a number representing time as Unix time in milliseconds (UTC/Zulu time).
Error Handling
There are no delivery guarantees on event forwarding. This means that if forwarding an event fails, Humio will neither retry later nor fail the ingest. This is done in order to not impact ingest.
Note that a failure during parsing does not necessarily mean a failure
during forwarding. As described in the
Parser Errors documentation, this will simply
set a number of fields to error values, such as
@event_parsed = false. Unless
filtered out by the forwarding rule (using e.g.
@event_parsed != false), these
events will still be forwarded.
Humio will only log that event forwarding failed if multiple events fail
to be forwarded, in order not to spam the logs. The number of events
that failed to be forwarded can be seen in the metric
event-forwarding-errors, see
LogScale Metrics.
Humio uses
circuit
breakers to prevent event forwarding errors from affecting the
performance and stability of Humio. The circuit breakers can be
configured using the configuration options that have the
FORWARDING_BREAKER_ prefix.
Humio also enforces a limit on the number of events concurrently waiting
to be forwarded. This is to prevent events exhausting memory due to a
stuck/blocked forwarder. This limit can be changed by setting
FORWARDING_MAX_CONCURRENCY.