Ingesting Data to Multiple Repositories

It's possible to specify that a parsers ingest data into multiple repositories.

It can be useful to ship all data to one parser and have that parser ship data to many different repositories. For example if logs are being sent from a single source, it is possible to setup one parser that can parse all events from this source and decide which repositories to send events to.

To ingest data to multiple repositories:

  • You must set the LogScale configuration variable ALLOW_CHANGE_REPO_ON_EVENTS to true, to allow LogScale to ingest to any repository specified by the repo tag.

  • Set the repo tag by adding repo as a tag and setting the repo field in the parser, or by setting #repo on the endpoints which support setting tags directly (e.g., the HEC endpoint).

  • Using a ingest token which must be valid for the repository a request is made against, but that also allows redirecting through the repo tag to any other repository. If the repository specified does not exist then the event remains in the repository designated by the ingest token.

  • Create a Parser following Example Configuration

This is a potential security issue on a public API endpoint, so this option should only be used inside a trusted environment.

Example Configuration

This is an example of parser code which copies events from one repository to another.

logscale
copyEvent("cloned_event") 
| case { #type="cloned_event" 
| repo := "target-repo-name"; * }

In this example this snippet achieves the following: copyEvent("cloned-event") creates a copy of the event, where the #type="cloned_event" field is set. So after the copyEvent call, we now have two events flowing through the parser, one where #type = "cloned_event", and one without that field. We then use the case construct to direct the two events to different targets.