Ingesting Data to Multiple Repositories

You can specify that a parser ingests data into multiple repositories. Shipping all data to one parser and having that parser ship data to a number of repositories can be useful: for example, if logs are being sent from a single source, it is possible to setup one parser that can parse all events from this source and decide which repositories to send events to.

To ingest data to multiple repositories:

  1. Set the LogScale configuration variable ALLOW_CHANGE_REPO_ON_EVENTS to true, to allow LogScale to ingest to any repository specified by the repo tag.

  2. Set the repo tag by adding repo as a tag and setting the repo field in the parser, or by setting #repo on the endpoints which support setting tags directly (e.g., the HEC endpoint).

  3. Use an ingest token which must be valid for the repository a request is made against, but that also allows redirecting through the repo tag to any other repository. If the repository specified does not exist, then the event remains in the repository designated by the ingest token.

  4. Create a parser following the Example Configuration below.

This is a potential security issue on a public API endpoint, so this option should only be used inside a trusted environment.

Example Configuration

This is an example of parser code which copies events from one repository to another.

logscale
copyEvent("cloned_event") 
| case { #type="cloned_event" 
| repo := "target-repo-name"; * }

In this example the snippet achieves the following:

  • copyEvent("cloned-event") creates a copy of the event, where the #type="cloned_event" field is set.

  • We now have two events flowing through the parser, one where #type = "cloned_event", and one without that field.

  • We then use the case construct to direct the two events to a different target repo.