LogScale Multi-Cluster Search

LogScale Multi-Cluster Search enables you to create views that search and aggregate data across multiple local and remote clusters. It improves workload distribution and lets you combine query data from different regional locations. The feature is available exclusively on Self-hosted deployments. It operates through a special Multi-Cluster View that establishes secure connections between clusters using Repository Tokens. You can run queries and combine results from multiple configured downstream clusters.

Important

Multi-Cluster search is available on Self-hosted deployments only.

Multi-Cluster search enables you to create a view that searches across multiple views from local and remote clusters. Using multi-cluster search provides a number of benefits:

  • Aggregation of results from multiple clusters and views, for example aggregating information from one or more clusters

  • Increased distribution of workload across clusters when performing searches

  • Ability to combine and aggregate query data across clusters in multiple regional locations

Multi-Cluster search works by creating a special type of view, the Multi-Cluster View. The multi-cluster view creates a connection between local and remote clusters. You can execute queries across each configured downstream cluster in the view configuration. LogScale executes the query using the standard query API. It uses a Repository Token which provides secure access to the remote repository or view.

The upstream, or parent, cluster that creates the Multi-Cluster view supports two connection types: local and remote:

  • A local connection refers to a view on the same cluster as the multi-cluster view. Only one local connection can be part of a multi-cluster view.

  • A remote connection refers to a view on a remote cluster. Multiple remote connections are supported, but only one connection to each remote cluster.

Remote cluster connectivity and authentication operate at the view or repository level. You must grant connectivity for each individual repository, and the secret token created through Repository Token system limits the access to those users with rights to create a suitable API token with Data read access. Remote cluster search does not enable unlimited access to all views on the remote cluster. For more information, see Multi-Cluster Security.

An overview of the connectivity between local and remote connections and the parent multi-cluster view are shown in the following diagram:

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% graph LR C1["Cluster"] V1["Local View"] V2["Local View"] F3["Multi-Cluster View"] C1---V1 C1---V2 C1---F3 RC1["Remote Cluster"] RC1V1["Remote View"] RC1V2["Remote View"] RC1V1---RC1 RC1V2---RC1 RC2["Remote Cluster"] RC2V1["Remote View"] RC2V2["Remote View"] RC2V1---RC2 RC2V2---RC2 F3---V2 F3---RC1V1 F3---RC2V2 linkStyle 8 stroke:#ff0000 linkStyle 9 stroke:#ff0000 linkStyle 7 stroke:#ff0000
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% graph LR C1["Cluster"] V1["Local View"] V2["Local View"] F3["Multi-Cluster View"] C1---V1 C1---V2 C1---F3 RC1["Remote Cluster"] RC1V1["Remote View"] RC1V2["Remote View"] RC1V1---RC1 RC1V2---RC1 RC2["Remote Cluster"] RC2V1["Remote View"] RC2V2["Remote View"] RC2V1---RC2 RC2V2---RC2 F3---V2 F3---RC1V1 F3---RC2V2 linkStyle 8 stroke:#ff0000 linkStyle 9 stroke:#ff0000 linkStyle 7 stroke:#ff0000

The search follows this process:

  • Send the query to each configured connection

  • Read the event data from each connection

  • Combine the result set

This process is outlined in the following diagram:

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% graph LR C1["Multi-Cluster View Search"] V1["Local View"] V2["Remote View"] V3["Remote View"] CM["Combine Results"] CCE["Combine Event Data"] C1 -- Send Query --- V1 -- Return Events---CM---CCE C1 -- Send Query --- V2 -- Return Events---CM C1 -- Send Query --- V3 -- Return Events---CM
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% graph LR C1["Multi-Cluster View Search"] V1["Local View"] V2["Remote View"] V3["Remote View"] CM["Combine Results"] CCE["Combine Event Data"] C1 -- Send Query --- V1 -- Return Events---CM---CCE C1 -- Send Query --- V2 -- Return Events---CM C1 -- Send Query --- V3 -- Return Events---CM

Because queries go to each configured connection, processing a multi-cluster query takes longer than a purely local query. LogScale must collect and merge results from every cluster connection before returning them.

Each remote cluster executes the query. The resources available on each cluster affect query performance. The speed of the query across a Multi-Cluster search will be as slow as the slowest cluster. If your Multi-Cluster search is slow, ensure that each cluster has been sized accordingly.