Skip to content
LogScale Self-Hosted Deployment
This is an archived version of the documentation downloaded from LogScale Documentation Archives. Up to date documentation, and documentation for later versions, is available on LogScale Documentation

### Cluster topology

The LogScale cluster topology consists of multiple logical components including Ingest for receiving data, Digest for processing events, Storage for managing segment files, Query Coordination for handling search requests, UI/API nodes for user interaction, Kafka for internal communication, and Bucket Storage for object storage capabilities. Each component plays a specific role in the cluster architecture, with nodes capable of handling single or multiple responsibilities to create a robust and scalable log management system.

LogScale clusters are made up of different multiple logical components. It is possible to run nodes for a specific purpose, but it is also possible to run nodes with several responsibilities. The overall logical cluster topology looks like this:

Figure 11. Kubernetes Deployment Cluster Topology

The individual components are:

- Ingest

  Ingest processes receive requests from Third-Party Log Shippers that contain events through several supported ingest APIs. Events are parsed using system or user-defined Parse Data and are subsequently placed in a Kafka queue for further processing by the digest processes.

- Digest

  Digest processes read events from the Kafka ingest queue and build data files called segments. Queries for recent data in segment files are handled by the digest nodes. This includes data pushed to LogScale's live queries which are queries that are continuously running and aggregating data. Once segment files are completed they are placed in bucket storage, if configured, and future queries are serviced by the storage processes.

- Storage

  Storage processes store segment files and process queries for the segment files for which they are assigned. Older segments that may no longer reside on a storage instance will download the segment from bucket storage if configured. For most cases, it is recommended that digest nodes are configured to also be storage nodes.

- Query Coordination

  Query coordination processes receive queries from users, dashboards, alerts, and scheduled searches and create a query plan that sends internal queries to the digest and storage processes that own the segment files required for the query. These do not need to be reachable via the load balancer, but can be reached via the UI/API nodes.

- UI and API

  UI and API processes handle requests from client using a browsers or clients making API requests against the LogScale cluster.

- Kafka

  Kafka is used by LogScale as an internal cluster communication mechanism and as a queue for ingested events.

- Bucket Storage

  Bucket storage relies on a compatible object storage system such as Google Cloud Storage, MinIO, S3, or an S3 compatible API (support may be limited). When using bucket storage the segment files that are completed by the digest processes will upload the segment files to object storage. This feature allows redundancy for the segment files in the face of server failure in addition to allowing the querying of segment files that are no longer stored on storage process nodes due to age or storage capacity.

  The bucket storage functionality assumes it can read objects, so it is not compatible with write-only object storage systems. Any segment file that is being uploaded to bucket storage will be encrypted and decrypted on the LogScale nodes, even if the bucket storage system contains a built in encryption feature. More details on bucket storage can be found here: Bucket Storage.

#### Enter search term