Ingest Nodes

Ingest nodes read and parse the incoming data and identify individual fields and optionally tag and identify the data so that it can be more organized to optimize searching. Ingestion is designed to be fast so that data can be ingested and processed.

For details on the overall ingest process, see Ingesting Data Flow. A more detailed description is available in Ingesting Data.

Digest Nodes

Digest nodes take the ingested data and convert that to the store format used by the rest of the LogScale so that the data can be easily searched.

Digestion of data is an internal process that performs the operation process of physically storing the data on disk or bucket storage. For information on the digest process, see Ingestion: Digest Phase.

Storage Nodes

Storage nodes provide access to the stored data that, including executing the access to the data during a search request.

Storage nodes handle both the storage of information but also the recovery of data when running a query on historical information. For details on how LogScale stores data, see Ingestion: Storage Phase.

Query Coordination Nodes

Query coordination nodes process a query and distribute the requests for the data to data storage nodes to access the data and return the query data. The query coordinators assemble, format and aggregate the data.

Query coordination nodes handling searching. For detailed information on the search process, see Search Architecture.

UI Nodes

The UI nodes support the web interface to the underlying data. This includes a query interface, widgets and dashboards that query, summarize and display the data.

Bucket Storage

LogScale stores data in files called segments and uses and caches as much of this information as possible to ensure the highest performance when querying data. Because the volumes of data can be so large, having enough local storage (for example a local SSD or NVME drive) would be impractical. For this reason, LogScale can optionally store data on bucket storage, including Amazon S3, Google Cloud Storage, or MinIO. When using bucket storage, LogScale writes the data locally and to the bucket store. Older data in the local is expired, but if a query is executed that needs data from bucket storage, the segments are copied back from bucket storage for processing. This enables LogScale to store and process petabytes of information in an efficient and performant manner.

For more information on bucket storage, see Ingestion: Storage Phase.

Global Database

A shared store of key cluster information, including cluster members, segment files, and the distribution of data. The management and update of the information is exchanged between nodes by using a Kafka queue with the changes. Because the global database includes the reference location information for all the stored data, the global database is also backed up periodically to bucket storage.

For more information on the global database, see Global Database.

Access to cluster should be via a Load Balancer, so that ingestion, queries and UI access can be spread across the different nodes in the cluster while presenting a single IP address for accessing the entire cluster.

Clusters can be scaled by adding further nodes. Adding nodes increases the potential for ingest and query capacity, but also implies additional management load.

Data is distributed, and duplicated, across the cluster to improve performance and support high availability. Node unavailability, for example during maintenance, upgrades or failure, are mitigated through this data and workload distribution.

LogScale stores its own logs and metadata within LogScale itself. This means, to understand the operation or performance of LogScale you can query and monitor LogScale by querying these internal data stores.

All data is timestamped. If a timestamp cannot be determined from the incoming data (for example a log file), the time of ingest is used as the timestamp.

The smallest data structure within LogScale is the event, this is a single, timestamped, entry within the LogScale data and typically refers to a single row or record of information from the source input logs. Events are used internally by LogScale to exchange information. A repository is a collection of timestamped events, and when searching, data is exchanged as a list of events between each section.

All event data is stored using the identified fields of the data and also the raw incoming string. This allows for data to be filtered on explicit fields but doesn't limit the searchable data to this structured information.

LogScale uses a special query language, LogScale Query Language (LQL), that is modeled on the Unix-shell principle of pipes. A list of events is processed through a language fragment that filters, augments, or aggregates the list of events from the previous step, with each step connected by a pipe.