Global Database

A LogScale cluster has a globally shared database which holds critical information about the cluster, storage and status. The global database has the following characteristics:

  • The global database is in memory on all LogScale nodes.

  • Event sourced database where all writes are messages put on a global-events Kafka queue.

    The Kafka queue (topic) only has one partition to guarantee a global ordering of messages. Each node in a LogScale cluster will read message of the queue and update its in memory database.

  • LogScale nodes make local snapshots of the global database.

    If bucket storage has been configured, a snapshot is also saved to bucket storage for use in recovery. If a node does not have a snapshot, the latest version can be fetched from bucket storage or transferred from another node.

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% graph LR subgraph K ["Kafka"] KGE["Global Events"] end subgraph SN ["Node"] IM["In Memory"] LS[("Local Snapshot")] end BS[("Bucket Storage")] IM-->K KGE-->IM IM-->LS IM-->BS

The global database does not store ingested event data, but does hold metadata that must be shared by all nodes, this includes (but is not limited to):

  • Record of segment files and timespans used to store data for each datasource

  • Map of which node stores which segment file, including those segment files in bucket storage

  • Dashboards and widgets

  • Parsers

  • Repositories

  • Views

  • Users

  • Groups

  • Roles

  • Saved queries

  • Alerts

  • Scheduled searches

  • Scheduled reports

The global database is a key part of the LogScale infrastructure and can be used during disaster recovery in the event of a failure. See LogScale High-Availability and Disaster Recovery (HA/DR) Implementation.