Global Database

A LogScale cluster has a globally shared database which holds critical information about the cluster, storage and status. The global database has the following characteristics:

  • The global database is in memory on all LogScale nodes.

  • Event sourced database where all writes are messages put on a global-events Kafka queue.

    The Kafka queue (topic) only has one partition to guarantee a global ordering of messages. Each node in a LogScale cluster will read message of the queue and update its in memory database.

  • LogScale nodes make local snapshots of the global database.

    If bucket storage has been configured, a snapshot is also saved to bucket storage for use in recovery. If a node does not have a snapshot, the latest version can be fetched from bucket storage or transferred from another node.

The global database does not store ingested event data, but does hold metadata that must be shared by all nodes, this includes (but is not limited to):

  • Record of segment files and timespans used to store data for each datasource

  • Map of which node stores which segment file, including those segment files in bucket storage

  • Dashboards and widgets

  • Parsers

  • Repositories

  • Views

  • Users

  • Groups

  • Roles

  • Saved queries

  • Alerts

  • Scheduled searches

  • Scheduled reports

The global database is a key part of the LogScale infrastructure and can be used during disaster recovery in the event of a failure. See LogScale High-Availability and Disaster Recovery (HA/DR) Implementation.