Query Scheduling

LogScale uses a query scheduler internally to help manage the sequence and order of query execution:

The FairQuery scheduler aims to balance the execution of queries across the cluster in a fair manner to get the best performance. The following rules apply when considering query scheduling:

  • Replication of data segments takes priority over querying data to ensure that events are stored.

  • Uploading segments to object storage is prioritized over downloading existing segments for queries.

  • A first-in, first-out (FIFO) queue is used to manage the download of segments.

    When inserting requests into the queue, segments are selected by looking at all the queries that want to fetch the same segments and then ordering the segment requests by how work will need to be performed locally to process them. The goal is to try to prevent any queries from being unable to run due to waiting for bucket storage. As a query gets closer to having no work ahead of it locally, it will get more priority to ensure that the query can still be executed.