Variables Governing Limits

Some variables limit the operation of both queries and individual functions, and they govern the overall execution of queries within LogScale. These limits manage cluster resources and therefore overall cluster performance. Specifically, they prevent a single query from consuming all the memory or CPU capacity of the cluster. As a result, LogScale sustains data ingestion and continues to execute other queries.

Some of these limits are themselves configurable with dynamic variables.

These limits control the following:

  • Overall limits for the number of generated or processed events. They exist to help manage resource limits for the cluster, preventing queries from consuming all the resources. For example, the dynamic configuration parameter QueryResultRowCountLimit limits the maximum number of events returned by any query.

  • Limits for the operation of individual functions, often configured either as an absolute limit, or both a default and function-configurable overall limit.

    For example, the dynamic configuration parameter GroupDefaultLimit sets the default number of events in the function groupBy(), while the dynamic configuration parameter GroupMaxLimit sets the absolute limit that a user can set for the limit parameter to the groupBy() function.

  • Limits for resource usage for the query engine. For example, the dynamic configuration parameter QueryCoordinatorMemoryLimit sets the maximum amount of memory the query coordinator, which manages the execution of a single query, including the final aggregation step. This is designed to prevent single query coordinators from consuming all available resources.

Note

Pay attention to the limits and their impact on how a query executes- if too few results are returned or the data is not as robust as expected, it may be because the result has been limited by one or more of the functions in the pipeline.