Limits & Standards

This page lists the various limits and standard operating parameters of LogScale. See Ingest via API Best Practices for the best practices relative to ingest via the Ingest API.

General Limits & Parameters

Below is a list of general limits and parameters of LogScale:

Note

Unless otherwise specified, all multiple-bye data sizes in LogScale are expressed in SI units using decimal (Base 10). For example:

  • 1KB = 1,000 Bytes

  • 1MB = 1,000,000 Bytes

  • 1GB = 1,000,000,000 Bytes

  • 1TB = 1,000,000,000,000 Bytes

Description Default Limit Controlling Variable LogScale version
Character set support UTF-16   All
Max number of fields in an event

During ingest, fields are sorted alphabetically by name and the first 1000 fields are parsed, the remainder of the named fields are dropped. Please note that the @rawstring is not modified and will contain all data.

1,000 MAX_EVENT_FIELD_COUNT All
Max event size 1 MB MAX_EVENT_SIZE All
Max file size 100 MB MAX_FILEUPLOAD_SIZE All
Max CSV lookup file upload size 200 MB MaxCsvFileUploadSizeBytes dynamic configuration. If MAX_FILEUPLOAD_SIZE environment variable is set, its value will be the default. From 1.124
Max JSON lookup file upload size 100 MB MaxJsonFileUploadSizeBytes dynamic configuration. If MAX_FILEUPLOAD_SIZE environment variable is set, its value will be the default. From 1.124
Max query length in characters 66,000   All
Memory limit for the mapper phase of a collect() function running as a top-level function i.e., how much data such a function can store. 10 MiB   From v1.79
Memory limit for the mapper phase of a collect() function running in a subquery, or as a subaggregator to another function i.e., how much data such a function can store. 1 MiB   From v1.79
Max number of events in tail(), head(), and sort() functions. 20,000 StateRowLimit All
Max number of events in an RDNS request. See rdns() function. — StateRowLimit Up to v1.136
  RdnsMaxLimit From v1.137
Default number of events set in an RDNS request. See rdns() function. — RdnsDefaultLimit From v1.137
Max number of data sources in a repository 10,000 MAX_DATASOURCES All
Max number of rows that join() and selfJoin() can return. 200,000 or the value of MAX_JOIN_LIMIT, if it has been modified Dynamic configuration JoinRowLimit , replacement for MAX_JOIN_LIMIT environment variable. All
Default value for the limit parameter in groupBy(), selfJoin() and some other functions, when not specified. Replacement for the environment variable MAX_STATE_LIMIT to determine the default value. GroupDefaultLimit has a default value of either 20,000 or the value of the config GroupMaxLimit, whichever is smallest. GroupDefaultLimit cannot be larger than GroupMaxLimit. 20,000 Dynamic configuration GroupDefaultLimit All
Max value for limit parameter in the groupBy() function, meaning that the function can collect up to a million groups. Previously, this parameter was bounded by MAX_STATE_SIZE, so if you have made any modifications to this variable, set GroupMaxLimit to the same value for a seamless upgrade. Due to stability concerns this variable will not allow groupBy() to return the full million rows as a result when this function is the last aggregator: this is governed by the QueryResultRowCountLimit dynamic configuration. 1,000,000 Dynamic configuration GroupMaxLimit All (200,000 up until version 1.126)
StateRowLimit is recommended to use as an alternative for the environment variable MAX_STATE_LIMIT. If StateRowLimit is set, then its value will be used instead of MAX_STATE_LIMIT. If it is not set, then MAX_STATE_LIMIT will be used. 20,000 or the value of StateRowLimit if it has been modified Dynamic configuration StateRowLimit All
Max number of results that any query can give. 100,000 or the value of StateRowLimit, whichever is largest Dynamic configuration QueryResultRowCountLimit All
Max memory, in bytes, a historic/static query can consume during its execution. Replacement for the environment variable MAX_MEMORY_FOR_REDUCE, so if you have changed the value of MAX_MEMORY_FOR_REDUCE, use QueryMemoryLimit instead. See also LiveQueryMemoryLimit for live queries. 100,000,000 bytes (100MB) Dynamic configuration QueryMemoryLimit Up to v. 1.115
Maximum amount of memory, in bytes, that a worker node can allocate to each historic/static query during its execution. Replacement for the environment variable MAX_MEMORY_FOR_REDUCE, so if you have changed the value of MAX_MEMORY_FOR_REDUCE, use QueryMemoryLimit instead. See also LiveQueryMemoryLimit for live queries. It cannot be configured directly: its value is set by the QueryCoordinatorMemoryLimitconfiguration (see below). 100,000,000 bytes (100MB) Dynamic configuration QueryMemoryLimit (depends on QueryCoordinatorMemoryLimit) From v. 1.116
Max memory a live query can consume during its execution. LiveQueryMemoryLimit determines how much memory in bytes a live query can consume during its execution. For non-live queries, their memory limit is determined by the QueryMemoryLimit, which is 100MB by default. By default LiveQueryMemoryLimit has the same value as QueryMemoryLimit. 100,000,000 bytes (100MB) Dynamic configuration LiveQueryMemoryLimit Up to v. 1.115
Maximum amount of memory, in bytes, that a worker node can allocate to each live query during its execution. It cannot be configured directly: its value is set by the QueryCoordinatorMemoryLimitconfiguration (see below). 100,000,000 bytes (100MB) Dynamic configuration LiveQueryMemoryLimit (depends on QueryCoordinatorMemoryLimit) From v. 1.116
Maximum memory usage that the query coordinator node can allocate during the execution of a query. This memory limit will, in turn, detemine the limits of the static query state size (QueryMemoryLimit) and the live query state size (LiveQueryMemoryLimit), which are are computed based on the value of the QueryCoordinatorMemoryLimit configuration. The memory limits for static and live queries will be computed to 1/4 of the memory limit of the query coordinator. 4,000,000,000 bytes (4GB) Dynamic configuration QueryCoordinatorMemoryLimit From v.1.116
Limits for GraphQL queries on the total number of selected fields and fragments. 1000 for authenticated and 150 for unauthenticated users. Dynamic configurations GraphQLSelectionSizeLimit and UnauthenticatedGraphQLSelectionSizeLimit Fom v1.121