Limits & Standards
This page lists the various limits and standard operating parameters of LogScale. See Ingest via API Best Practices for the best practices relative to ingest via the Ingest API.
General Limits & Parameters
Below is a list of general limits and parameters of LogScale:
Note
Unless otherwise specified, all multiple-bye data sizes in LogScale are expressed in SI units using decimal (Base 10). For example:
1KB = 1,000 Bytes
1MB = 1,000,000 Bytes
1GB = 1,000,000,000 Bytes
1TB = 1,000,000,000,000 Bytes
| Description | Default Limit | Controlling Variable | LogScale version |
|---|---|---|---|
| Character set support | UTF-16 | All | |
|
Max number of fields in an event
During ingest, fields are sorted alphabetically by name and the first 1000 fields are parsed, the remainder of the named fields are dropped. Please note that the @rawstring is not modified and will contain all data. | 1,000 |
MAX_EVENT_FIELD_COUNT
| All |
|
Max event size
When the configured event size max is reached, either in
@rawstring and/or in other fields, the
overall data will be truncated. Fields will be removed entirely,
and @rawstring will be truncated down to
the allowed max size with added | 1 MB |
MAX_EVENT_SIZE
| All |
| Max tag-key size (not configurable) (see Tag Fields) | 65535 | All | |
| Max tag-value size (not configurable) (see Tag Fields) | 65535 | All | |
| Max JSON file size (see Lookup Files) | 100 MB |
MAX_FILEUPLOAD_SIZE
| All |
| Max CSV lookup file size (see Lookup Files) | 200 MB |
MaxCsvFileUploadSizeBytes dynamic configuration. If
MAX_FILEUPLOAD_SIZE environment variable is set,
its value will be the default.
| From 1.124 |
| Max JSON lookup file upload size | 100 MB |
MaxJsonFileUploadSizeBytes dynamic configuration.
If MAX_FILEUPLOAD_SIZE environment variable is set,
its value will be the default.
| From 1.124 |
| Max query length in characters | 66,000 | All | |
Memory limit for the mapper phase of a
collect() function running as a top-level
function i.e., how much data such a function can store.
| 10 MiB | From v1.79 | |
Memory limit for the mapper phase of a
collect() function running in a subquery, or
as a subaggregator to another function i.e., how much data such a
function can store.
| 1 MiB | From v1.79 | |
Max number of events in tail(),
head(), and sort()
functions.
| 20,000 |
StateRowLimit
| All |
Max number of events in an RDNS request. See
rdns() function.
| — |
StateRowLimit
| Up to v1.136 |
RdnsMaxLimit
| From v1.137 | ||
Default number of events set in an RDNS request. See
rdns() function.
| — |
RdnsDefaultLimit
| From v1.137 |
| Max number of data sources in a repository | 10,000 |
MAX_DATASOURCES
| All |
Max number of rows that join() and
selfJoin() can return.
|
200,000 or the value of MAX_JOIN_LIMIT, if it has
been modified
|
Dynamic configuration
JoinRowLimit,
replacement for MAX_JOIN_LIMIT environment
variable.
| All |
Default value for the limit parameter in
groupBy(), selfJoin()
and some other functions, when not specified. Replacement for the
environment variable MAX_STATE_LIMIT to determine
the default value.
GroupDefaultLimit
has a default value of either 20,000 or the value of the config
GroupMaxLimit,
whichever is smallest.
GroupDefaultLimit
cannot be larger than
GroupMaxLimit.
| 20,000 |
Dynamic configuration
GroupDefaultLimit
| All |
Max value for limit parameter in the
groupBy() function, meaning that the function
can collect up to a million groups. Previously, this parameter was
bounded by MAX_STATE_SIZE, so if you have made any
modifications to this variable, set
GroupMaxLimit
to the same value for a seamless upgrade. Due to stability
concerns this variable will not allow
groupBy() to return the full million rows as
a result when this function is the last aggregator: this is
governed by the
QueryResultRowCountLimit
dynamic configuration.
| 1,000,000 |
Dynamic configuration
GroupMaxLimit
| All (200,000 up until version 1.126) |
StateRowLimit
is recommended to use as an alternative for the environment
variable MAX_STATE_LIMIT. If
StateRowLimit
is set, then its value will be used instead of
MAX_STATE_LIMIT. If it is not set, then
MAX_STATE_LIMIT will be used.
|
20,000 or the value of
StateRowLimit
if it has been modified
|
Dynamic configuration
StateRowLimit
| All |
| Max number of results that any query can give. |
100,000 or the value of
StateRowLimit,
whichever is largest
|
Dynamic configuration
QueryResultRowCountLimit
| All |
Maximum amount of memory, in bytes, a historic/static query can
consume during its execution. Replacement for the environment
variable MAX_MEMORY_FOR_REDUCE, so if you have
changed the value of MAX_MEMORY_FOR_REDUCE, use
QueryMemoryLimit
instead. See also
LiveQueryMemoryLimit
for live queries.
| 100,000,000 bytes (100MB) |
Dynamic configuration
QueryMemoryLimit
| Up to v. 1.115 |
Maximum amount of memory, in bytes, that a
worker node can allocate to
each historic/static query during its execution. It cannot be
configured directly: its value is set by the
QueryCoordinatorMemoryLimitconfiguration
(see below). See also
LiveQueryMemoryLimit
for live queries.
| 1,000,000,000 bytes (1GB) |
Dynamic configuration
QueryMemoryLimit
(depends on
QueryCoordinatorMemoryLimit)
| From v. 1.116 |
Max memory a live query can consume during its execution.
LiveQueryMemoryLimit
determines how much memory in bytes a live query can consume
during its execution. For non-live queries, their memory limit is
determined by the
QueryMemoryLimit,
which is 100MB by default. By default
LiveQueryMemoryLimit
has the same value as
QueryMemoryLimit.
| 100,000,000 bytes (100MB) |
Dynamic configuration
LiveQueryMemoryLimit
| Up to v. 1.115 |
Maximum amount of memory, in bytes, that a
worker node can allocate to
each live query during its execution. It cannot be configured
directly: its value is set by the
QueryCoordinatorMemoryLimitconfiguration
(see below).
| 1,000,000,000 bytes (1GB) |
Dynamic configuration
LiveQueryMemoryLimit
(depends on
QueryCoordinatorMemoryLimit)
| From v. 1.116 |
Maximum memory usage that the query
coordinator node can allocate during the execution of a
query. This memory limit will, in turn, detemine the limits of the
static query state size
(QueryMemoryLimit)
and the live query state size
(LiveQueryMemoryLimit),
which are computed based on the value of the
QueryCoordinatorMemoryLimit
configuration. The memory limits for static and live queries will
be computed to 1/4 of the memory limit of the query coordinator.
The limits of the static query state size and the live query state
size are the state sizes on the workers.
| 4,000,000,000 bytes (4GB) |
Dynamic configuration
QueryCoordinatorMemoryLimit
| From v.1.116 |
| Limits for GraphQL queries on the total number of selected fields and fragments. | 1000 for authenticated and 150 for unauthenticated users. |
Dynamic configurations
GraphQLSelectionSizeLimit
and
UnauthenticatedGraphQLSelectionSizeLimit
| Fom v1.121 |