Limits & Standards
This page lists the various limits and standard operating parameters of LogScale. See Best Practice for the best practices relative to ingest via the Ingest API.
General Limits & Parameters
Below is a list of general limits and parameters of LogScale:
Note
Unless otherwise specified, all multiple-bye data sizes in LogScale are expressed in SI units using decimal (Base 10). For example:
1KB = 1,000 Bytes
1MB = 1,000,000 Bytes
1GB = 1,000,000,000 Bytes
1TB = 1,000,000,000,000 Bytes
| Description | Default Limit | Controlling Variable | LogScale version |
|---|---|---|---|
| Character set support | UTF-16 | All | |
|
Max number of fields in an event
During ingest, fields are sorted alphabetically by name and the first 1000 fields are parsed, the remainder of the named fields are dropped. Please note that the @rawstring is not modified and will contain all data. | 1,000 |
MAX_EVENT_FIELD_COUNT
| All |
|
Max event size
When the configured event size max is reached, either in
@rawstring and/or in other fields, the
overall data will be truncated. Fields will be removed entirely,
and @rawstring will be truncated down to
the allowed max size with added | 1 MB |
MAX_EVENT_SIZE
| All |
| Max tag-key size (not configurable) (see Tag Fields) | 65535 | All | |
| Max tag-value size (not configurable) (see Tag Fields) | 65535 | All | |
| Max JSON file size (see Lookup Files) | 100 MB |
MAX_FILEUPLOAD_SIZE
| All |
| Max CSV lookup file size (see Lookup Files) | 200 MB |
MaxCsvFileUploadSizeBytes dynamic configuration. If
MAX_FILEUPLOAD_SIZE environment variable is set,
its value will be the default.
| From 1.124 |
| Max JSON lookup file upload size | 100 MB |
MaxJsonFileUploadSizeBytes dynamic configuration.
If MAX_FILEUPLOAD_SIZE environment variable is set,
its value will be the default.
| From 1.124 |
| Max query length in characters | 66,000 | All | |
Memory limit for the mapper phase of a
collect() function running as a top-level
function i.e., how much data such a function can store.
| 10 MiB | From v1.79 | |
Memory limit for the mapper phase of a
collect() function running in a subquery, or
as a subaggregator to another function i.e., how much data such a
function can store.
| 1 MiB | From v1.79 | |
Max number of events in tail(),
head(), and sort()
functions.
| 20,000 |
StateRowLimit
| All |
Max number of events in an RDNS request. See
rdns() function.
| — |
StateRowLimit
| Up to v1.136 |
RdnsMaxLimit
| From v1.137 | ||
Default number of events set in an RDNS request. See
rdns() function.
| — |
RdnsDefaultLimit
| From v1.137 |
Max number of rows that join() and
selfJoin() can return.
|
200,000 or the value of MAX_JOIN_LIMIT, if it has
been modified
|
Dynamic configuration JoinRowLimit, replacement for
MAX_JOIN_LIMIT environment variable.
| All |
Default value for the limit parameter in
groupBy(), selfJoin()
and some other functions, when not specified.
GroupDefaultLimit has a default value of either
20,000 or the value of the config GroupMaxLimit,
whichever is smallest. GroupDefaultLimit cannot be
larger than GroupMaxLimit.
| 20,000 |
Dynamic configuration GroupDefaultLimit
| All |
Max value for limit parameter in the
groupBy() function, meaning that the function
can collect up to a million groups. Due to stability concerns this
variable will not allow groupBy() to return
the full million rows as a result when this function is the last
aggregator: this is governed by the
QueryResultRowCountLimit dynamic configuration.
| 1,000,000 |
Dynamic configuration GroupMaxLimit
| All (200,000 up until version 1.126) |
| Maximum number of events processed in a given function |
20,000 or the value of StateRowLimit if it has been
modified
|
Dynamic configuration StateRowLimit
| All |
| Max number of results that any query can give. |
100,000 or the value of StateRowLimit, whichever is
largest
|
Dynamic configuration QueryResultRowCountLimit
| All |
Maximum amount of memory, in bytes, a historic/static query can
consume during its execution. See also
LiveQueryMemoryLimit for live queries.
| 100,000,000 bytes (100MB) |
Dynamic configuration QueryMemoryLimit
| Up to v. 1.115 |
Maximum amount of memory, in bytes, that a
worker node can allocate to
each historic/static query during its execution. It cannot be
configured directly: its value is set by the
QueryCoordinatorMemoryLimitconfiguration (see
below). See also LiveQueryMemoryLimit for live
queries.
| 1,000,000,000 bytes (1GB) |
Dynamic configuration QueryMemoryLimit (depends on
QueryCoordinatorMemoryLimit)
| From v. 1.116 |
Max memory a live query can consume during its execution.
LiveQueryMemoryLimit determines how much memory in
bytes a live query can consume during its execution. For non-live
queries, their memory limit is determined by the
QueryMemoryLimit, which is 100MB by default. By
default LiveQueryMemoryLimit has the same value as
QueryMemoryLimit.
| 100,000,000 bytes (100MB) |
Dynamic configuration LiveQueryMemoryLimit
| Up to v. 1.115 |
Maximum amount of memory, in bytes, that a
worker node can allocate to
each live query during its execution. It cannot be configured
directly: its value is set by the
QueryCoordinatorMemoryLimit configuration (see
below).
| 1,000,000,000 bytes (1GB) |
Dynamic configuration LiveQueryMemoryLimit (depends
on QueryCoordinatorMemoryLimit)
| From v. 1.116 |
Maximum memory usage that the query
coordinator node can allocate during the execution of a
query. This memory limit will, in turn, detemine the limits of the
static query state size (QueryMemoryLimit) and the
live query state size (LiveQueryMemoryLimit), which
are computed based on the value of the
QueryCoordinatorMemoryLimit configuration. The
memory limits for static and live queries will be computed to 1/4
of the memory limit of the query coordinator. The limits of the
static query state size and the live query state size are the
state sizes on the workers.
| 4,000,000,000 bytes (4GB) |
Dynamic configuration QueryCoordinatorMemoryLimit
| From v.1.116 |
| Limits for GraphQL queries on the total number of selected fields and fragments. | 1000 for authenticated and 150 for unauthenticated users. |
Dynamic configurations GraphQLSelectionSizeLimit
and UnauthenticatedGraphQLSelectionSizeLimit
| Fom v1.121 |