Limits and Standards

This page lists the various limits and standard operating parameters of LogScale. See Best Practice for the best practices relative to ingest via the Ingest API.

General Limits and Parameters

Below is a list of general limits and parameters:

Note

Unless otherwise specified, all multiple-byte data sizes are expressed in SI units using decimal (Base 10). For example:

  • 1KB = 1,000 Bytes

  • 1MB = 1,000,000 Bytes

  • 1GB = 1,000,000,000 Bytes

  • 1TB = 1,000,000,000,000 Bytes

Data Structure

Limit NameLimitDynamic Variable ConfigurationAvailability
Max number of datasources in a repository.

10,000 datasources

  

Max number of fields in an event. During ingest, fields are sorted alphabetically by name and the first fields are parsed, the remainder of the named fields are dropped. The @rawstring field is not modified and will contain all data.

8,000

  

Max event size. When the configured event size max is reached, either in @rawstring and/or in other fields, the overall data will be truncated. Fields will be removed entirely, and @rawstring will be truncated down to the allowed max size with added ... at the end, such that the of all other fields + size of @rawstring is less than the configured max event size. Only @rawstring, @timestamp and @timezone are added when truncation occurs.

1 MB

  

Max CSV lookup file size (see Lookup Files).

200 MB

MaxCsvFileUploadSizeBytes

Variable default: 209,715,200 bytes (200 MB)

 

Max JSON lookup file upload size (see Lookup Files).

100 MB

MaxJsonFileUploadSizeBytes

Variable default: 104,857,600 bytes (100 MB)

 

Max file upload size (see Lookup Files)

2,048 MB

  

Query Related Limits

Limit NameLimitDynamic Variable ConfigurationAvailability

Max memory a live query can consume during its execution.

100,000,000 bytes

LiveQueryMemoryLimit

Variable default: 100,000,000 bytes

 
Max query length in characters

66,000 characters

  

Maximum memory usage that the query coordinator node can allocate during the execution of a query. The memory limits for static and live queries will be computed to 1/4 of the memory limit of the query coordinator. The limits of the static query state size and the live query state size are the state sizes on the workers.

This value allows cluster operators to effectively control how many concurrent queries are allowed on a system based on the hardware configuration. The lower the value, the more queries any given coordinator can run concurrently, since each query is allowed less of the available memory. The reason this then indirectly controls the memory allowed for each query state is that each coordinator node holds 4 copies of the query state / query result at any given point during the query execution.

4GB

QueryCoordinatorMemoryLimit

Variable default: 4,000,000,000 bytes

 

Maximum amount of memory, in bytes, that a worker node can allocate to each historic/static query during its execution.

1 GB

QueryMemoryLimit

Variable default: 100,000,000 bytes

 

Maximum amount of memory, in bytes, a historic/static query can consume during its execution.

100,000,000 bytes

QueryMemoryLimit

Variable default: 100,000,000 bytes

 

Number of pipes that can be included in a query.

99 pipes

  

Max number of results that any query can give.

100,000

StateRowLimit

Variable default: 200,000 rows

 

Maximum number of events processed in a given function. It is strongly recommended to not change this limit beyond its current default value as the consequences of doing so are cluster instability or cluster crashes. If you experience a lot of query load, particularly from the sort(), table(), head(), and tail() functions, decreasing this limit can help alleviate said load, as it lowers the amount of events those functions are allowed to collect.

200,000 rows

StateRowLimit

Variable default: 200,000 rows

 

Maximum amount of memory, in bytes, that a worker node can allocate to each live query during its execution. It cannot be configured directly.

1 GB

LiveQueryMemoryLimit

Variable default: 100,000,000 bytes

 

Function Limits

Limit NameLimitDynamic Variable ConfigurationAvailability

Default number of events set in an RDNS request. See rdns() function.

5,000

RdnsDefaultLimit

Variable default: 5,000 events

 

Default value for the limit parameter in groupBy(), selfJoin() and some other functions, when not specified.

20,000 rows

GroupDefaultLimit

Variable default: 20,000 group elements

 

Max value for the limit parameter in the groupBy() function, meaning that the function can collect up to one million groups.

1,000,000

GroupMaxLimit

Variable default: 1,000,000 group elements

 

Max number of rows that join() and selfJoin() can return.

200,000 rows

JoinRowLimit

Variable default: 200,000 rows

 

Max number of events in tail(), head(), and sort() functions.

20,000 events

StateRowLimit

Variable default: 200,000 rows

 

Max number of events in an RDNS request. See rdns() function.

20,000

RdnsMaxLimit

Variable default: 20,000 events

 

Memory limit for the mapper phase of a collect() function running as a top-level function, that is, how much data such a function can store.

10 MiB

  

Memory limit for the mapper phase of a collect() function running in a subquery, or as a subaggregator to another function, that is, how much data such a function can store.

1 MiB

  

API Limits

Limit NameLimitDynamic Variable ConfigurationAvailability
Limits for GraphQL queries on the total number of selected fields and fragments for authenticated users.

1,000

AuthenticatedGraphQLSelectionSizeLimit

 
Limits for GraphQL queries on the total number of selected fields and fragments for unauthenticated users.

150

UnauthenticatedGraphQLSelectionSizeLimit

Variable default: 150 queries

 
Max body size in POST requests after decompression.

32M bytes

  
Max body size in POST requests before decompression.

32M bytes

  

Trigger Limits

Limit NameLimitDynamic Variable ConfigurationAvailability
Maximum allowed catch up period for aggregate alert. See Aggregate alerts.

24 hours

  
Maximum allowed throttle period for aggregate alert. See Throttling.

24 hours

  
Maximum allowed time window for aggregate alert. See Aggregate alerts.

24 hours

  
Maximum CSV file size in Action Type: Email.

5 mb

  
Maximum number of events on which a filter alert with an email action triggers. See Filter alerts.

15 events

  
Maximum number of events on which a filter alert with a non-email action triggers. See Filter alerts.

100 events

  
Maximum frequency a filter alert can trigger. See Filter alerts.

10 seconds

  
Maximum catch up period for filter alert. See Filter alerts.

24 hours

  
Maximum time a filter alert waits for missing data query warnings to disappear. See Filter alerts.

10 minutes

  
Maximum file size for lookup file in Action Type: Lookup File.

2 gb

  
Maximum throttle period for filter alerts and legacy alerts. See Aggregate alerts.

1 week

  
Maximum allowed backfill limit for a scheduled search. See Backfill Limit.

100

  
Maximum allowed time window for a scheduled search. See Time window.

10 years

  
Maximum time a scheduled search waits for missing data query warnings to disappear.

10 minutes

  
Maximum number of values a throttled field can have. See Throttling.

100 values

  

Scheduled Report Limits

Limit NameLimitDynamic Variable ConfigurationAvailability
Maximum size of email attachment when sending scheduled report

5 mb

  
Number of times per hour a scheduled report can run

1 per hour

  
Number of scheduled reports per repository or view

20 per repo

  
Maximum number of rows the Table and Event List widgets can output

1,000 rows

  

Standards

Below are some standards in LogScale.

Character sets

LogScale works with the JVM standard charsets. The character sets below are guaranteed to be supported in future JVM versions.

Note

These are the character sets LogScale accepts in the HTTP request and this may not necessarily be the same character sets accepted by the log shipper, such as Log Collector or another third-party log shipper, when ingesting files. When ingesting files, they are read by a log shipper that then makes HTTP requests to LogScale, and the character set supported by the log shipper is unlikely to have anything to do with what LogScale supports in the HTTP layer.

  • US-ASCII

  • ISO-8859-1

  • UTF-8

  • UTF-16

  • UTF-16BE

  • UTF-16LE

  • UTF-32

  • UTF-32BE

  • UTF-32LE

If you have any questions about whether a character set is accepted or supported, contact LogScale Support Team.