Limits and Standards

This page lists the various limits and standard operating parameters of LogScale. See Best Practice for the best practices relative to ingest via the Ingest API.

General Limits and Parameters

Below is a list of general limits and parameters:

Note

Unless otherwise specified, all multiple-byte data sizes are expressed in SI units using decimal (Base 10). For example:

  • 1KB = 1,000 Bytes

  • 1MB = 1,000,000 Bytes

  • 1GB = 1,000,000,000 Bytes

  • 1TB = 1,000,000,000,000 Bytes

Data Structure

Limit NameLimitDynamic Variable ConfigurationEnvironment Variable ConfigurationAvailability
Max number of datasources in a repository.

10,000 datasources

   

Max number of fields in an event. During ingest, fields are sorted alphabetically by name and the first fields are parsed, the remainder of the named fields are dropped. The @rawstring field is not modified and will contain all data.

  

MAX_EVENT_FIELD_COUNT

Variable default: 1,000 fields (from 1.0)

 

Max event size. When the configured event size max is reached, either in @rawstring and/or in other fields, the overall data will be truncated. Fields will be removed entirely, and @rawstring will be truncated down to the allowed max size with added ... at the end, such that the of all other fields + size of @rawstring is less than the configured max event size. Only @rawstring, @timestamp and @timezone are added when truncation occurs.

1 MB

 

MAX_EVENT_SIZE

Variable default: 1 MiB

 

Max file upload size (see Lookup Files)

2,048 MB

   

Query Related Limits

Limit NameLimitDynamic Variable ConfigurationEnvironment Variable ConfigurationAvailability

Max memory a live query can consume during its execution.

LiveQueryMemoryLimit determines how much memory in bytes a live query can consume during its execution. For non-live queries, their memory limit is determined by QueryMemoryLimit, which is 100MB by default. By default LiveQueryMemoryLimit has the same value as QueryMemoryLimit.

100,000,000 bytes

LiveQueryMemoryLimit

Variable default: 100,000,000 bytes

  
Max query length in characters

66,000 characters

   

Maximum amount of memory, in bytes, that a worker node can allocate to each historic/static query during its execution.

Since version 1.116, it cannot be configured directly: its value is set by the QueryCoordinatorMemoryLimitconfiguration. See also LiveQueryMemoryLimit for live queries.

100,000,000 bytes

QueryMemoryLimit

Variable default: 100,000,000 bytes

  

Maximum amount of memory, in bytes, a historic/static query can consume during its execution.

See also LiveQueryMemoryLimit for live queries.

100,000,000 bytes

QueryMemoryLimit

Variable default: 100,000,000 bytes

  

Number of pipes that can be included in a query.

99 pipes

   

Max number of results that any query can give.

Increasing this value beyond the default limit may cause browser performance issues, including memory exhaustion or UI hanging when rendering large result sets.

StateRowLimit

  

Maximum number of events processed in a given function. It is strongly recommended to not change this limit beyond its current default value as the consequences of doing so are cluster instability or cluster crashes. If you experience a lot of query load, particularly from the sort(), table(), head(), and tail() functions, decreasing this limit can help alleviate said load, as it lowers the amount of events those functions are allowed to collect.

20,000

StateRowLimit

  

Maximum amount of memory, in bytes, that a worker node can allocate to each live query during its execution. It cannot be configured directly.

For non-live queries, their memory limit is determined by the QueryMemoryLimit. By default LiveQueryMemoryLimit has the same value as QueryMemoryLimit. From v 1.116, it cannot be configured directly: its value is set by the QueryCoordinatorMemoryLimit configuration.

100,000,000 bytes

LiveQueryMemoryLimit

Variable default: 100,000,000 bytes

  

Function Limits

Limit NameLimitDynamic Variable ConfigurationEnvironment Variable ConfigurationAvailability

Default value for the limit parameter in groupBy(), selfJoin() and some other functions, when not specified.

GroupDefaultLimit has a default value of either 20,000 or the value of the GroupMaxLimit configuration, whichever is smallest. GroupDefaultLimit cannot be larger than GroupMaxLimit.

20,000 rows

GroupDefaultLimit

Variable default: 20,000 group elements

  

Max value for the limit parameter in the groupBy() function, meaning that the function can collect up to one million groups.

Due to stability concerns this variable will not allow groupBy() to return the full million rows as a result when this function is the last aggregator: this is governed by the QueryResultRowCountLimit dynamic configuration.

200,000(from 1.0)

GroupMaxLimit

Variable default: 1,000,000 group elements

  

Max number of rows that join() and selfJoin() can return.

Dynamic configuration JoinRowLimit replaces MAX_JOIN_LIMIT environment variable. If the value of MAX_JOIN_LIMIT has been modified, that is the default.

200,000 rows

JoinRowLimit

Variable default: 200,000 rows

  

Max number of events in tail(), head(), and sort() functions.

20,000 events

StateRowLimit

  

Max number of events in an RDNS request. See rdns() function.

20,000

RdnsMaxLimit

  

API Limits

Limit NameLimitDynamic Variable ConfigurationEnvironment Variable ConfigurationAvailability
Max body size in POST requests after decompression.

32M bytes

   
Max body size in POST requests before decompression.

32M bytes

   

Trigger Limits

Limit NameLimitDynamic Variable ConfigurationEnvironment Variable ConfigurationAvailability
Maximum allowed catch up period for aggregate alert. See Aggregate alerts.

24 hours

 

AGGREGATE_ALERTS_MAX_CATCH_UP_LIMIT

 
Maximum allowed throttle period for aggregate alert. See Throttling.

24 hours

   
Maximum allowed time window for aggregate alert. See Aggregate alerts.

24 hours

   
Maximum CSV file size in Action Type: Email.

5 mb

   
Maximum number of events on which a filter alert with an email action triggers. See Filter alerts.

15 events

 

FILTER_ALERT_EMAIL_TRIGGER_LIMIT

 
Maximum number of events on which a filter alert with a non-email action triggers. See Filter alerts.

100 events

 

FILTER_ALERT_NON_EMAIL_TRIGGER_LIMIT

 
Maximum frequency a filter alert can trigger. See Filter alerts.

10 seconds

 

FILTER_ALERT_POLL_INTERVAL_SECONDS

 
Maximum catch up period for filter alert. See Filter alerts.

24 hours

 

FILTER_ALERTS_MAX_CATCH_UP_LIMIT

Variable default: 24h (24 hours)

 
Maximum time a filter alert waits for missing data query warnings to disappear. See Filter alerts.

10 minutes

 

FILTER_ALERTS_MAX_WAIT_FOR_MISSING_DATA

Variable default: 10 (10 minutes)

 
Maximum file size for lookup file in Action Type: Lookup File.

2 gb

   
Maximum throttle period for filter alerts and legacy alerts. See Aggregate alerts.

1 week

   
Maximum allowed backfill limit for a scheduled search. See Backfill Limit.

100

 

SCHEDULED_SEARCH_BACKFILL_LIMIT

Variable default: 5

 
Maximum allowed time window for a scheduled search. See Time window.

10 years

   
Maximum time a scheduled search waits for missing data query warnings to disappear.

10 minutes

 

SCHEDULED_SEARCH_MAX_WAIT_FOR_MISSING_DATA

 
Maximum number of values a throttled field can have. See Throttling.

100 values

 

ALERT_MAX_THROTTLE_FIELD_VALUES_STORED

Variable default: 100

 

Scheduled Report Limits

Standards

Below are some standards in LogScale.

Character sets

LogScale works with the JVM standard charsets. The character sets below are guaranteed to be supported in future JVM versions.

Note

These are the character sets LogScale accepts in the HTTP request and this may not necessarily be the same character sets accepted by the log shipper, such as Log Collector or another third-party log shipper, when ingesting files. When ingesting files, they are read by a log shipper that then makes HTTP requests to LogScale, and the character set supported by the log shipper is unlikely to have anything to do with what LogScale supports in the HTTP layer.

  • US-ASCII

  • ISO-8859-1

  • UTF-8

  • UTF-16

  • UTF-16BE

  • UTF-16LE

  • UTF-32

  • UTF-32BE

  • UTF-32LE

If you have any questions about whether a character set is accepted or supported, contact LogScale Support Team.