Datatypes in CQL

Data stored within LogScale including during parsing or querying, and in events within repositories, is stored as strings and numbers. These datatypes have an impact on querying and how the data is handled and manipulated within LogScale.

LogScale has two defined data types, string and double (floating-point number). All other data types used in LogScale are virtual data types that are parsed or interpreted by the functions at the time they are used to indicate the format of the incoming or outgoing data.

For example, the data below contains two fields:

ip_addrusername
192.168.0.1abedford

The data is an IP address, but the information is stored within LogScale as a string. It's only when the value is used within a function that the actual content of the data is interpreted. If the data format does not match the required format or style required by the function, then the operation will fail.

Because the data is stored as a string, the ip_addr field can be used in various ways depending on the function. For example, the field can be interpreted as an IP address when used with the cidr() function to validate the IP address against a subnet:

logscale
cidr(field=ip_addr,subnet=["192.168.0.0/23"])

The cidr() is expecting an IP address and interprets the ip_addr string as an IP address, validating the input when the function is called.

The flexibility of this model means that data can be manipulated (during parsing or during query) while still ensuring that functions are processing values of the correct type.

There are two internal datatypes:

  • String

    A quoted or unquoted text value. Strings are the base type for specialized interpreted datatypes such as relative time values (7days) and arrays.

    Arrays have special semantics: an array is identified by appending square brackets to the field name (fieldname[]), which tells LogScale to interpret that field as containing an array of values. The array values themselves are stored as strings, but the field naming convention controls how LogScale processes them. For more details, see Array.

  • Number

    A positive or negative number. LogScale supports both integer and floating-point values, and double-precision equivalents (long and double). Numbers (including those in strings) are converted during processing automatically.

How Types Are Interpreted in Queries

While data is stored in the two base types (string and number), CQL query syntax interprets these values in specific ways depending on context. The following principles govern how LogScale interprets values when executing queries and functions:

All datatypes used within the language and function syntax are interpreted at the point of execution. This means that the actual interpretation of a value or field is only determined when the value is used:

  • Values are interpreted at the point of execution.

  • A given value may be interpreted in different ways depending on how it is declared within the function.

  • Values may be converted when it is possible/sensible to do so. For example numbers and strings may be silently converted when possible.

The table below outlines the key interpreted datatypes, their corresponding base type, and links to a full description of both the base and interpreted types and how to use them.

Type (Interpreted) Base Type Description Examples of valid values
Aggregate Function String An aggregate function that combines values together by another field, identifier or key. Returns fewer events than the original input. groupBy(), count(as=_count)
Array String An ordered collection of values. Arrays are represented as array of type, where type can be strings, numbers, fieldnames, expressions, aggregate functions, or other arrays. Arrays are typically homogeneous, containing elements of a specific type, for example an array of strings. ["sort","class"]
Arrayname String The name of an array; an array name is always composed of the field name and a pair of square brackets. This indicates to LogScale that the field should be interpreted as an entire array. array[], "a[]"
Array of aggregate functions String An array of aggregate functions. Used when multiple aggregations are required over the same set of data efficiently. [count(),sum()]
Array of arrays of strings String Also known as a multidimensional array. [["fielda","newfieldy"],["fieldb","newfieldz"]]
Array of expressions String An array of expressions, typically used as filters for content as subqueries. ["x", "y", "z"], [lower(source.user.name), lower(destination.user.name)]
Array of fieldnames String An array of fieldnames. Used within the functions where a list of fields is required, during aggregation or manipulation. concat(["firstname", "initial", "lastname"], [hostname,severity]
Array of numbers String An array of number values. [3,9,4,10,10],
Array of strings String An array or list of String values. ["a.b"], ["192.0.2.0/24", "203.0.113.0/24"], @timestamp, [a,b,c]
Boolean String A value of true or false. The value must be lowercase. true, false
Double Number A floating-point number. 0.01, 0.001, 0.1, 2
Enumerated Values String A set of named constants used to identify a particular value or configuration setting. include,exclude
Expression String Represents a CQL expression that can be evaluated, or a query to be executed. onFalse, "informational", success := if(status < 500, then=if(status!=404, then=1, else=0), else=0)
Filename String Filename of a lookup file uploaded to Lookup Files. "host_names.csv"
Function String The name of function. Used in functions that make use of another function to perform an operation. array:eval(array=hosts,function=lower)
Integer Number Any positive or negative whole number with no decimal. 12, -25
Long Number A long (64-bit) integer. The value can be a positive or negative whole number. 98756
Non-aggregate function String Any other LogScale function that is not in the list of aggregate functions. concat()
Regular Expression String A regular expression. For more information on regular expressions, see Regular Expression Syntax. regex("incidents[]", regex="^Cozy Bear.*"), regex="bear$", flags="i"
Relative-time or Timepoint String A time defined using the Relative Time Syntax, consisting of a numeric value and character-based qualifier, for example 7d for 7 days. Relative time is used when defining time according to a reference point; typically the start time or execution time of the query. 7d, 3quarters, 30m
String String Strings in LogScale can either be unquoted, or quoted. "0.0.0.0", "%,.2f", arrivaltime
Timezone String The time zone name. For a list of supported timezones, see ???. timezone="Europe/Copenhagen"