Supported File Types and Formats

LogScale supports two different file formats for uploaded lookup files: JSON and CSV.

  CSV Files JSON Files
Viewable within LogScale UI Yes No
Appendable within LogScale Yes No
Editable within LogScale UI Yes No
File Size Limit

Variable default: 2,147,483,639 bytes (2048 MB)

(MaxCsvFileUploadSizeBytes)

Variable default: 2,147,483,647 bytes (2048 MB)

(MaxJsonFileUploadSizeBytes)

Each file format has some specific rules that are described in the section for each file format type. For both formats, the following apply:

  • Individual keys and values should be quoted, even as a number.

Important

Nested objects, that is an object within the returned object, are not supported. For example:

Invalid Example for Demonstration - DO NOT USE
json
{
  "1": { "name": "chr", "roles": { "user" : true }},
  "2": { "name": "krab" },
  "4": { "name": "pmm" },
  "7": { "name": "mgr" }
}

would return only the simple field, name when used with match(); the remainder of the embedded object would be not be returned or included in the events. LogScale does not reject files in this format.

Lookup files using CSV format

When using CSV for lookup files, the following rules apply:

  • Individual fields should be separated by a comma (,)

  • Whitespace is always included in the imported fields, the input takes the literal contents split by the comma character.

  • Fields can optionally be quoted by double quotes, for example to include commas in the imported values.

  • The first line of the CSV is interpreted as the column header and can be used as the field name when looking up values with functions like match().

For example, the CSV file:

csv
number,code,description
17,udp,UDP
80,http,HTTP Service
ip,"Internet Protocol, pseudo protocol"

Would be interpreted as:

numbercodedescription
17udpUDP
80httpHTTP Service
ipInternet Protocol, pseudo protocol 

CSV files can be viewed within the Lookup files interface to confirm how the information has been interpreted.

Lookup files using JSON format

When using JSON files, two different formats are supported: object-based and array-based.

JSON must be formatted in strict notation format. This requires no trailing commas (where there is no additional value),

Important

Once uploaded, JSON files cannot be viewed or updated. They can be exported to confirm the file format.

  • Object-based

    In the object-based format, format the JSON as a hash or associative array, with a single key and corresponding object. For example:

    json
    {
  "1": { "name": "chr" },
  "2": { "name": "krab" },
  "4": { "name": "pmm" },
  "7": { "name": "mgr" }
}

    When performing a lookup, match() will return the object (as an event with multiple fields), based on the matching key.

  • Array-based

    In the array-based format, format the JSON as an array of objects. In this model, the keys for each individual object become fields that can be matched when performing a lookup. For example, in the file:

    json
    [
  { "userid": "1", "name": "chr" },
  { "userid": "2", "name": "krab" },
  { "userid": "4", "name": "pmm" },
  { "userid": "7", "name": "mgr" }
]

    The userid and name fields in the JSON object can be used to lookup and return other key/value pairs as event fields/values. For example, the fragment:

    logscale Syntax
    ...
| match(file="long.json",field=codename,column="name")

    would return the userid field for objects within the lookup file array.