Parse events encoded with key/value pairs into individual fields and values. This function can run an extra key/value parser on events.

ParameterTypeRequiredDefault ValueDescription
asstringoptional[a]   Prefix for all resolved field keys.
excludeEmptyarray of stringsoptional[a] false If the value of a key is empty, exclude the field.
field[b]array of stringsoptional[a] @rawstring Fields that should be key-value parsed.
onDuplicatestringoptional[a] keepLast Set the value for duplicate keys that exist in the event.
   Values
   keepFirstKeep the first duplicate value
   keepLastKeep the last duplicate value
overridebooleanoptional[a] false Override existing values for keys that already exist in the event.
separatorstringoptional[a] = The token that separates the key from the value — a single char only.
separatorPaddingstringoptional[a] unknown Help the function recognize unquoted empty values and parse them by specifying whether there is a whitespace around the key-value separator (typically =). For a list of interpretations, see kvParse() separatorPadding argument handling.
   Values
   noAssumes the source does not have a whitespace around the key-value separator, as in a=1, b=2
   unknownWhether the source has a padding (whitespace) around values is not recognized.
   yesIndicates the source has a whitespace around the key-value separator, as in a = 1, b = 2

[a] Optional parameters use their default value unless explicitly set.

[b] The parameter name field can be omitted.

Hide omitted argument names for this function

Show omitted argument names for this function

It is used to parse key/values of the form:

  • key=value

  • key="value"

  • key='value'

  • key = value

Both key and value can be either quoted using " or ', or unquoted. If using quotes, the quotes must be terminated.

For a log line like this:

ini
2017-02-22T13:14:01.917+0000 [main thread] INFO UserService - creating new user id=123, name='john doe' email=john@doe

The key/value parser extracts the fields:

  • id=123

  • name=john doe

  • email=john@doe

Use the parameter field to specify which fields should be key/value parsed. Specify @rawstring to key/value parse the rawstring.

kvParse() Syntax Examples

  • Key/value parse the log line:

ini
creating new user id=123, name='john doe' email=john@doe.

This will add the fields id=123, name='john doe' and email=john@doe to the event:

logscale
kvParse()

  • Key/value parse the log line:

ini
creating new user id=123, name='john doe' email=john@doe loglevel=ERROR.

Assuming the event already has a loglevel field, replacing the value of that field with ERROR requires parameter override=true:

logscale
kvParse(override=true)

  • Key/value parse a nested field. In this example we will use JSON input:

json
{
  "service": "paymentService",
  "type": "payment",
  "metadata": "host=server5,transactionID=123,processingTime=100"
}

and parse out the key/values in the metadata field:

logscale
parseJson()
| kvParse(metadata)

  • Key/value parse the log line and export fields with a prefix:

ini
creating new user id=123, name='john doe' email=john@doe.

The following query will add the fields user.id=123, user.name='john doe' and user.email=john@doe to the event:

logscale
kvParse(as="user")

  • Key/value parse the log line:

ini
firstname = John middlename = lastname = Doe

This will add the fields firstname=John, middleName= (empty value) and lastname=Doe to the event with a whitespace around the key-value separator (=):

logscale
kvParse(separatorPadding="yes")

  • When parsing a key/value line, the impact of spacing between the key, value and equals sign can lead to interpretation differences. The separatorPadding parameter controls this by defining how different patterns are interpreted with and without spacing, as follows:

    Raw Data separatorPadding Field a Value Field b Value Notes
    a = b = c unknown b   c dropped
    a = b = c yes (Empty) c  
    a = b = c no (Empty) (Empty) c dropped
    a=b=c unknown b=c   
    a=b=c yes b=c   
    a=b=c no b=c   
    a = b=c unknown b=c   
    a = b=c yes b=c   
    a = b=c no (Empty) c c dropped
    a=b = c unknown b   
    a=b = c yes (Empty) (Empty) c dropped
    a=b = c no b   

  • Keep the first value for duplicated keys.

    ini
    name='john doe' name='jane doe'

    By default kvParse() will keep the last seen value. To keep the first value instead, set as follows:

    logscale
    kvParse(onDuplicate=keepFirst)
    name
    john doe

  • Keep the first value for duplicated keys, with a preset field: if name is set to alice beforehand, then onDuplicate=keepFirst parameter has no effect and name will keep such preset value. If you want to ignore the preset value, use the override parameter.

    ini
    name='john doe' name='jane doe'
    logscale
    kvParse(onDuplicate=keepFirst, override=true)
    name
    john doe

  • Keep the last value for duplicated keys, with a preset field: if name is set to alice beforehand, then the onDuplicate=Last parameter has no effect and name will keep such preset value. If you want to ignore the preset value, use the override parameter.

    ini
    name='john doe' name='jane doe'
    logscale
    kvParse(onDuplicate=keepLast, override=true)
    name
    jane doe

kvParse() Examples

Click + next to an example below to get the full details.

Parse String as CSV

Parse a CSV-encoded field into known columns using parseCsv() function

Query
logscale
kvParse()
| parseCsv(result, columns=[count, status,
          completion, precision, sourcetask])
Introduction

The parseCsv() function can be used to parse a CSV-encoded field into known columns.

Example incoming data might look like this:

Raw Events
2017-02-22T13:14:01.917+0000 [main thread] INFO statsModule got result="117 ,success ,27% ,3.14"

Step-by-Step

  1. Starting with the source repository events.

  2. logscale
    kvParse()

    Parses the string into key value pairs.

  3. logscale
    | parseCsv(result, columns=[count, status,
          completion, precision, sourcetask])

    CSV parses the result field from a log line (extracted by the kvParse() function) and adds the following fields to the event: count with the value 117, status with the value success, completion with the value 27%, and precision with the value 3.14.

  4. Event Result set.

Summary and Results

The query is used to parse a string as CSV.

Sample output from the incoming example data:

completioncountprecisionresultstatus
27% 117 3.14 117 ,success ,27% ,3.14 success