Parse events encoded with key/value pairs into individual fields and values. This function can run an extra key/value parser on events.

ParameterTypeRequiredDefaultDescription
asstringoptional[a]  Prefix for all resolved field keys.
excludeEmptyarray of stringsoptional[a]false If the value of a key is empty, exclude the field.
field[b]Array of stringsoptional[a]@rawstring Fields that should be key-value parsed.
overridebooleanoptional[a]false Override existing values for keys that already exist in the event.
separatorstringoptional[a]= The token that separates the key from the value — a single char only.
separatorPaddingstringoptional[a]unknown Help the function recognize unquoted empty values and parse them by specifying whether there is a whitespace around the key-value separator (typically =). For a list of interpretations, see kvParse() separatorPadding argument handling. (introduced in 1.34.0)
  Valid Values
   noAssumes the source does not have a whitespace around the key-value separator, as in a=1, b=2
   unknownWhether the source has a padding (whitespace) around values is not recognized.
   yesIndicates the source has a whitespace around the key-value separator, as in a = 1, b = 2

[a] Optional parameters use their default value unless explicitly set

[b] The argument name field can be omitted.

Omitted Argument Names

The argument name for field can be omitted; the following forms of this function are equivalent:

logscale
kvParse("@rawstring")

and:

logscale
kvParse(field="@rawstring")

It is used to parse key/values of the form:

  • key=value

  • key="value"

  • key='value'

  • key = value

Both key and value can be either quoted using " or ', or unquoted. If using quotes, the quotes must be terminated.

Individual key/value pairs can be separated and/or embedded:

  • ,

    For example:

    logscale
    Shape=Square, Color=Red

  • ;

    For example:

    logscale
    Shape=Square;Color=Red

  • Enclosed in square brackets, starting with [

    For example:

    logscale
    [Shape=Square;Color=Red]

    Or including quoting:

    logscale
    ["Shape=Square;Color=Red"]

  • Enclosed in braces (curly brackets), starting with {

    For example:

    logscale
    {Shape=Square;Color=Red}

    Or including quoting:

    logscale
    {"Shape=Square;Color=Red"}

For a log line like this:

ini
2017-02-22T13:14:01.917+0000 [main thread] INFO UserService - creating new user id=123, name='john doe' email=john@doe

The key/value parser extracts the fields:

  • id=123

  • name=john doe

  • email=john@doe

Use the parameter field to specify which fields should be key/value parsed. Specifying @rawstring to key/value parse the rawstring.

kvParse() Examples

  • Key/value parse the log line:

ini
creating new user id=123, name='john doe' email=john@doe.

This will add the fields id=123, name='john doe' and email=john@doe to the event:

logscale
kvParse()

  • Key/value parse the log line:

ini
creating new user id=123, name='john doe' email=john@doe loglevel=ERROR.

Assuming the event already has a loglevel field, replacing the value of that field with ERROR requires parameter override=true:

logscale
kvParse(override=true)

  • Key/value parse a nested field. In this example we will use JSON input:

json
{
  "service": "paymentService",
  "type": "payment",
  "metadata": "host=server5,transactionID=123,processingTime=100"
}

and parse out the key/values in the metadata field:

logscale
parseJson()
| kvParse(metadata)

  • Key/value parse the log line and export fields with a prefix:

ini
creating new user id=123, name='john doe' email=john@doe.

This will add the fields user.id=123, user.name='john doe' and user.email=john@doe to the event:

logscale
kvParse(as="user")

  • Key/value parse the log line:

ini
firstname = John middlename = lastname = Doe

This will add the fields firstname=John, middleName= (empty value) and lastname=Doe to the event with a whitespace around the key-value separator (=):

logscale
kvParse(separatorPadding="yes")

  • When parsing a key/value line, the impact of spacing between the key, value and equals sign can lead to interpretation differences. The separatorPadding parameter controls this by defining how different patterns are interpreted with and without spacing, as follows:

    Raw Data separatorPadding Field a Value Field b Value Notes
    a = b = c unknown b   c dropped
    a = b = c yes (Empty) c  
    a = b = c no (Empty) (Empty) c dropped
    a=b=c unknown b=c   
    a=b=c yes b=c   
    a=b=c no b=c   
    a = b=c unknown b=c   
    a = b=c yes b=c   
    a = b=c no (Empty) c c dropped
    a=b = c unknown b   
    a=b = c yes (Empty) (Empty) c dropped
    a=b = c no b   

  • Keep the first value for duplicated keys.

    ini
    name='john doe' name='jane doe'

    By default kvParse() will keep the last seen value. To keep the first value instead, set as follows:

    logscale
    kvParse(onDuplicate=keepFirst)
    name
    john doe

  • Keep the first value for duplicated keys, with a preset field: if name is set to alice beforehand, then onDuplicate=keepFirst parameter has no effect and name will keep such preset value. If you want to ignore the preset value, use the override parameter.

    ini
    name='john doe' name='jane doe'
    logscale
    kvParse(onDuplicate=keepFirst, override=true)
    name
    john doe

  • Keep the last value for duplicated keys, with a preset field: if name is set to alice beforehand, then the onDuplicate=Last parameter has no effect and name will keep such preset value. If you want to ignore the preset value, use the override parameter.

    ini
    name='john doe' name='jane doe'
    logscale
    kvParse(onDuplicate=keepLast, override=true)
    name
    jane doe