Parses data or a field as JSON, converting the data into named fields and arrays within the event. Use parseJson(field=@rawstring) to parse the original ingested rawstring into JSON. Use the prefix parameter to prefix the names of the extracted fields.

Furthermore, to exclude some of the extracted fields, use the exclude parameter to specify the JSON object structure to be excluded. For example, use parseJson(field=input,exclude=a.b.c) to exclude c and all of its descendants (a path-based exclusion) or specify parseJson(field=input,exclude="a.b[*].c") to exclude all c inside the array b (array-based exclusion). Note that all non-nested fields are also excluded.

Use the include parameter if you need to keep certain descendants of an otherwise excluded path or an excluded non-nested field in the output.

ParameterTypeRequiredDefault ValueDescription
excludearray of stringsoptional[a] [] Removes specified JSON fields from the output. Use on fields that should be excluded from the result. Supports dot-pathing and array wildcards. If used with prefix, the excluded fields will use the specified prefix. The exclusion works as a wildcard match for the given field; for example, the value query will match both nested fields (like query.string) and similarly non-nested named fields (like queryString, queryStart, and queryEnd). To retain specific fields while excluding others, use the include parameter.
excludeEmptybooleanoptional[a] false Whether to exclude if the field is empty.
   Values
   falseDo not exclude the field, even if the value is empty
   trueExclude the field if the value is empty
field[b]stringrequired @rawstring Fields that should be parsed as JSON.
handleNullstringoptional[a] keep How null values are handled.
   Values
   discardDiscard the null value and field null value with an empty string ""
   emptyReplaces a null value with an empty string ""
   keepConverts the value to the "null" string
includearray of stringsoptional[a] [] Retains specific descendants of excluded paths. Use on fields that should be included, even if they had been previously excluded by use of exclude. Supports dot-pathing and array wildcards. If used with prefix, the include fields will also use the specified prefix.
prefixstringoptional[a] blank Prefix the name of the extracted JSON fields with the value of this parameter.
removePrefixesarray of stringsoptional[a] [] Prefixes that should be removed from the names of the extracted JSON fields; supports dot-pathing. If multiple prefixes are supplied, the longest matching prefix will be used.

[a] Optional parameters use their default value unless explicitly set.

[b] The parameter name field can be omitted.

Hide omitted argument names for this function

Show omitted argument names for this function

When parsing JSON, the following apply:

  • When excludeEmpty=true is used, the key/value pairs will be discarded completely whenever the original json contains foo: ""

  • When the JSON contains foo: null, using handleNull=discard then the entire key/value pair is discarded, regardless of the setting for excludeEmpty

parseJson() Syntax Examples

If the whole event sent to LogScale is JSON like:

javascript
{"service": "userService", "timestamp": "2017-12-18T20:39:35Z", "msg": "user with id=47 logged in"}
logscale
parseJson()
| parseTimestamp(field=timestamp)

If a field in the incoming event contains JSON like:

ini
2017-12-18T20:39:35Z user id=47 logged in details="{"name": "Peter", "email": "peter@test.com", "id":47}"

In the example below the details field is extracted using the kvParse() function and then parseJson is used to parse the JSON inside the details field.

logscale
/(?<timestamp>\S+)/
| parseTimestamp(field=timestamp)
| kvParse()
| parseJson(field=details)

It is possible to prefix names of the extracted JSON fields. This can be useful for avoiding collisions with existing fields with the same name. For example the input line:

logscale
details="{"email": "foo@test.com", "name": "Peter"}"

Could be parsed into these fields: user.email=foo@test.com, user.name=Peter.

logscale
kvParse()
| parseJson(field=details, prefix="user.")

It is possible to remove prefixes as well. For example the input line:

logscale
details="{"a": { "b": { "c": { "d": "e", "f": "g"}, "h": "i" }, "j": "k" } }"

If the JSON object is expanded:

json
{
   "a" : {
      "b" : {
         "c" : {
            "d" : "e",
            "f" : "g"
         },
         "h" : "i"
      },
      "j" : "k"
   }
}

Would be parsed into these fields with the following function if the a prefix is removed: b.c.d=e, b.c.f=g, b.h=i, j=k.

logscale
kvParse()
| parseJson(field=details, removePrefixes=a.)

It is possible to exclude extracted fields. This can be useful for removing sensitive data or, for example, large arrays. For example, the input line:

logscale
details="{"a": { "b": { "c": { "d": "e", "f": "g"}, "h": "i" }, "j": "k" } }"

If the raw JSON is formatted:

json
{
   "a" : {
      "b" : {
         "c" : {
            "d" : "e",
            "f" : "g"
         },
         "h" : "i"
      },
      "j" : "k"
   }
}

The JSON string would be parsed into these fields: a.b.h=i, a.j=k but not, for example, a.b.c.d=e

logscale
kvParse()
| parseJson(field=details, exclude=a.b.c)

It is also possible to exclude extracted fields within arrays. For example the input line:

logscale
details="{"a": { "b": [{ "c": { "d": 1 }, "e": "f" }, { "c": { "d": 2 }, "e": "h" }] } }"

As a formatted JSON string:

json
{
   "a" : {
      "b" : [
         {
            "c" : {
               "d" : 1
            },
            "e" : "f"
         },
         {
            "c" : {
               "d" : 2
            },
            "e" : "h"
         }
      ]
   }
}

Would be parsed into these fields: a.b[0].e=f, a.b[1].e=h but not, for example, a.b[0].c.d=1.

logscale
kvParse()
| parseJson(field=details, exclude="a.b[*].c")

It is possible to include fields that had previously been excluded. For example the input line:

logscale
details="{"a": { "b": { "c": { "d": 1, "e": 2} } } }"

Would be parsed into these fields: a.b.c.e=2.

logscale
kvParse()
| parseJson(field=details, exclude=a.b.c, include=a.b.c.e)

If includes and excludes are used with prefix, you need to prefix the includes and excludes as well. For example the input line:

logscale
details="{"a": { "b": { "c": { "d": 1, "e": 2} } } }"

Would be parsed into these fields: x.a.b.c.e=2.

logscale
kvParse()
| parseJson(field=details, prefix=x., exclude=x.a.b.c, include=x.a.b.c.e)