Converting Data

LogScale only has two internal data types, a string and a number, all other types within LogScale are interpreted or converted at the time of use according to the function or operation being performed.

Even when using more complex types, such as an IP address, the underlying value is a string which is then converted or translated by a function that knows how to interpret the value. This means that internally values are not stored in their specific format, just in their string representation. This allows for some flexibility when manipulating and converting data. For example, values can be added together and then converted or translated.

This also applies to the way arrays are interpreted and handled. See Array Operations for some examples.

However, there will be times when you need to convert various types into different formats, or be more explicit about identifying and interpreting a value:

Converting Numbers

LogScale will automatically convert a string into a number when used in a numerical context. For example:

logscale
myValue := "69"
| multiValue := myValue * 2

Sets the field multiValue to 138 without an explicit conversion.

This process also works for floating point values:

logscale
floatValue := "1.234E45"
| format(field=floatValue,format="%0.2f")

Converting Hex, Octal, Binary Numbers

The parseInt() function can convert numbers that are represented in other base-numbering systems than base 10 (decimal). For example, to convert a binary number:

logscale
hexValue := 0xFF
| parseInt(hexValue,radix=16)

The radix parameter controls the base, so the following also work:

logscale
binValue := "01001010"
| parseInt(binValue,radix=2)

Or octal:

logscale
octValue := "644"
| parseInt(octValue,radix=8)

To convert a number to a hex, binary or other value, use the format() function. For example, to convert an integer to hex:

logscale
decimal := 49857
| format(format="%x",field=decimal)

Will create a field, _format with the value c2c1.

Converting and Parsing Encoded Data

LogScale includes a number of functions that convert or extract information from differnt encoded data formats:

  • Hex-encoded strings

    Use the parsHexString() function. This translates multiple characters from a hex-encoded string into it's UTF-8 equivalent:

    logscale
    hex := "0x4 865 6c6c6f576f726c6420 plus F 0 9 F 9 8 8 0"
| text := parseHexString(hex)

  • To convert CEF and LEEF formatted data use the corresponding parseCEF() and parseLEEF() functions. These are specialized functions designed to extract the data from these binary formats.

  • base64Decode()

    To convert a Base64 encoded string:

    logscale
    decoded := base64Decode(encoded)

  • Fixed width encoded strings

    The text:substring() function can extract data based on the index of the substrings. So with a fixed width string:

    logscale
    message := "JOHN      DOE       123 MAIN ST              ANYTOWN    CA90210555-123-4567ACTIVE    " as example

    Could be extracted with multiple calls:

    logscale
    firstName := text:substring(message, begin=0, end=10)
| lastName := text:substring(message, begin=10, end=20)
| address := text:substring(message, begin=20, end=45)
| city := text:substring(message, begin=45, end=55)
| stateZip := text:substring(message, begin=55, end=62)
| phone := text:substring(message, begin=62, end=74)
| status := text:substring(message, begin=74, end=84)

Converting and Parsing JSON or Formats

The parseJson() function will convert JSON and parseXml() XML data into

Structured data, including data that might have been extracted or converted from JSON or XML data. When the data is processed and converted, the information, the data is converted to a compound field name. For example, the JSON structure:

json
{
  "a" : {
  "b" : [ "c", "d"]
  }
  }

Is converted into a series of fields in the form #.#:

a.b[0]c
a.b[1]d

The top level a key becomes the first element of the compound field name, the b the second, etc.

This can create confusion and complexity when processing later. Ensure you use the appropriate array (array:* or nested array (ojbectArray:*) function.

Converting and Parsing XML

Converting XML is similar to the conversion for JSON; XML tags are converted to elements of a compound field name, and nested XML is translated. For example:

xml
<parent>
 <child>88</child>
 <child>105</child>
 <cousin type="niece">Sarah</cousin>
 <cousin type="nephew">Simon</cousin>
</parent>

Is translated into the following fields:

Raw Events
"parent.child","parent.cousin","parent.cousin._type","parent.cousin[0]","parent.cousin[0]._type","parent.cousin[1]","parent.cousin[1]._type"
"88","Sarah","niece","Sarah","niece","Simon","nephew"

When translating:

  • XML tags are translated into the names of a compound field

  • Repeated XML tags are translated into arrays of a compound field

  • Nested XML tags are translated into additioanl elements of a compound field

  • Attributes are translated into underscore-prefixe elements of a compound field

Converting and Manipulating URLs

There are two primary functions for manipulating URLs (and URIs), but they operate slightly differently:

  • parseUrl() translates a given full URL into it's consistuent parts (hostname, username, path etc)

  • parseUri() translates a full or partial URI with an optional scheme prefix (i.e. FTP, HTTP etc)

With a full URL, parsing will generate each each:

logscale
myurl:="https://crowdstrike.com/training"
| parseUrl(field=myurl)

Extracts into separate fields:

myurlmyurl.pathmyurl.hostmyurl.scheme
https://crowdstrike.com/training/trainingcrowdstrike.comhttps

However, if parsing a partial URL then parseUrl() returns results based on a best guess of the content:

logscale
myurl:="crowdstrike.com/training"
| parseUrl(field=myurl)

Extracts fields but has no way to determine the right elements:

myurlmyurl.pathmyurl.hostmyurl.scheme
crowdstrike.com/trainingcrowdstrike.com/training<no value><no value>

Using parseUri() we can provide a hint of the protocol and extract the right components:

logscale
myurl:="crowdstrike.com/training"
| parseUri(field=myurl,defaultBase="https://")

Which returns correct fields by applying the default scheme to the partial URL:

myurlmyurl.pathmyurl.hostmyurl.scheme
https://crowdstrike.com/training/trainingcrowdstrike.comhttps

For decoding URL encoded parameter values use urlDecode() to translate the URL elemnts from the escape (%FF). A corresponding urlEncode() can be used for encoding URL and parameters.