Custom Parsers

Security Requirements and Controls

A parser consists of a script and parser settings like Event Tags and Fields to Remove. The parser script, written in the LogScale Query Language, defines how a single incoming event is transformed before it becomes one (or more) searchable event.

The goal for a parser script is to:

  • Extract the correct timestamp from the event

  • Set the fields you want to use frequently in your searches

The following diagram provides an overview of where parsers fit in the configuration flow to ingest data using LogScale.

graph LR; A["Install and Configure LogScale"]--> B B["Create a Repository"]--> C C["Configure Data Ingest"]--> D D["Parse and Filter Data"]--> E E["Enrich Data"]--> F F["Query Data"] style D fill:#A6A0D2

Figure 60. Flow


The main text of the ingested event is present in the field @rawstring, and many functions used for parsing will default to using @rawstring if no field is specified, so a parser may easily parse the incoming data without ever referring explicitly to @rawstring in the script.

Other fields may also be present though, depending on how logs are sent to LogScale. For example, Falcon LogScale Collector will add a few fields such as @collect.timestamp which are present and usable in the parser. In other words, an input event for a parser is really a collection of key-value pairs. The main key is @rawstring, but others can be present from the beginning as well, and the parser can use those as it would do with any other fields.

The contents of @rawstring can also be any kind of text value. It's common to see e.g. JSON objects or single log lines;@rawstring doesn't require any specific format, and you can send whatever data you like.

Setting the correct timestamp is important, as LogScale relies on this field to find the right results when you search in a given time interval. You do this by assigning the timestamp to the @timestamp field, formatted as a UNIX timestamp. Functions such as parseTimestamp() are designed to make this easy. For more information, see Parsing Timestamps.

Setting fields you want to search for in the parser is optional, though highly recommended. That's because fields can also be extracted at search time, so the parser does not need to meticulously set every field you might want to use. However, searching on fields which have been set by the parser is generally easier, in terms of writing queries, and also performs better, in terms of search speed.

LogScale has built-in parsers for common log formats like accesslog, see Built-in Parsers for more information.

LogScale also offers a list of preconfigured parsers.

If you have checked both and found that nothing quite matches what you need, then this guide will show you how to create your own parser (or edit an existing one).