Whether you are ingesting structured or formatted data that has already identified the fields and information, or raw text log lines where the information needs to be extracted, the role of the parser is to extract, format and augment or enrich the incoming data stream for storage.

Parsers within LogScale allow for the following operations during ingestion:

The process of parsing is one of enrichment of the data. Most log data is free text, but storing the information in a fixed field improves the ability to query and process the information during search. LogScale always stores the original raw text along with any extracted field data.

For a more detailed example, let's look at the output from an HTTP web server. Each line represents a request/response from the web server and will be turned into an event just consisting of the raw string. The raw string for the web server is typically structured in a well known format and contains information on HTTP status code, HTTP method, response time, URL, user agent etc. It is possible to create parsers that can parse a given structure. For a web server this could look like this:

47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0" 200 5316 "https://domain1.com/?p=1" "Mozilla/5.0 (Windows NT 6.1)"

The structure of the data contains a lot of information that we can parse and extract:

Creating a parser for this format will add the following fields to the LogScale event and give it structure:

This raw data will be parsed into the following fields:

Parsers are written using the LogScale Query Language (LQL). Using the same language as you use for querying enables you to use the same functions and constructs as when querying data. In addition, because you can always parse and extract information during the query process by re-examining the original @rawstring all of the principles are familiar.

Typically, a parser makes use of the following functions and syntax:

With the full suite of LQL tools available, you can also perform enrichment of the data. In the above example, it is possible to make the parser geocode the client ip address, enriching the event with the country/area/city of the request:

...| ipLocation(clientIP)

For more information on writing and creating parsers, see Parsing Data.

In the parser configuration it is also possible to specify which fields in the event should be tags. Tags are discussed in Tag Fields and Datasources.

LogScale organizes data into the logical store of repositories and an underlying physical structure called datasource. During ingest, the token (and sometimes the parser) will define the repository that data is stored in.

Tags are fields that define the datasources within a repository. Datasources (and their tags) are used to group data which is important for physically organizing and distributing the data and optimizing search performance. Tags are defined during ingest, either by the log shipper, or by the parser.

Tags can be static, for example a parser could specify all data parsed with this parser should have the tag #type=syslog. The parser can also specify fields on the events being parsed that are tags. Using network data as an example, events could have a protocol field that could serve as a tag. The protocol field could have different values like #protocol=dns or #protocol=http. When searching, performance would be optimized when searching using the #protocol field.

In the example below we have a set of web server hosts and we are collecting the database logs on each host. We use the fields #host and #source as tag fields. A datasource is created for each combination of tag fields and their values, and this configures how segment files are created:

Tag fields start with the # character. Tags can be used to search this specific set of data, with the performance of the query optimized because only the segments that match the given tag need to be loaded. For example this search is used to only look at the datasource for the http.log on server1:

#host=server1 #source=http.log

The datasource is the combination of these two tags, and it affects how the data is physically stored. When doing a search, if the tags are used then only those physical datasources are being scanned which can improve the performance of the search. This is described in detail in Metadata.

For more information on datasources and how they are used and managed by LogScale, see Datasources.

Avoid using tags that have a high-cardinality (i.e. have a large number of unique values), as this increases the number of combinations, memory requirements, and segment organization. Please consider this and other recommendations listed in Event Tags when defining tag fields.

LogScale requires clients to provide an Ingest Token. Ingest tokens are created in LogScale and support:

Using an Ingest Token means having a unique string for communicating a specific type of data and processing (through the parser) for that data. To limit the ingestion of data to specific hosts, create a unique Ingest Token for each host and parser configuration.

Alternatively, the parser can choose how the incoming data is parsed and processed based on a field within the source log file or data.