Built-in Parsers

LogScale supplies built-in parsers for common log formats, including a parser for the widely-used accesslog format for web servers like Apache and Nginx.

You can examine each of the built-in parsers directly in the LogScale UI. Open the parser page and check the supported regular expression and timestamp formats. When you paste in test data, LogScale shows the result of that parsing.

When shipping data to LogScale, you want to check to see if there is a built-in parser for the logs before writing a custom parser. The built-in parsers are a good starting point when creating custom parsers.

accesslog

This parser can handle the accesslog format, which is the default log format used by Apache and Nginx. The parser also supports putting the response time at the end of the log line. If you have modified the logging of your web server, copy the built-in accesslog parser and modify it to suit your customizations.

Example Input

Raw Events
localhost - - [25/Feb/2017:21:05:16 +0100] "POST /api/v1/ingest/elastic-bulk HTTP/1.1" 200 50 "-" "Go-http-client/1.1" 0.000 848`
192.168.1.102 - - [25/Feb/2017:21:06:15 +0100] "GET /api/v1/repositories/gotoconf/queryjobs/855620e9-1d1f-4b0e-91fe-c348795e68c9 HTTP/1.1" 200 591 "referrer" "Mozilla/5.0" 0.008 995

audit-log

This parser can process audit logs in JSON format from LogScale itself.

corelight-es

This is a built-in parser that supports Corelight's Zeek sensors. Corelight sensors have default support for streaming out Zeek logs in either JSON or Elasticsearch format. LogScale can receive the streaming data in Elasticsearch format using this parser. For the JSON format, see corelight-json.

The name of the Zeek log file will become a #path tag in LogScale.

corelight-json

This is a built-in parser that supports Corelight's Zeek sensors. Corelight sensors have default support for streaming out Zeek logs in either JSON for Elasticsearch format. LogScale can receive the streaming data in JSON format using this parser and Ingest Listeners. For the Elasticsearch format, see corelight-es above.

The name of the Zeek log file will become a #path tag in LogScale.

json

This parser can process JSON data in log lines. It expects to find a JSON property called @timestamp containing an ISO 8601-formatted time string.

If you don't have control over the JSON format, you can create a Example Input.

Example Input

javascript
{
 "@timestamp": "2017-02-25T20:18:43.598+00:00",
 "loglevel": "INFO",
 "service": {
 "name": "user service",
 "time": 123
}
}

json-for-action

This parser is the default parser for the Action Type: Falcon LogScale Repository. It processes JSON data in the @rawstring field. It expects to find a JSON property called @timestamp containing a time in Unix Time in milliseconds.

kv

This parser is the key-value parser. It is the default parser, that LogScale uses if no other parser is specified. It can process standard key-value patterns in log lines. It expects the log line to contain a timestamp with a time zone within the first 128 characters (configurable, see findTimestamp()). The parser processes the rest of the line for key-value pairs.

Example Input

Raw Events
2017-02-25T20:18:43.598+0000 created a new user user="John Doe" service=user-service as a freemium user

Given the above log line, LogScale parses the fields

serviceuser
user-serviceJohn Doe

kv-generic

This parser, like the kv, is a key-value parser. It works in the same way, except that it will also parse timestamps without a timezone. Such timestamps will be assumed to be in UTC.

kv-millis

This parser, like the kv, is a key-value parser. However, it expects the timestamp to be at the start of the log line and to be in UTC time in milliseconds.

Example Input

ini
1488054417000 created a new user user="John Doe" service=user-service as a freemium user

Given the above log line, LogScale parses the fields

serviceuser
user-serviceJohn Doe

serilog-jsonformatter

This parser can process log lines written by Serilog's JsonFormatter.

Example serilog configuration:

serilog
Log.Logger = new LoggerConfiguration()
             .WriteTo.File(formatter: new JsonFormatter(renderMessage: true), path:logPath, rollingInterval: RollingInterval.Day)

Important

The required renderMessage: true part of the configuration. LogScale will display the rendered message output by Serilog instead of the raw event.

Example Input

json
{"Timestamp":"2019-01-21T13:26:25.1354930+01:00","Level":"Information","MessageTemplate":"Processed {@Position} in {Elapsed:000} ms.","RenderedMessage":"Processed { Latitude: 25, Longitude: 134 } in 034 ms.","Properties":{"Position":{"Latitude":25,"Longitude":134},"Elapsed":34,"ProcessId":"15133"},"Renderings":{"Elapsed":[{"Format":"000","Rendering":"034"}]}}

Properties output by Serilog are available within the parsed event, such as Properties.Position.Latitude from the above example input.

syslog

This parser aims to be compatible with a variety of syslog formats. This includes RFC 3164 and RFC 5424. The parser does not implement every aspect of the syslog RFCs, but is instead liberal in what it accepts.

Example Input

Raw Events
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for foo on /dev/pts/8
Raw Events
<34>1 2003-10-11T22:14:15.003Z server1.com sshd - - pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.2.
Raw Events
<34>Oct 11 22:14:15 mymachine su: 'su root' failed for foo on /dev/pts/8
Raw Events
Oct 11 22:14:15 su: 'su root' failed for foo on /dev/pts/8

If no timezone is specified, as in the last two examples, the parser defaults to UTC time. To change that, you may create a new parser by copying this parser and modifying:

logscale
timezone="UTC"

to your desired timezone.

The parser also leverages LogScale's built-in key value parser kv.

The parser syslog-utc will have better performance when the logs display this specific format.

syslog-utc

This parser can process standard lines generated by the syslog system.

The parser expects lines to starts with a timestamp, followed by the optional fields host, app, and pid. It also expects the timestamp to be in the UTC time zone. If your timestamps are in your local timezone, and that is not UTC, you will want to create a new parser by copying this parser and modifying

logscale
timezone="UTC"

to your desired timezone.

This parser also leverages LogScale's built-in key value parser kv.

Example Input

Raw Events
Feb 25 19:17:01 Myhost CRON[24886]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 25 06:35:01 Myhost CRON[24272]: (root) CMD (command -v deb-sa1 > /dev/null && deb-sa1 1 1)

zeek-json

This parser can process JSON data generated from Zeek. It is tailored to handle the output generated from the Zeek script, and you can read about how to send Zeek data to LogScale on the Zeek (Bro) Network Security Monitor documentation page.

The name of the Zeek log file will become a #path tag in LogScale.