Built-in Parsers
LogScale supplies built-in parsers for common log formats, including a parser for the widely-used accesslog format for web servers like Apache and Nginx.
You can examine each of the built-in parsers directly in the LogScale UI. Open the parser page and check the supported regular expression and timestamp formats. When you paste in test data, LogScale shows the result of that parsing.
When shipping data to LogScale, you want to check to see if there is a built-in parser for the logs before writing a custom parser. The built-in parsers are a good starting point when creating custom parsers.
accesslog
This parser can handle the accesslog format, which is the default log format used by Apache and Nginx. The parser also supports putting the response time at the end of the log line. If you have modified the logging of your web server, copy the built-in accesslog parser and modify it to suit your customizations.
Example Input
localhost - - [25/Feb/2017:21:05:16 +0100] "POST /api/v1/ingest/elastic-bulk HTTP/1.1" 200 50 "-" "Go-http-client/1.1" 0.000 848`
192.168.1.102 - - [25/Feb/2017:21:06:15 +0100] "GET /api/v1/repositories/gotoconf/queryjobs/855620e9-1d1f-4b0e-91fe-c348795e68c9 HTTP/1.1" 200 591 "referrer" "Mozilla/5.0" 0.008 995
audit-log
This parser can process audit logs in JSON format from LogScale itself.
corelight-es
This is a built-in parser that supports Corelight's Zeek sensors. Corelight sensors have default support for streaming out Zeek logs in either JSON or Elasticsearch format. LogScale can receive the streaming data in Elasticsearch format using this parser. For the JSON format, see corelight-json.
The name of the Zeek log file will become a #path tag in LogScale.
corelight-json
This is a built-in parser that supports Corelight's Zeek sensors. Corelight sensors have default support for streaming out Zeek logs in either JSON for Elasticsearch format. LogScale can receive the streaming data in JSON format using this parser and Ingest Listeners. For the Elasticsearch format, see corelight-es above.
The name of the Zeek log file will become a #path tag in LogScale.
json
This parser can process JSON data in log lines. It expects to find a JSON property called @timestamp containing an ISO 8601-formatted time string.
If you don't have control over the JSON format, you can create a Example Input.
Example Input
{
"@timestamp": "2017-02-25T20:18:43.598+00:00",
"loglevel": "INFO",
"service": {
"name": "user service",
"time": 123
}
}
json-for-action
This parser is the default parser for the Action Type: Falcon LogScale Repository. It processes JSON data in the @rawstring field. It expects to find a JSON property called @timestamp containing a time in Unix Time in milliseconds.
kv
This parser is the key-value parser. It is the default parser, that
LogScale uses if no other parser is specified. It can process standard
key-value patterns in log lines. It expects the log line to contain a
timestamp with a time zone within the first 128 characters
(configurable, see findTimestamp()
). The parser
processes the rest of the line for key-value pairs.
Example Input
2017-02-25T20:18:43.598+0000 created a new user user="John Doe" service=user-service as a freemium user
Given the above log line, LogScale parses the fields
user=John Doe
and
service=user-service
kv-generic
This parser, like the kv, is a key-value parser. It works in the same way, except that it will also parse timestamps without a timezone. Such timestamps will be assumed to be in UTC.
kv-millis
This parser, like the kv, is a key-value parser. However, it expects the timestamp to be at the start of the log line and to be in UTC time in milliseconds.
Example Input
1488054417000 created a new user user="John Doe" service=user-service as a freemium user
Given the above log line, LogScale parses the fields
user=John Doe
and
service=user-service
serilog-jsonformatter
This parser can process log lines written by Serilog's JsonFormatter.
Example serilog configuration:
Log.Logger = new LoggerConfiguration()
.WriteTo.File(formatter: new JsonFormatter(renderMessage: true), path:logPath, rollingInterval: RollingInterval.Day)
Important
The required renderMessage: true
part of the configuration. LogScale will display the rendered message
output by Serilog instead of the raw event.
Example Input
{"Timestamp":"2019-01-21T13:26:25.1354930+01:00","Level":"Information","MessageTemplate":"Processed {@Position} in {Elapsed:000} ms.","RenderedMessage":"Processed { Latitude: 25, Longitude: 134 } in 034 ms.","Properties":{"Position":{"Latitude":25,"Longitude":134},"Elapsed":34,"ProcessId":"15133"},"Renderings":{"Elapsed":[{"Format":"000","Rendering":"034"}]}}
Properties output by Serilog are available within the parsed event, such as Properties.Position.Latitude from the above example input.
syslog
This parser aims to be compatible with a variety of syslog formats. This includes RFC 3164 and RFC 5424. The parser does not implement every aspect of the syslog RFCs, but is instead liberal in what it accepts.
Example Input
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for foo on /dev/pts/8
<34>1 2003-10-11T22:14:15.003Z server1.com sshd - - pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.2.
<34>Oct 11 22:14:15 mymachine su: 'su root' failed for foo on /dev/pts/8
Oct 11 22:14:15 su: 'su root' failed for foo on /dev/pts/8
If no timezone is specified, as in the last two examples, the parser defaults to UTC time. To change that, you may create a new parser by copying this parser and modifying:
timezone="UTC"
to your desired timezone.
The parser also leverages LogScale's built-in key value parser kv.
The parser syslog-utc will have better performance when the logs display this specific format.
syslog-utc
This parser can process standard lines generated by the syslog system.
The parser expects lines to starts with a timestamp, followed by the optional fields host, app, and pid. It also expects the timestamp to be in the UTC time zone. If your timestamps are in your local timezone, and that is not UTC, you will want to create a new parser by copying this parser and modifying
timezone="UTC"
to your desired timezone.
This parser also leverages LogScale's built-in key value parser kv.
Example Input
Feb 25 19:17:01 Myhost CRON[24886]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 25 06:35:01 Myhost CRON[24272]: (root) CMD (command -v deb-sa1 > /dev/null && deb-sa1 1 1)
zeek-json
This parser can process JSON data generated from Zeek. It is tailored to handle the output generated from the Zeek script, and you can read about how to send Zeek data to LogScale on the Zeek (Bro) Network Security Monitor documentation page.
The name of the Zeek log file will become a #path tag in LogScale.