Parsers Best Practices
The parser needs to extract the timestamp correctly and apply any field names that will help people navigate and search the data.
Take care to get the right balance between completeness of functionality and speed of the parser. Follow the guidance given in our documentation on Creating a Parser.
Tags are labels that can be applied to identify a class/type of log message but they need to be used carefully and it's probably best to check in with us if you are considering use Tags.
Testing the parser with a representative sample of logs is critical. This is especially important for systems that generate different logs that can have different numbers of fields, different value types etc.
The parser should contain multiple test events that are incorporated into the parser yaml file when the package is created or the parser is exported. This demonstrates basic parser functionality by using only the parser yaml file uploaded to LogScale.
Note that these test cases are publicly available, and should be scrubbed of any sensitive information. If you still want the data to look real, we recommend using data that is explicitly made for testing or documentation purposes. For example, when using IPv4 addresses, section 3 of RFC5737 declares that:
The blocks
192.0.2.0/24(TEST-NET-1),198.51.100.0/24(TEST-NET-2), and203.0.113.0/24(TEST-NET-3) are provided for use in documentation.
Similarly for URLs, section 3 of RFC2606 says:
The Internet Assigned Numbers Authority (IANA) also currently has the following second level domain names reserved which can be used as examples.
example.com example.net example.org
For email addresses, use the domains from the URL example data as your
email address domains, and make sure the local part of the address is
not something that can identify a user. Use for example a random text
string, or text like support or
contact.
The same attention should be paid to user names examples. Acceptable
user names can be e.g. root,
test, or random text strings like
qwerty.
For North American phone numbers, there are 100 numbers reserved for
fictional use: 555-0100 through
555-0199. If you need a North
American phone number for your data, please use one of those. For more
information, see section 2.0 of the
555
NXX Line Number Reference Document.
Figure 72, “Parser” shows a parser that includes 8 test events.
 |
Figure 72. Parser