When you send server logs and metrics to Humio, that data almost always needs to be parsed before it can be stored in a repository. A parser takes text as input; the text can be structured text (e.g., in json format) or unstructured text (e.g., like syslog or application stdout). It then extracts fields which are stored along with the original text. Parsers in general, as well as specific built-in parsers are described in the Parsers section of the Documentation.
On the Settings page of this section of the documentation on the User Interface, we mention that you can have an API Tokens for each parser that’s used by the repository. By this method, you can check the Parsers section of the User Interface to see if the parsers are having difficulty parsing the data its ingesting.
In the cropped screenshot shown in Figure 1 here, you can see the first couple of built-in parsers. The first one is the
accesslog parser. We’re using it to parse Apache HTTP access logs that are sent to Humio by a log shipper (i.e.,
vector in this case) from a server. Notice it says there were errors recorded. These are errors parsing events. You would probably want to fine tune that parser. You can do so by clicking on it to see the parser script.
To create a parser, from the Parsers page of the User Interface, click on the large button at the top left labeled, + New Parser. That will open a box, like the one shown in Figure 2, for you to give the parser a name.
Enter a name for the parser, something you’ll recognize. If you want to base your new parser off of an existing one, click on Clone Existing and then choose which parser from the pull-down menu. Otherwise, just leave it with New Parser selected. When you’re done, click Create.
Next you will see a code editor, similar to Figure 3 below. If you cloned an existing parser, you’ll see the code for it in the left panel under where it says Parser Script, and some test data in the right panel under where it says Test Data.
If you’re creating a parser, you’ll have to copy some log entries from your server to add as test data. To do this, click on the button near the top right labeled, + Add Example. This will add a row in the right panel for you to paste a log entry. You’ll have to click the + Add Example button for each entry you want to enter.
While considering the test data, you can construct the query. You may use any of the Humio Query Functions. When you feel your query is correction, click the purple button labeled, Save above the Parser Script panel. If you have any errors in your coding, you’ll see a red box saying you have an error and offering suggestions for fixing it.
When your query is error free and you have sufficient number of entries in the Test Data, click on the button near the top right where it says, Run Tests. It will let you know if there are any errors. If you have errors, try to determine how your query is incompatible with the test data and try again. When you have everything right, you should have no errors.
For more information on creating a parser, see the Creating Parsers documentation page.
After you’ve created a parser, you can click on the Settings tab near the top left, to change some settings there.
The only Options available are to rename the parser. This might cause stop ingesting of data if you have a client that uses the parser by name by way of the
#type field in a FileBeat configuration. You can avoid this by assigning parsers to API tokens instead and stop relying on the name in external clients.
Under Parser Settings, there’s also a tab called, Tagging. Tagging defines how events are stored in Humio and can impact performance when searching if you’re working with plenty of data — or it can work against you. For more information on tagging a parser, see the Tagging documentation page.
The last things you can set under Parser Settings are the Ingest Tokens. As you can see in the screenshot here in Figure 4, these are the same ingest tokens from the general, repository Settings tab. See the Settings UI |–| API Tokens for more information on this topic.