Use Case: Integrating LogScale with Grafana

Using the Falcon LogScale Web UI, you can create dashboards that display the results of your queries in real time and in a way that makes them easy to understand. This is important, as having a comprehensive dashboard can mean the difference between spotting an important development in your infrastructure and letting it pass by unnoticed.

While we are proud of our UI, it is not your sole option for creating dashboards with LogScale data. We know that some of our customers like to create their dashboards in Grafana, a popular open source project that has been around since 2014. With Grafana, you may create dashboards with data extracted from various data sources, including Prometheus, Sensu, Splunk, Elasticsearch, and of course LogScale.

You can use LogScale as a data source within Grafana by using our LogScale plugin for Grafana. To install and set up this plugin, please refer to the ??? section of this documentation, which also explains general usage of the add-on.

On this page, we wish to show you the many ways in which you can use LogScale query results within Grafana, thanks to our plugin. This should hopefully inspire you to try it out, and start including LogScale data within your existing Grafana dashboards.

Panels

A Grafana dashboard may contain many panels, each showcasing data from a different type of data source. After installing the LogScale plugin, you will be able to create a LogScale data source within Grafana, and start adding panels to your dashboards. These panels can then be populated with results from LogScale queries, which you can type directly into the Grafana UI.

Here is a showcase of some the panels that you'll be able to display your LogScale data in.

Logs

When you use the Logs panel, you can show LogScale events in a log format. Such events will be represented by the @rawstring

LogScale Events

Figure 17. LogScale Events


Time Series Graphs

When you use the timeChart() function in one of your queries, it creates one or more time-series based on the event data fed into the function. Time-series data can easily be plotted as graphs in Grafana by using the Graph panel type, which can be seen below.

Time Series Graphs

Figure 18. Time Series Graphs


This specific panel showcases the number of incoming connections over time across different hosts in a network. It was created by querying a LogScale repository filled with Windows logs, coming from different Windows PCs on the same network. The query used can be seen below. It filters for all events that indicate a network connection, and then pipes those into timeChart(), generating a time series for each unique hostname discovered in the filtered events.

default
event.action = "Network connection detected (rule: NetworkConnect)" |
timechart(series=host.name)

Like in the LogScale UI, you are able to zoom into specific parts of your graph in Grafana, and you may plot multiple time-series from multiple different queries on the same panel.

Bar Gauges

LogScale supports the use of the groupBy() function in its queries, following the same concept as the GROUP BY found in SQL. The grouped data can be visualized using Grafana's Bar Gauge panel type, as shown below.

Bar Gauges

Figure 19. Bar Gauges


This specific panel is generated from a HaProxy server, which sits in front of a LogScale instance. It shows the average number of bytes read per LogScale query job, created on that particular LogScale instance. The query used to create the panel can be seen below. As the logs from HaProxy only leave details about web requests, we have to use LogScale's /regex/ feature to filter for the events, which describe a web requesting polling from a query job. These events are then grouped by query job, summing up the number of bytes read from the job. Finally the data is grouped across the different repos on the LogScale instance.

default
method=GET |
"status_code"=200 |
regex("/api/v1/dataspaces/(?<refdomain>.+?)/queryjobs/1-(?<jobId>.{24})",field=path) |
groupby([refdomain, jobId],  function=sum(bytes_read)) |
groupby(refdomain, function=avg(_sum))

As with all other Grafana panels, the Bar Gauge panel is visually customizable, and you can change its look and feel by editing the settings of the panel. In this example we have chosen a retro LCD look for each of our gauges.

Tables

LogScale generates tabular data, when the table() function is used within a query. Naturally, this type of data can be displayed in Grafana's Table panel type, as shown below. As a neat feature, if you do not want to use the header names generated by the query, Grafana allows you to change them through the Grafana UI.

Grafana Tables

Figure 20. Grafana Tables


This particular panel was created from the query below, which looks at Windows logs in a LogScale repository. First, we use filters to get a hold of all the unsuccessful login attempts. Afterwards we group the logons by username before finally piping the grouped data into the aforementioned table() function.

default
winlog.task = Logon |
winlog.keywords[0] = "Audit Failure" |
groupby(winlog.event_data.TargetUserName) |
table([_count, winlog.event_data.TargetUserName], sortby=_count, limit=5)
Single Stats

While some queries produce very simple results, they can still be very useful. As an example, take queries that use the count() function, which only returns a single number. When displaying this in a Singlestat panel, as shown below, you can add thresholds values. Now each time the value breaks a threshold the panel will change color.

Grafana Single Stats

Figure 21. Grafana Single Stats


Again, this panel has been populated by querying a LogScale repo filled with Windows logs. It uses a few filters to get all unsuccessful login attempts, and then pipes these into count().

default
winlog.task = Logon  |
winlog.keywords[0] = "Audit Failure"  |
count()
Worldmap

LogScale has a great feature in the worldMap() function, which can take input from log events, such as IP addresses, and map them into a data format that can be visualized on a map.

While Grafana doesn't have a native Worldmap panel, there is a third party Worldmap plugin. This can be combined with the LogScale plugin to showcase locational data on your Grafana dashboard, as shown below.

Grafana Worldmap

Figure 22. Grafana Worldmap


The query to create this panel was very simple. It only required us to filter on http events and then pipe those into the worldMap() function.

default
type=http | worldmap(ip=client_ip)

Please note that in order to get the results shown on the panel above, you will need to configure the panel a bit. This is however covered in our ???.

Trace to Logs

This Grafana capability allows you to correlate traces from different data sources including Tempo, Get, Jaeger or Zipkin with logs. In our case we are going to configure a Tempo data source to correlate its traces with LogScale events.

Tempo configured to link with LogScale events

Figure 25. Tempo configured to link with LogScale events


In the above configuration we added a tag to associate a trace with the corresponding LogScale field. In our case it's trace_ID.

Tempo configured to link with LogScale events

Figure 26. Tempo configured to link with LogScale events


We executed a query for the selected Tempo trace id to see trace details, showing all the spans covered by the lists-articles request. To jump directly to the corresponding LogScale events click on the Explore Logs icon.

In our case we see that 2 events have been found along with other details which might be useful when performing further log analysis.

Query Variables

In Grafana, a variable is a named set of values, which you can use to dynamically generate dashboard content. While you can of course hard-code your own set of variable values, i.e. a list of system host names, the LogScale plugin allows you to populate your variables with the results of LogScale queries using the Query Variable feature in Grafana. This saves you from having to type in each variable value, and it ensures that the values of your variables can be updated automatically over time, depending on LogScale query results.

If you take a look at the figure below, you can see how we configure a query variable Hostnames, by setting up a groupBy([host.name], function=[]) query against our WindowsServers repository. To turn the query results into variable values, we specify that we want to extract the values from the host.name field of the results. Finally, we click Run Query, which yields the following set of variable values: (gateway, WIN-TTMD6GMVLH9). Note that while the query result can include up to 200 events, duplicate values are discarded when populating our variable, leaving us with the names of the only two hosts ingesting logs into WindowsServers .

Grafana Query Variables

Figure 27. Grafana Query Variables


Variables have various uses within Grafana, but we want to highlight how they can be used to template LogScale queries within dashboard panels. This will let you avoid having to define multiple near identical queries for multiple panels. Let us say you want to create a panel for each host showing the number of windows logs generated by the user SYSTEM over time. We can template the query using the variable Hostnames like so:

default
host.name=$Hostnames
| winlog.user.name = SYSTEM
| timechart(series=host.name)

The same principle works if we want to display a panel for each host, showcasing the number of failed logins:

default
host.name=$Hostnames
| winlog.task = Logon
| winlog.keywords[0] = "Audit Failure"
| count()

With these queries plugged into their respective panels in a dashboard, we can use the Hostnames variable dropdown on the dashboard to switch between which host we are displaying data from. In the following figure, you can see how our dashboard shows data from LWIN-TTMD6GMVLH9:

More Grafana Query Variables

Figure 28. More Grafana Query Variables


And by just picking another option in the Hostnames dropdown, we can update the dashboard to show data from gateway instead:

More Grafana Query Variables

Figure 29. More Grafana Query Variables


Annotations

Grafana annotations allow you to visually display important log events across your Grafana graph panels.

Let's take the graph from the Query Variables example, which showcases a graph over the number of windows logs generated by the SYSTEM user. Using annotations, this graph can be annotated to show each Windows Update event that has occurred across the time period displayed by the graph.

In the figure below, you can see how we configure a new annotation Windows Update Events with the following filter query:

default
host.name = $Hostnames | winlog.task = "Windows Update Agent"

Which yields log events for each Windows Update event that has occurred on a particular host. Note that this is another example of using a query variable to template a query. In addition to the query, we also specify the text displayed alongside each annotation as corresponding to the message field of that particular log event.

Grafana Annotations

Figure 30. Grafana Annotations


With the annotation configured, our graph panels will now be marked with an annotation for each log event returned by our annotation query, which is executed alongside the graph query, marking each Windows Update event:

Grafana Annotations

Figure 31. Grafana Annotations