A Grafana dashboard may contain many panels, each showcasing data from a different type of data source. After installing the LogScale plugin, you will be able to create a LogScale data source within Grafana, and start adding panels to your dashboards. These panels can then be populated with results from LogScale queries, which you can type directly into the Grafana UI.

Here is a showcase of some the panels that you'll be able to display your LogScale data in.

Logs

When you use the Logs panel, you can show LogScale events in a log format. Such events will be represented by the @rawstring

Figure 17. LogScale Events

Time Series Graphs

When you use the timeChart() function in one of your queries, it creates one or more time-series based on the event data fed into the function. Time-series data can easily be plotted as graphs in Grafana by using the Graph panel type, which can be seen below.

Figure 18. Time Series Graphs

This specific panel showcases the number of incoming connections over time across different hosts in a network. It was created by querying a LogScale repository filled with Windows logs, coming from different Windows PCs on the same network. The query used can be seen below. It filters for all events that indicate a network connection, and then pipes those into timeChart(), generating a time series for each unique hostname discovered in the filtered events.

default

event.action = "Network connection detected (rule: NetworkConnect)" |
timechart(series=host.name)

Like in the LogScale UI, you are able to zoom into specific parts of your graph in Grafana, and you may plot multiple time-series from multiple different queries on the same panel.

Bar Gauges

LogScale supports the use of the groupBy() function in its queries, following the same concept as the GROUP BY found in SQL. The grouped data can be visualized using Grafana's Bar Gauge panel type, as shown below.

Figure 19. Bar Gauges

This specific panel is generated from a HaProxy server, which sits in front of a LogScale instance. It shows the average number of bytes read per LogScale query job, created on that particular LogScale instance. The query used to create the panel can be seen below. As the logs from HaProxy only leave details about web requests, we have to use LogScale's /regex/ feature to filter for the events, which describe a web requesting polling from a query job. These events are then grouped by query job, summing up the number of bytes read from the job. Finally the data is grouped across the different repos on the LogScale instance.

default

method=GET |
"status_code"=200 |
regex("/api/v1/dataspaces/(?<refdomain>.+?)/queryjobs/1-(?<jobId>.{24})",field=path) |
groupby([refdomain, jobId],  function=sum(bytes_read)) |
groupby(refdomain, function=avg(_sum))

As with all other Grafana panels, the Bar Gauge panel is visually customizable, and you can change its look and feel by editing the settings of the panel. In this example we have chosen a retro LCD look for each of our gauges.

Tables

LogScale generates tabular data, when the table() function is used within a query. Naturally, this type of data can be displayed in Grafana's Table panel type, as shown below. As a neat feature, if you do not want to use the header names generated by the query, Grafana allows you to change them through the Grafana UI.

Figure 20. Grafana Tables

This particular panel was created from the query below, which looks at Windows logs in a LogScale repository. First, we use filters to get a hold of all the unsuccessful login attempts. Afterwards we group the logons by username before finally piping the grouped data into the aforementioned table() function.

default

winlog.task = Logon |
winlog.keywords[0] = "Audit Failure" |
groupby(winlog.event_data.TargetUserName) |
table([_count, winlog.event_data.TargetUserName], sortby=_count, limit=5)

Single Stats

While some queries produce very simple results, they can still be very useful. As an example, take queries that use the count() function, which only returns a single number. When displaying this in a Singlestat panel, as shown below, you can add thresholds values. Now each time the value breaks a threshold the panel will change color.

Figure 21. Grafana Single Stats

Again, this panel has been populated by querying a LogScale repo filled with Windows logs. It uses a few filters to get all unsuccessful login attempts, and then pipes these into count().

default

winlog.task = Logon  |
winlog.keywords[0] = "Audit Failure"  |
count()

Worldmap

LogScale has a great feature in the worldMap() function, which can take input from log events, such as IP addresses, and map them into a data format that can be visualized on a map.

While Grafana doesn't have a native Worldmap panel, there is a third party Worldmap plugin. This can be combined with the LogScale plugin to showcase locational data on your Grafana dashboard, as shown below.

Figure 22. Grafana Worldmap

The query to create this panel was very simple. It only required us to filter on http events and then pipe those into the worldMap() function.

default

type=http | worldmap(ip=client_ip)

Please note that in order to get the results shown on the panel above, you will need to configure the panel a bit. This is however covered in our Grafana.

Thanks to data links, you can link other data sources like Tempo/GET/Jaeger/Zipkin with the a LogScale data source.

Figure 23. Data Links on a LogScale Data Source

This particular data link has been configured to correlate LogScale events with Tempo data. The trace_ID is a field which will link our data with Tempo traces.

Figure 24. LogScale events correlated with Tempo trace

On the explore panel the link which is created is visible as a button in the Links section inside the Log details view. In our case we linked our LogScale event with Tempo data source. When you click the Tempo button, it will open a new panel with detailed data coming from Tempo. This allows you to quickly and easily view related data coming from the other systems and dive into more details.

This Grafana capability allows you to correlate traces from different data sources including Tempo, Get, Jaeger or Zipkin with logs. In our case we are going to configure a Tempo data source to correlate its traces with LogScale events.

Figure 25. Tempo configured to link with LogScale events

In the above configuration we added a tag to associate a trace with the corresponding LogScale field. In our case it's trace_ID.

Figure 26. Tempo configured to link with LogScale events

We executed a query for the selected Tempo trace id to see trace details, showing all the spans covered by the lists-articles request. To jump directly to the corresponding LogScale events click on the Explore Logs icon.

In our case we see that 2 events have been found along with other details which might be useful when performing further log analysis.

In Grafana, a variable is a named set of values, which you can use to dynamically generate dashboard content. While you can of course hard-code your own set of variable values, i.e. a list of system host names, the LogScale plugin allows you to populate your variables with the results of LogScale queries using the Query Variable feature in Grafana. This saves you from having to type in each variable value, and it ensures that the values of your variables can be updated automatically over time, depending on LogScale query results.

If you take a look at the figure below, you can see how we configure a query variable Hostnames, by setting up a groupBy([host.name], function=[]) query against our WindowsServers repository. To turn the query results into variable values, we specify that we want to extract the values from the host.name field of the results. Finally, we click Run Query, which yields the following set of variable values: (gateway, WIN-TTMD6GMVLH9). Note that while the query result can include up to 200 events, duplicate values are discarded when populating our variable, leaving us with the names of the only two hosts ingesting logs into WindowsServers .

Figure 27. Grafana Query Variables

Variables have various uses within Grafana, but we want to highlight how they can be used to template LogScale queries within dashboard panels. This will let you avoid having to define multiple near identical queries for multiple panels. Let us say you want to create a panel for each host showing the number of windows logs generated by the user SYSTEM over time. We can template the query using the variable Hostnames like so:

host.name=$Hostnames
| winlog.user.name = SYSTEM
| timechart(series=host.name)

The same principle works if we want to display a panel for each host, showcasing the number of failed logins:

host.name=$Hostnames
| winlog.task = Logon
| winlog.keywords[0] = "Audit Failure"
| count()

With these queries plugged into their respective panels in a dashboard, we can use the Hostnames variable dropdown on the dashboard to switch between which host we are displaying data from. In the following figure, you can see how our dashboard shows data from LWIN-TTMD6GMVLH9:

Figure 28. More Grafana Query Variables

And by just picking another option in the Hostnames dropdown, we can update the dashboard to show data from gateway instead:

Figure 29. More Grafana Query Variables

Grafana annotations allow you to visually display important log events across your Grafana graph panels.

Let's take the graph from the Query Variables example, which showcases a graph over the number of windows logs generated by the SYSTEM user. Using annotations, this graph can be annotated to show each Windows Update event that has occurred across the time period displayed by the graph.

In the figure below, you can see how we configure a new annotation Windows Update Events with the following filter query:

host.name = $Hostnames | winlog.task = "Windows Update Agent"

Which yields log events for each Windows Update event that has occurred on a particular host. Note that this is another example of using a query variable to template a query. In addition to the query, we also specify the text displayed alongside each annotation as corresponding to the message field of that particular log event.

Figure 30. Grafana Annotations

With the annotation configured, our graph panels will now be marked with an annotation for each log event returned by our annotation query, which is executed alongside the graph query, marking each Windows Update event:

Figure 31. Grafana Annotations