Given events containing an incidents array:

Event 1

|--------------|-------------|
| host         | v1          |
| incidents[0] | Evil Bear   |
| incidents[1] | Cozy Bear   |
|--------------|-------------|

Event 2

|--------------|-------------|
| host         | v15         |
| incidents[0] | Fancy Fly   |
| incidents[1] | Tiny Cat    |
| incidents[2] | Cozy Bears  |
|--------------|-------------|

Finds all the events where the field incidents contains the exact value Cozy Bear and group them by which hosts were affected, giving output event:

The result is an aggregated count of the array elements matching Cozy Bear.

This alert query tries to balance reacting when there are problems with parsers, without being too restrictive.

This query is used to set up alerts for parsers issues. Setting up alerts for parsers issues will allow to proactively reach out to customers where their queries are being throttled and help them.

In this example, the session() function is used to analyze user sessions based on users click activity. The session() function groups events by a given timespan.

Example incoming data might look like this:

The query is used to analyze user sessions based on the users click activity. The query is useful, for example, to identify most/least active user sessions, detect potential automated behavior or just to understand user engagement levels.

Sample output from the incoming example data:

Note that each row represents an event (either pageview or click).

In this example, the query uses sort() and groupBy() functions to find query costs. The query filters logs in humio repository that are tagged with kind equal to logs and then returns the events where the class field has values containing c.h.j.RunningQueriesLoggerJob, searching for the specific value Highest Cost query.

Example incoming data might look like this:

The query is used to search across multiple repositories and output query costs for all users by repository. The query returns the count in a field named _count. Use this query to focus on live and static costs separately.

Sample output from the incoming example data:

In this example, the linReg() function is used to calculate the linear relationship between request_type (x variable) and server_load_pct (y variable). The example shows the relationship between server load and each of several types of HTTP request types across time.

Example incoming data might look like this:

The query is used to analyze how different HTTP request types affect server load. The analysis helps identify which HTTP request types have the strongest impact on server performance.

Sample output from the incoming example data:

_slope is the impact rate of request volume on server load.

_intercept is the baseline server load when there are no requests of a specific type.

_r2 is the statistical accuracy of the relationship.

_n is the total number of data points analyzed.

In this example, the collect() function is used to collect visitors, each visitor defined as non-active after one minute.

The query is used to collect fields from multiple events into one event. This query analyzes user behavior by grouping events into sessions for each unique client IP address. It then collects all URLs accessed during each session. Collecting should be used on smaller data sets to create a list (or set, or map, or whatever) when you actually need a list object explicitly (for example, in order to pass it on to some other API). This analysis is valuable for understanding user engagement, and identifying potential security issues based on unusual browsing patterns. Using collect() on larger data set may cause out of memory as it returns the entire data set.

In this example, the collect() function is used to collect fields from multiple events.

The query is used to collect fields from multiple events into one event. Collecting should be used on smaller data sets to create a list (or set, or map, or whatever) when you actually need a list object explicitly (for example, in order to pass it on to some other API). Using collect() on larger data set may cause out of memory as it returns the entire data set. The query is useful for network connection analysis and for identifying potential threats.

Sample output might look like this:

In this example, the session() function is used to count the unique visitors (each visitor defined as non-active for 15 minutes) of a site based on client IP addresses. The session() function groups events by a given timespan.

Example incoming data might look like this:

The query is used to group events by client IP addresses into sessions of 15m, and then make a count of the total number of unique sessions (returns the total count of sessions across all IP addresses). The query is, for example, useful for measuring unique website/application visitors and understanding real user engagement patterns. Also useful for security monitoring and detection of unusual spikes in unique visitors.

Sample output from the incoming example data:

The query counts 4 unique sessions total as the first IP address has activity that spans beyond the 15-minute session timeout, creating two distinct sessions.

If you make the count on the client_ip field: | count(client_ip), the query will return a more detailed result showing the session count per IP address:

The humio-audit repository contains audit events for the LogScale cluster. Reporting on this information can provide a wealth of information, but a useful summary can be created based on the activities, users and which the latest user of that particular operation.

Pivoting an event set of data allows for the information to be displayed and summarized in a format that may make more logical sense as a display format. The final table will look like this:

If you want to deduplicate events by a given field, for example to identify a unique list of events for further processing, you can use an aggregate function. In this example, the groupBy() function is used with tail() to use the last value in a sequence of events.

The query is used to deduplicate events by a given field. This is useful if you want to create a unique list of events for further processing.

In this example, the partition() function is used with the groupBy() function to detect all occurrences of event A before event B (brute force attack).

The query will detect instances where there were 3 or more failed attempts followed by a successful attempt within the specified 10-second window.

Note that the partition() function must be used after an aggregator function to ensure event ordering. Also note that the events must be sorted in order by timestamp to prevent errors when running the query. It is possible to select any field to use as a timestamp.

Example incoming data might look like this:

The query is used to detect all occurrences of potential brute force attack patterns. It looks for instances where there were 3 or more failed attempts (event A) followed by a successful attempt (event B), regardless of the time between failures. The timespan between each attempt is reported, which could be used to identify brute force attacks.

Sample output from the incoming example data:

In this example, the partition() function is used with the groupBy() function to detect event A happening X times before event B (brute force attack).

The query will detect instances where there were 3 or more failed attempts followed by a successful attempt within the specified 10-second window.

Note that the partition() function must be used after an aggregator function to ensure event ordering. Also note that the events must be sorted in order by timestamp to prevent errors when running the query. It is possible to select any field to use as a timestamp.

Example incoming data might look like this:

The query is used to detect instances where there are 3 or more failed attempts followed by a successful attempt. The query can be used to detect a brute force attack where an attacker tries multiple times before succeeding. Note that the effectiveness of this query depends on the nature of your data and the typical patterns in your system.

Sample output from the incoming example data:

In this example, the slidingTimeWindow() function is used with the groupBy() function to detect event A happening X times before event B within a specific timespan.

The query will detect instances where there are 3 or more failed attempts followed by a successful attempt, all occurring within a 3-second window.

Note that the slidingTimeWindow() function must be used after an aggregator function to ensure event ordering. Also note that the events must be sorted in order by timestamp to prevent errors when running the query. It is possible to select any field to use as a timestamp.

Example incoming data might look like this:

The query is used to detect event A happening X times before event B within a specific timespan. It looks for instances where there were 3 or more failed attempts followed by a successful attempt, all occurring within a 3-second window. Using a sliding time window of 3 seconds, provides a more precise time constraint compared to the usage of partition() in Detect Event A Happening X Times Before Event B.

The query can be used to detect potential brute force attack patterns within a specific timeframe. Note that the effectiveness of this query depends on the nature of your data and the typical patterns in your system.

Sample output from the incoming example data:

It is possible to filter out on fields with no values in a given returned search result. In this example, all statuscode fields containing no value is filtered out from the final search result.

Example incoming data might look like this:

The query is used to filter out fields not containing any values from the returned search result.

Sample output from the incoming example data:

Search Repository: humio

wildcard(field=class,pattern="*Data*")
| groupBy(class)

Find all events containing any Data string in their class, and count the occurrences for each class that is found. For example, it can be used to get a list of events that have items such as DataIngestRateMonitor, or LocalDatasource.

The result is an aggregated count of all events matching anything with Data (with one or more characters before or after), in the class field.

Search Repository: humio

wildcard(field=class, pattern="*S3Bucket*", ignoreCase=true)
| groupBy(class)

Find all events containing any S3Bucket item (and all before and after) in their class, and count the occurrences for each class that is found.

The result is an aggregated count of all events matching anything with S3Bucket, case-insensitive, in the class field.

Figure 189. Search S3Bucket With wildcard()

In this example, the regular expression is used to search for patterns where the value Cozy Bear appears in a certain position across arrays.

Example incoming data might look like this:

The query using the regex expression are used to quickly search and return results for specific values in arrays. Regular expressions are useful when searching for different strings containing the same patterns; such as social security numbers, URLs, email addresses, and other strings that follow a specific pattern.

Sample output from the incoming example data:

In this example, the regular expression is used to search for patterns where the value bear appears at the end of a value in an array element, ignoring the case.

The queries using the regex expression are used to quickly search and return results for specific values in arrays. Regular expressions are useful when searching for different strings containing the same patterns; such as social security numbers, URLs, email addresses, and other strings that follow a specific pattern.

In this example, the session() function is used to find minimum and maximum values of the field bet in a session. The session() function groups events by a given timespan.

Example incoming data might look like this:

The query is used to analyze the likelihood (the bet) of the behavior within user sessions. This query is, for example, useful for identifying if the event was an attempt to hack the system.

Sample output from the incoming example data:

Note that each session shows its own min/max values.

In this example, the groupBy() function is used to get a list of status codes for logged events. For instance, the status code 200 is returned when the request is successful, and 404 when the page is not found.

The query is used to extract a list of status codes, each with a count of how many events have that status. The query is useful for summarizing and analyzing log data.

Sample output from the incoming example data:

In this example, the time:hour() function is used with groupBy() to average the count of data events per hour.

The query is used to average the count of data events per hour. The results can be plotted onto a bar chart.

In this example, the readFile() function is used to look up a host_names.csv file, and then filter for host names that do not send any logs.

Example incoming data might look like this:

|--------------------|
| host_name, host_id |
| DESKTOP-VSKPBK8, 1 |
| FINANCE, 2         |
| homer-xubuntu, 3   |
| logger, 4          |
| DESKTOP-1, 5       |
| DESKTOP-2, 6       |
| DESKTOP-3, 7       |
|--------------------|

The query is used to preview content in CSV Lookup Files, and then filter for host names that do not send any logs.

Sample output from the incoming example data:

By transposing event set, the information can be viewed and summarized in a more human readable form. In this example, the transpose() function is used within a groupBy() function to search across multiple structured fields in the HUMIO repository.

Example incoming data might look like this:

The query is used to search across multiple structured fields in the HUMIO repository to show where httpd appears most often. It makes results more readable, identifies field patterns, and is very useful for statistical analysis.

Sample output from the incoming example data:

Search Repository: humio

timestamp := formatTime(format="%H:%M")
| groupBy([thread],
function=[{sort("timestamp")
| collect("timestamp")}])

When using aggregation, you may want to sort on a field that is part of the aggregated set but not the main feature of the aggregated value. For example, sorting the values by their timestamp rather than the embedded value. To achieve this, you should use a function that sorts the field to be used as the sort field, and then use collect() so that the value from before the aggregation can be displayed in the generated event set. This query can be executed in the humio respository.

The result set will contain a list of the aggregated thread names sorted by the timestamp:

By transposing event set, the information can be viewed and summarized in a more human readable form. Transposing also allows for aggregated data to be viewed in a form where the value of an aggregated field becomes the columns. This can be used to summarize the information further by showing multiple rows of data. For example, in the humio repository the event information contains a list of errors, warnings, or informational events for activity within the cluster.

The transpose() is a great way of reorganizing data into a format is either more readable, or easily applied to other display formats as part of a widget. The final table looks like this:

However, the information as it is now formatted can more easily be applied to a variety of visualizations. For example, the data can be formatted as a bar chart, as we now have a list of fields and a value:

The difference is that without transpose(), the aggregate result set is a list of events with a field name and value. With transpose(), the result set is a single event with multiple fields, and this is interpreted by the bar chart as a series of values.