Usage

The humio-organization-usage is a view that is available to Cloud customers, and which is built on top of two repositories, humio-measurements and humio-usage. Customers using Humio self-hosted solution have access to these repositories directly, and because of that, do not have humio-organization-usage view. By looking into data from the humio-measurements repository, you can get a detailed overview of your Humio usage for the last 30 days. By inspecting the logs from humio-usage repository, you get a breakdown of your usage since you started using Humio down to the hourly, per repository basis. Usage dashboard is created for you automatically, and gives you a glimpse into your usage.

The humio-organization-usage view contains logs with information on how much data you are ingesting to Humio, how much data you have stored in which repositories, and how much data you are scanning when searching through logs. This article explains what these measurements mean, how they are calculated, and how you can leverage them.

Measurements

When talking about Humio usage, look into these measurements: ingested data, stored data and scanned data, and possibly, into user seats, depending on the contract.

Ingested Data

Ingested data is the amount of data in bytes after it was parsed in Humio, and is based on the field called segmentWriteBytes (segment_save). The amount of data that is received at ingest endpoint is saved in the field ingestBytes (ingest_bytes).

Stored Data

Stored data is the amount of data that you have stored in Humio, in bytes.

Scanned Data

Scanned data is the amount of data that was searched through when running queries. Every time a query runs, Humio measures the amount of data it needs to look into to answer the query.

User Seats

The number of users your contract limits you to, if any.

humio-organization-usage View

The humio-organization-usage view holds logs on the aforementioned measurements, and is available to Cloud customers. It holds logs from two repositories: humio-usage and humio-measurements. These repositories have different retention values and have information with different levels of detail. You can filter the logs by which repository they come from by using #repo field, for instance, to see only logs from the humio-measurements repository, you would write the following query: #repo = humio-measurements.

humio-measurements Repository

The humio-measurements repository holds more details and has 30 days retention. Data is being logged to this repository once every minute.

In the table below, there are some common fields to all logs in this repository:

Field

Example Value

Explanation

#measurement

ingest_bytes

One of the usage measurements. It tells you what this log is about. It can be one of the following: ingest_bytes, segment_save or data_scanned.

#repo

humio-measurements

Repository the log comes from. It can be one of the following: humio-measurements or humio-usage.

In addition to the common fields, the logs will hold more fields depending on the #measurement field.

The fields that are available when #measurement equals ingest_bytes are as follows:

Field

Example Value

Explanation

byteCount

963075

The number of ingested bytes.

dataspaceId

humio

Dataspace identifier of the dataspace into which the amount of data from byteCount field was ingested.

ingestSource

appender

Ingested data source.

ingestSourceType

Type of ingest source.

repositoryName

humio

The name of the repository into which the logs were ingested.

The fields that are available when #measurement equals segment_save are as follows:

Field

Example Value

Explanation

byteCount

963075

The amount of bytes that are stored in the repository of repositoryName.

dataspaceId

humio

Identifier of the dataspace where data is stored.

repositoryName

humio

The name of the repository where the data is stored.

humio-usage Repository

The logs in this repository are the results of an hourly query to the humio-measurements repository. It differs from the humio-measurements repository in the following: it has unlimited retention, data is being logged once every hour, and it does not include data on ingestion source. Moreover, the usage measurements are provided as fields in the log.

In the table below, there are some of the more interesting fields a log line could have:

Field

Example Value

Explanation

#sampleRate

hour

To which period the values in this log pertain to. 1 hour in most cases.

#sampleType

usageTag

If this log line refers to a repository, or a set of repositories that are grouped under the same usageTag. The value can be one of the following: organization, usageTag or repository.

repo

your_repo_name

The repository name measurements in this log line pertain to, if #sampleType is repository.

dataScanned

123546

The amount of data that was scanned in the last hour in #sampleType.

ingestBytes

23123

The amount of data that was ingested to this #sampleType in the last #sampleRate, measured in bytes.

segmentWriteBytes

12313214

The amount of data in bytes written to the disk in the last hour.

storageSize

129071068836

Total disk usage in the #sampleType.

queryStart

2021-06-28T07:31:23.044Z

The time window beginning of querying the humio-measurements repository.

queryEnd

2021-06-28T07:31:23.044Z

The time window end of querying the humio-measurements repository.

logId

21

The id that binds the logs of different #sampleType together. See the section on LogId below.

LogId

The logs with different #sampleTypes share one value, which is the logId. For instance, ingest bytes of in the log line where #sampleType equals organization will be the sum of ingest bytes of all the repositories inside the organization.

#sampleType

ingestBytes

logId

repository

2909

2

repository

1290

2

repository

879

2

organization

5078

2

By tracing the logId, you can drill down into your usage, and find out what your usage was in a specific time period, down to an hour, by repository. Since there is unlimited retention on this repository, you will always be able to see your usage from beginning your usage of Humio.

Usage Dashboard

Figure 1, Usage Dashboard Example

Usage dashboard is built on top of the humio-organization-usage view if using Humio Cloud, and on top of humio-usage repository if you are using Humio self-hosted solution. This dashboard visualizes some measurements discussed above.