Usage
In this article, we will first explain the UI of the Usage page, where you can see usage relative to your license: the current status as well as historical. These are the measurements that your Humio contract is based on.
Then, we will explain in depth what these numbers are based on, and how you can dig into those numbers yourself.
Attention
Beta
The Usage page and its underlying measurement tools are in beta.
Usage Page
On the Usage page, you can track and get an overview of your organization's usage, which includes ingest, storage, scanned data, and, if applicable, user seats.
The Usage page is available to organization owners only, and it can be found under Organization settings → Usage.
Figure 288. Usage Page
Current Usage
The current usage gives you an indication of your usage at the current moment and whether you are going above or below your contracted values.
If you're exceeding your contract, the panel will indicate this with a warning.
Figure 289. Current Usage
Ingest Over Time
In the ingest chart, you can get an overview of ingestion within a selected time period.
The Average ingest per day is calculated as a 30-day moving average. This means, for example, that the value shown for the 15th of July is the average daily ingest in the period 15th of June to 15th of July. This is to allow for spikes in ingest.
The ingest chart also shows the license limit, and an indication for which periods the rolling average has passed the limit.
Figure 290. Ingest Over Time With Spikes Example
You can select a single date, which will update the data shown in the repository table.
Stored Data Over Time
In the stored data chart, you can get an overview of storage usage within a selected time period.
The storage chart will also show the license limit and indicate for which periods the storage has passed the limit.
Figure 291. Stored data Over Time
As was the case for the ingest chart, you can select a single date, which will update the data shown in the repository table.
Repository Table
For both ingest and stored data, you can get an overview of the usage data based on the repositories that the data is in.
The data shown in the table correlates with the selected year, month and day from the chart.
In the table, you are able to search for specific repositories and sort based on name and value to get a better idea of which repositories have the most or least usage.
From the table, you can navigate to each repository or run a usage
query in humio-organization-usage, which will show
logs for that particular repository
(#repo=NAME_OF_REPO.). Note that you need to have
permissions to search in the
humio-organization-usage repository for this to
work as intended.
Figure 292. Repository Table
What We Measure
The measurements your contract is based on are the following: ingested data, stored data and scanned data, and possibly, the number of user seats, depending on the contract.
Ingested Data
Ingested data is the amount of data in bytes after it was parsed in Humio.
Stored Data
Stored data is the amount of data that you have stored in Humio, in bytes.
Scanned Data
Scanned data is the amount of data that was searched through when running queries. Every time a query runs, Humio measures the amount of data it needs to look into to answer the query.
User Seats
The number of users your contract limits you to, if any.
How We Measure Usage
We collect your usage data by logging it internally in Humio.
The diagram below shows the flow of ingest and all the points where we measure your usage for infrastructure maintenance needs.
M3 is the point that we use to measure
ingested data. As you can see in the chart, it is based on a field
called segmentWriteBytes
(segment_save).
Note that parsing can either reduce or expand the log size. Adding to your data during parsing can make it more useful, but carries additional ingest cost as it increases the amount of data.
Figure 293. Ingest Flow
Humio Measurement Repositories
We log your data volume in multiple repositories. You can use them to run audits to see how much data you ingest, which repositories it went to, and how much are you storing.
humio-organization-usage View
The humio-organization-usage view is available to
Cloud customers, and contains data from two repositories,
humio-measurements and
humio-usage. The
humio-organization-usage view contains logs with
information on how much data you are ingesting to Humio, how much
data you have stored, and in which repositories. It also tells you
how much data you are scanning when searching through logs. You can
filter the logs by which repository they come from by using #repo
field. For instance, to see only logs from the
humio-measurements repository, you would write
the following query: #repo = humio-measurements.
Customers using Humio self-hosted solution have access to these
repositories directly, and because of that, do not have
humio-organization-usage view.
humio-usage Repository
The logs in this repository are the results of an hourly query to
the humio-measurements repository. It differs
from the humio-measurements repository in the
following: it has unlimited retention, data is being logged once
every hour, and it does not include data on ingestion source.
Moreover, the usage measurements are provided as fields in the log.
In the table below, there are some of the more interesting fields a log line could have:
| Field | Example Value | Explanation |
|---|---|---|
#sampleRate
| hour | To which period the values in this log pertain to. 1 hour in most cases. |
#sampleType
|
usageTag
|
If this log line refers to a repository, or a set of
repositories that are grouped under the same usageTag. The
value can be one of the following:
organization, usageTag
or repository.
|
repo
| your_repo_name |
The repository name measurements in this log line pertain
to, if #sampleType is
repository.
|
dataScanned
| 123546 |
The amount of data that was scanned in the last hour in
#sampleType.
|
ingestBytes
| 23123 |
The amount of data that was ingested to this
#sampleType in the last
#sampleRate, measured in bytes.
|
segmentWriteBytes
| 12313214 | The amount of data in bytes written to the disk in the last hour. |
storageSize
| 129071068836 |
Total disk usage in the #sampleType.
|
queryStart
| 2021-06-28T07:31:23.044Z |
The time window beginning of querying the
humio-measurements repository.
|
queryEnd
| 2021-06-28T07:31:23.044Z |
The time window end of querying the
humio-measurements repository.
|
logId
| 21 |
The id that binds the logs of different
#sampleType together. See the section on
LogId below.
|
humio-measurements Repository
The humio-measurements repository holds more
fine-grained details, and has 30 days retention. Data is being
logged to this repository once every minute.
In the table below, there are some common fields to all logs in this repository:
| Field | Example Value | Explanation |
|---|---|---|
| #measurement |
ingest_bytes
|
One of the usage measurements. It tells you what this log is
about. It can be one of the following:
ingest_bytes,
segment_save or
data_scanned.
|
| #repo |
humio-measurements
|
Repository the log comes from. It can be one of the
following: humio-measurements or
humio-usage.
|
In addition to the common fields, the logs will hold more fields depending on the #measurement field.
The fields that are available when #measurement
equals ingest_bytes are as follows:
| Field | Example Value | Explanation |
|---|---|---|
byteCount
| 963075 | The number of ingested bytes. |
dataspaceId
|
humio
|
Dataspace identifier of the dataspace into which the amount
of data from byteCount field was
ingested.
|
ingestSource
|
appender
| Ingested data source. |
ingestSourceType
| Type of ingest source. | |
repositoryName
|
humio
| The name of the repository into which the logs were ingested. |
The fields that are available when #measurement equals segment_save are as follows:
| Field | Example Value | Explanation |
|---|---|---|
byteCount
| 963075 |
The amount of bytes that are stored in the repository of
repositoryName.
|
dataspaceId
|
humio
| Identifier of the dataspace where data is stored. |
repositoryName
|
humio
| The name of the repository where the data is stored. |
LogId
The logs with different #sampleTypes share one
value, which is the logId. For instance, ingest bytes of in the log
line where #sampleType equals organization will
be the sum of ingest bytes of all the repositories inside the
organization.
| #sampleType | ingestBytes | logId |
|---|---|---|
repository
| 2909 | 2 |
repository
| 1290 | 2 |
repository
| 879 | 2 |
organization
| 5078 | 2 |
By tracing the logId, you can drill down into
your usage, and find out what your usage was in a specific time
period, down to an hour, by repository. Since there is unlimited
retention on this repository, you will always be able to see your
usage from beginning your usage of Humio.