Measuring Data Ingest

For monitoring your current usage in Organization Settings, see Usage Page. The following diagram illustrates the flow of data into LogScale and the various measurement points. For most customers, Daily Ingest Capacity will correspond to MP below:

Ingest Flow

Figure 21. Ingest Flow


When calculating the amount of data ingested by LogScale we use the following formula: 

ingestAfterFieldRemovalSize = @rawstring + fields + tags - removed fields and tags

  • @rawstring

    This is the length of the original ingested event string

  • fields

    New keys and values of any additional fields created during parsing; fields that were extracted from the rawstring are not included.

  • tags

    New tags created during parsing; tags that are extracted from the rawstrings are not counted.

  • removed fields and tags (MR)

    Fields removed from the incoming data using a filter removed before parsing. See Optimizing Ingestion for more details.

You can measure it by querying your humio-usage repository for ingestAfterFieldRemovalSize. See Measurement Repositories for more details.

Note

Your data needs to have a @timestamp field in order to be searchable in LogScale. This field counts as ingest, unless it's included in the rawstring.