Measuring Data Ingest

To gain a deeper understanding of how to measure data ingest, LogScale uses the following formula when calculating the amount of data ingested by LogScale: 

ingestAfterFieldRemovalSize = @rawstring + fields + tags - removed fields and tags

  • @rawstring

    This is the length of the original ingested event string

  • fields

    New keys and values of any additional fields created during parsing; fields that were extracted from the rawstring are not included.

  • tags

    New tags created during parsing; tags that are extracted from the rawstrings are not counted.

  • removed fields and tags (MR)

    Fields removed from the incoming data using a filter removed before parsing. See Optimizing Ingestion for more details.

So the ingest accounting calculation at the base level is the size of the incoming data (@rawstring) plus the fields needed for storage (such as, @timestamp if it was not sent with the data).

Note

Your data needs to have a @timestamp field in order to be searchable in LogScale. This field counts as ingest, unless it's included in the rawstring.

For parsed data, to be more explicit, the more detailed description is the size of the incoming data (@rawstring) plus the fields needed for storage and classification (@timestamp and any #tags and standard fields NOT extracted or derived from @rawstring).

For parsed data, in the majority of cases, there are no added/implied fields that are not derived from the incoming rawstring. So while LogScale may add fields based on that information, it doesn't trigger/apply to the ingest accounting. There may be some types of ingested data that must have fields added. So in practice, the ingest accounting only counts the original raw data and a timestamp if that isn't found in the original event data.

To measure the ingest amount, query the humio-usage repository for ingestAfterFieldRemovalSize. See The humio-usage Repository for more details.

To monitor current usage in Organization Settings, see Usage Page.