Measuring Data Ingest

For monitoring your current usage in Organization Settings, see Usage Page. The following diagram illustrates the flow of data into LogScale and the various measurement points. For most customers, Daily Ingest Capacity will correspond to MP below:

Ingest Flow

Figure 21. Ingest Flow


When calculating the amount of data ingested by LogScale we use the following formula:

processed_events = @rawstring + fields + tags - removed fields and tags
  • @rawstring

    This is the length of the message from the request

  • Fields

    Are the keys and values of any additional fields, as fields that were extracted from the rawstring are not counted.

  • Tags

    Are the keys and values of any additional tag, as the tags that are extracted from the rawstrings are not counted.

  • Removed fields and tags (MR)

    In cases where it's not possible to remove data that is not needed before ingesting it, you can filter it out by adjusting a parser setting. See Optimizing Ingestion for more details.

You can measure it by querying your humio-usage repository for ingestAfterFieldRemovalSize. See Measurement Repositories for more details.

Note

Your data needs to have a @timestamp field in order to be searchable in LogScale. This field counts as ingest, unless it's included in the rawstring.