Removing Fields

Important

The ability to remove fields during parsing is designed to assist in cost optimization when ingesting data.

Where data has been ingested with defined fields (i.e., the event information was submitted as structured data), the metadata fields can be removed from the incoming events during ingest, before parsing.

This feature cannot be used in the following scenarios:

  • Cannot be used to change or remove the original @rawstring

  • Cannot remove fields extracted from @rawstring

  • Can only remove fields that were ingested outside the @rawstring, i.e. from structured data or metadata.

For example, when data has been ingested through the API using the following payload:

json
[
  {
    "fields": {
      "host": "webhost1"
    },
    "messages": [
      "192.168.1.21 - user1 [02/Nov/2017:13:48:26 +0000] \"POST /humio/api/v1/ingest/elastic-bulk HTTP/1.1\" 200 0 \"-\" \"useragent\" 0.015 664 0.015",
      "192.168.1.49 - user1 [02/Nov/2017:13:48:33 +0000] \"POST /humio/api/v1/ingest/elastic-bulk HTTP/1.1\" 200 0 \"-\" \"useragent\" 0.014 657 0.014",
      "192.168.1..21 - user2 [02/Nov/2017:13:49:09 +0000] \"POST /humio/api/v1/ingest/elastic-bulk HTTP/1.1\" 200 0 \"-\" \"useragent\" 0.013 565 0.013",
      "192.168.1.54 - user1 [02/Nov/2017:13:49:10 +0000] \"POST /humio/api/v1/ingest/elastic-bulk HTTP/1.1\" 200 0 \"-\" \"useragent\" 0.015 650 0.015"
    ]
  }
]

Only the defined field, host can be removed using this method. The @rawstring and any fields parsed or extracted from the @rawstring cannot be removed using this method.

When removing fields using this method, the fields specified will be removed before events are parsed. Removed fields will not count as ingest towards your license. See Ingest Usage Management for more information on how ingest is measured.

  1. Go to your repository and click Parsers.

  2. On the Parsers page, click on the required parser. The Parser script editor is displayed.

  3. Click Settings next to Code, then click Fields to remove in the side menu.

    On the Fields to remove page, enter the name of the field to remove and click Save.

    Remove Fields

    Figure 51. Remove Fields


To remove data or fields during ingest that cannot be removed using this method, use the replace() to modify the incoming @rawstring.