Published on 2021-01-14 00:00:00

Humio 1.19.0

New Query Editor, Packages (Beta)

Version

Type

Release Date

End of Support

Upgrades From

Data Migration

Config. Changes

1.19.0

Preview

2021-01-14

2021-01-28

1.16.0

No

Yes

JAR Checksum

Value

MD5

63d03b5a7d362d1d9a5dfcb5a7d6fcea

SHA1

7ed7776a690ff76afd4ff77ac585a28ef7ee1b2c

SHA256

532dd54bc612b6f771a142899277430469a85c3a431a7824105c1ab69d21974e

SHA512

9edc286d2409cdf36496cc9a7c69ab525ac3207006f1ce3aa194bd17e59e4601f676dbdec0c71cfecd197364b45d83398e5566cebbed82e0a10e1d19ae2e91eb

TGZ Checksum

Value

MD5

63d03b5a7d362d1d9a5dfcb5a7d6fcea

SHA1

7ed7776a690ff76afd4ff77ac585a28ef7ee1b2c

SHA256

532dd54bc612b6f771a142899277430469a85c3a431a7824105c1ab69d21974e

SHA512

9edc286d2409cdf36496cc9a7c69ab525ac3207006f1ce3aa194bd17e59e4601f676dbdec0c71cfecd197364b45d83398e5566cebbed82e0a10e1d19ae2e91eb

Important Information about Upgrading

Beginning with version 1.17.0, if your current version of Humio is not directly able to upgrade to the new version, you will get an error if you attempt to start up the incompatible version. The 1.19.0 release is only compatible with Humio release 1.16.0 and newer. This means that you will have to ensure that you have upgraded at least to 1.16.0 before trying to upgrade to 1.19.0. In case you need to do a rollback, this can also ONLY happen back to 1.16.0 or newer. Rolling directly back to an earlier release can result in data loss.

Packages

This version introduces Humio packages - a way of bundling and sharing assets such as dashboards and parsers. You can create your own packages to keep your Humio assets in Git or create utility packages that can be installed in multiple repositories. All assets can be serialized to YAML files (like what has been possible for dashboards for a while). With tight integration with Humio’s CLI humioctl you can install packages from local disk, URL, or directly from a Github repository. Packages are still in beta, but we encourage you do start creating packages yourself, and sharing them with the community. At Humio we are also very interested in talking with package authors about getting your packages on our upcoming marketplace.

Read more about packages on our Packages documentation.

Humio Insights Application

With the introduction of Humio packages we have created the Humio Insights Application. The application is a collection of dashboards and saved searches making it possible to monitor and observe a Humio cluster.

New Query Editor

The new query editor has a much better integration with Humio’s query language. It will give you suggestions as you type, and gives you inline errors if you make a mistake. We will continue to improve the capabilities of the query editor to be aware of fields, saved queries, and other contextual information.

New Test Function

A new function called test() has been added for convenience. What used to be done like: tmp := <expression> | tmp=true can now be done using: test( <expression> ). Inside <expression> field names appearing on the right hand side of an equality test, such as field1==field2 compares the values of the two fields. When comparing using = at top-level field1=field2 compares the value of field1 against the string "field2". This distinction is a cause of confusion for some users, and using test() simplifies that.

Changes to Internal Logging

We have made small changes to how Humio logs internally. We did this to better support the new Humio Insights Application. We have tried to keep the changes as small and compatible as possible, but we have made some changes that can break existing searches in the humio repository (or other repositories receiving Humio logs). We made these changes as we think they are important in order to improve things moving forward. One of the benefits is the new Humio Insights Application.

Read more about the details of internal logging.

Change Log

  • Introduced humio insights package that is installed per default on startup on the humio repository

  • Raised the limit for note widget text length to 20000

  • Raised the parser test character length to 20000.

  • No longer overwrite the humio parser in the humio repository on startup.

  • Improved app loading logic.

  • Kafka client inside Humio has been bumped from 2.4.1 to 2.6.0.

  • Improve handling of broken local cache files

  • New config QUERY_QUOTA_EXCEEDED_PENALTY with value 50 by default. When set >= 1.0 then this throttles queries from users that are over their quota by this factor rather than stopping their queries. Set to 0 to disable and revert to stopping queries.

  • For ingest using a URL with a repository name in it, Humio now fails ingest if the repository in the URL does not match the repository of the ingest token. Previously, it would just use the repository of the ingest token.

  • “Notifiers” have been renamed to “actions” throughout the UI and in log statements. The REST APIs have not been changed and all message templates can still be used.

  • In the GraphQL API, on the Alert type, the notifiers field has been deprecated and will be removed in a later release. It has been replaced by the actions field.

  • In the GraphQL API, the value ChangeAlertsAndNotifiers on the Permission enum has been deprecated and will be removed in a later release. It has been replaced by the ChangeTriggersAndActions value. The same is true for the ViewAction enum. On the ViewPermissionsType type, the administerAlerts``field has been deprecated and will be removed in a later release. It has been replaced by the ``administerTriggersAndActions field.

  • The built-in json-for-notifier parser used by the Humio Repository action (formerly notifier) is deprecated and will be removed in a later release. It has been replaced by an identical parser with the name json-for-action, see documentation.

  • The built-in bro-json parser is deprecated and will be removed in a later release. It has been replaced by an identical parser with the name zeek-json, see zeek-json.

  • The configuration option IP_FILTER_NOTIFIERS has been renamed to IP_FILTER_ACTIONS. The old name will continue to work.

  • The configuration option HTTP_PROXY_ALLOW_NOTIFIERS_NOT_USE has been renamed to HTTP_PROXY_ALLOW_ACTIONS_NOT_USE. The old name will continue to work.

  • New feature “Scheduled Searches” making it possible to run queries on a schedule and trigger actions (formerly notifiers) upon query results. See documentation.

  • New feature “Event forwarding” making it possible to forward events during ingest out of Humio to a Kafka server. See Event Forwarding documentation. Currently only available for on-prem customers.

  • The Humio Repository action (formerly notifier) now replaces a prefix ‘#’ character in field names with @tag. so that #source becomes @tag.source. This is done to make them searchable in Humio. You can change the name by creating a custom parser. See related documentation for more details.

  • The names of the metadata fields added by the Humio Repository action (formerly notifier) has been changed to accomodate that it can now also be used from scheduled searches. See related documentation for more details.

  • New filter function test( <boolean expression> ) makes it convenient to test complex expressions.

  • New ingest endpoint /api/v1/ingest/raw for ingesting singular webcalls as events. See Ingest API - Raw Data documentation.

  • API Changes (Non-Documented API): getFileContent has been moved to a field on the SearchDomain type.

  • API Changes (Non-Documented API): Queries and Mutations for Parser now expects an id field in place of a name field, when fetching and updating parsers.

  • API Changes (Non-Documented API): Getting Alert by ID has been moved to a field on the SearchDomain type.

  • New feature: “stateless Ingest-only nodes”: A node that the rest of the cluster does not know exists, but is capable of ingesting events into the ingest queue. Enable using NODE_ROLES=ingestonly.

  • unit on timechart (and bucket) now works also when the function within uses nesting and anonymous pipelines.

  • New feature “Custom ingest tokens” making it possible for root users to create ingest tokens with a custom string.

  • The function parseCEF() now deals with extension fields with labels, i.e. cs1=Value cs1Label=Key becomes cef.label.Key=Value.

  • Added asn function for retrieving the ASN number for a given IP address, see asn() reference page.

  • New config AUTO_UPDATE_MAXMIND for enabling/disabling updating of all maxmind databases. Deprecates AUTO_UPDATE_IP_LOCATION_DB, but old config will continue to work.

  • New config MAXMIND_IP_LOCATION_EDITION_ID for selecting the maxmind edition of the IP location database. Deprecates MAXMIND_EDITION_ID, but old config will continue to work.

  • New function hash for computing hashes of fields. See hash() reference page.

  • Renamed LOG4J_CONFIGURATION environment variable to HUMIO_LOG4J_CONFIGURATION. See Configuration reference pages. The old variable will no longer work.

  • New validation when creating an ingest token using the API that the parser, if specified, actually exists in the repository.

  • Humio no longer deletes an existing humio-search-all view if the CREATE_HUMIO_SEARCH_ALL environment variable is false. The view instead becomes deleteable via the admin page.

  • When a host dies and Humio reassigns digest, it will warn if a fallback host is picked that is in the same zone as existing replicas. Eliminate warning if falling back to a host in the null zone.

  • Reduced the number of writes to global on restart, due to merge targets not being properly reused.

  • Cluster management stats now shows segments as underreplicated if they are replicated to enough hosts, but are not present on all configured hosts.

  • Add an error message to the event if the user is trying to redirect it to another repo using #repo, and the target repo is invalid.

  • Make the query functions window() and series() be enabled by default. They can be disabled by setting the configuration options WINDOW_ENABLED and SERIES_ENABLED to false, respectively.

  • The transfer job will delete primary copies shortly after transferring the segments to secondary storage. The copies would previously only be deleted once a full bulk had been moved.

  • Added config option for Auth0 based sign on method: AUTH_ALLOW_SIGNUP defaults to true. The config is forwarded to the auth0 configuration for the lock widget setting: allowSignUp

  • Reduce contention on the query scheduler input queue. It was previously possible for large queries to prevent each other from starting, leading to timeouts.

  • Made cluster nodes log their own version as well as the versions of all other nodes. This makes it easier to tell which versions are running in the cluster.

  • Introduction of the new log file humio-requests.log. Also the log format for the files humio-metrics.log and humio-nonsensitive.log has changed as described above. The guide for sending Humio logs to another Humio cluster has been updated.

  • Custom made saved queries, alerts and dashboards in the humio repository searching for events of the kinds metrics, requests or nonsensitive may need to be modified. This is described in more detail in the Internal Logging documentation.

  • Upgraded Log4j2 from 2.13.3 to 2.14.0.

  • Removed config IDLE_POLL_TIME_BEFORE_DASHBOARD_QUERY_IS_CANCELLED_MINUTES. Queries on dashboards now have the same life cycle as other queries.

  • Updated the permission checks when polling queries. This will results in dashboard links “created by users who are either deleted or lost permissions to the view” to get unauthorized. To list all dashboard links, run this graphql query as root: query { searchDomains {dashboards { readOnlyTokens { createdBy name token } } } }

  • Added mutation to update the runAsUser for a read only dashboard token.

  • Added timeout for TCP ingest listeners. By default the connection is closed if no data is received after 5 minutes. This can be changed by setting TCP_INGEST_MAX_TIMEOUT_SECONDS. See Ingest Listeners documentation.

  • Humio will only allow using Zookeeper for node id assignment (ZOOKEEPER_URL_FOR_NODE_UUID) when configured for ephemeral disks (USING_EPHEMERAL_DISKS). When using persistent disks, there is no need for the extra complexity added by Zookeeper.

  • Fixed a rare issue where a node that was previously assigned digest could write a segment to global, even though it was no longer assigned the associated partition.

  • Fixed an rare issue where the digest coordinator would assign digest fewer hosts than configured.

  • Fixed an issue which caused free-text-search to not work correctly for large (>64KB) events.

  • Fixed crash in CleanupDatasourceFilesJob when examining a file size fails due to that file being deleted concurrently.

  • Fixed an issue where the segment rewrite job handling event deletion might rewrite segments sooner than configured.

  • Fixed an issue where canceling queries could produce a spurious error log.

  • Fixed an issue with the cidr function that would make some IPv4 subnets accept IPv6 addresses and some strings that were not valid IP addresses.

  • Fixed an issue that could cause node id assignment to fail when running on ephemeral disks and using Zookeeper for node id assignment. Nodes in this configuration will now try to pick a new id if their old id has been acquired by another node.

  • Fixed an issue causing Humio to retain deleted minisegments in global for longer than expected.

  • Fixed an issue where unit-conversion (by timechart) did not take effect through groupBy() and window().

  • Fixed an issue causing queries using kvParse() to filter out too much in specific circumstances when filtering on a field assigned before kvParse().

  • Fixed an issue causing queries using kvParse() to be executed incorrectly in certain circumstances when kvParse() assigned fields starting with a non-alphanumeric character.

  • Fixed an issue causing the secondary storage transfer job to select and queue too many segments for transfer at once. The job will now stop and recalculate the bulk to transfer periodically.

  • Fixed an issue with updating user profile, in some situations save failed.

  • Fixed an issue where the filter and groupBy buttons on the search page would not restart the search automatically

  • Fixed logic for when the organization owner panel should be shown in the User’s Danger zone.

  • Fixed an issue where segment merge occasionally reported BrokenSegmentException when merging, while the segments where not broken.

  • Fixed timeout issue in S3 Archiving

  • Fixed bug where repeating queries would not validate in alerts.

  • Fixed a bug where fullscreen mode could end up blank