Published on 2020-11-18 00:00:00

Humio 1.17.0

Repeating Queries, Search Result Caching and New Query Functions

Version

Type

Release Date

End of Support

Upgrades From

Data Migration

Config. Changes

1.17.0

Preview

2020-11-18

2020-11-26

1.16.0

No

Yes

JAR Checksum

Value

MD5

9c47009810132eb67177966d10471544

SHA1

341a2607265420b1a2d461ad111c88aca06419ed

SHA256

59001d9a738930f8774c11864ce3c36b3b4b4003f8cad0d4b36af13275ceefa9

SHA512

ee8e2837a8fba2b28aab8d18b16269f452f19a5c80ad9826ecbeb73eaa062236acdf2191cc448e60d81390f1a2881374637adce7603ebf9f0861ca6246f6c82f

TGZ Checksum

Value

MD5

9c47009810132eb67177966d10471544

SHA1

341a2607265420b1a2d461ad111c88aca06419ed

SHA256

59001d9a738930f8774c11864ce3c36b3b4b4003f8cad0d4b36af13275ceefa9

SHA512

ee8e2837a8fba2b28aab8d18b16269f452f19a5c80ad9826ecbeb73eaa062236acdf2191cc448e60d81390f1a2881374637adce7603ebf9f0861ca6246f6c82f

Important Information about Upgrading

Beginning with version 1.17.0, if your current version of Humio is not directly able to upgrade to the new version, you will get an error if you attempt to start up the incompatible version. The 1.17.0 release is only compatible with Humio release 1.16.0 and newer. This means that you will have to ensure that you have upgraded to minimum 1.16.0 before trying to upgrade to 1.17.0. In case you need to do a rollback, this can also ONLY happen back to 1.16.0 or newer, rolling directly back to earlier release can result in data loss.

Repeating Queries

Humio can now run repeating queries using the beta:repeating() function. These are live queries that are implemented by repeatedly making a query. This allows using functions in alerts and dashboards that typically do not work in live queries, such as selfJoin() or selfJoinFilter(). See the beta:repeating() reference page for more information.

Improved Security for Alerts

In order to prevent alert notifiers being used to probe services on the internal network (eg. Zookeeper or the AWS metadata service), Humio now has an IP filter on alert notifiers. The default is to block access to all link-local addresses and any addresses on the internal network; however, you can opt-in to the old behavior by setting the configuration option IP_FILTER_NOTIFIERS to allow all. See IP Filter documentation.

New Experimental Query Function series()

A new experimental query function called series() has been added. It needs to be explicitly enabled on the cluster using the config option SERIES_ENABLED=true.

The function series() improves upon session() and collect() for grouping events into transactions. What used to be done with groupby(id, function=session(function=collect([fields, …]))) can now be done using groupby(id, function=series([fields, ..])).

See series() reference page for more details.

Search Result Caching

This new feature stores a copy of live search results to the local disk in the server nodes, and reuses the relevant parts of that cached result when an identical live search is later started. Caching is controlled with the config option QUERY_CACHE_MIN_COST, which has a default value of 1000. To disable caching, set the config option to a very high number, such as 9223372036854775807 (max long value).

Change Log

  • New config option IP_FILTER_NOTIFIERS, to set up IP filters for Alert Notifications, see IP Filter reference page.

  • New config option ENABLE_ALERTS makes it possible to disable alerts from running (enabled by default).

  • New config option ALERT_DESPITE_WARNINGS makes it possible to trigger alerts even when warnings occur.

  • New config option DEFAULT_MAX_NUMBER_OF_GLOBALDATA_DUMPS_TO_KEEP.

  • New query function parameter to parseJson, removePrefixes, see parseJson() reference page.

  • New query function concatArray, see concatArray() reference apge.

  • New query function parseCEF used to parse events which are formatted according to the Common Event Format(CEF), see parseCEF().

  • New experimental query function beta:repeating(), see beta:repeating() reference page.

  • New experimental query function series(), enabled by config option SERIES_ENABLED=true, see series() reference page.

  • New experimental query function window(), enabled by config option WINDOW_ENABLED=true, see window() reference page.

  • The {events_html} notifier template will now respect the field order from the query.

  • It is again possible to override a built-in parser in a repository by creating a parser with the same name.

  • Periodically release object pools used by mapper pipeline, to avoid a possible source of memory leaks.

  • Fix negating join expressions.

  • Fixes a bug where join function in some circumstances would fetch subquery results from other cluster nodes more than once.

  • Setting the default query for a view in the UI has been moved from the “Save as Query” to the View’s “Settings” tab.

  • The notifier list is sorted when selecting notifiers for an alert.

  • Improved wording of diagnostics regarding function arguments.

  • Tweaked location of diagnostics regarding missing function arguments.

  • API Changes (Non-Documented API): Saved Query REST API has been replaced by GraphQL.

  • API Changes (Non-Documented API): View Settings REST API has been replaced by GraphQL.

  • Allow running Humio on JDK-14 and JDK-15 to allow testing these new builds.

  • Free-text search has been fixed to behave more in line with the specification.

  • Refuse to boot if the global topic in Kafka does not contain the expected starting offset.

  • Crash the node if an exception occurs while reading from the global Kafka topic, rather than trying to recover.

  • Reduce the max fetch size for Kafka requests, as the previous size would sometimes lead to request timeouts.

  • Improve logic attempting to ensure other live nodes can act as substitutes in case the preferred digest nodes are not available when writing new segments.

  • Fixes the case where datasources receiving data might not be marked idle, causing Humio to retain too much ingest data in Kafka.

  • Fixes the case where Humio would consider local node state when deciding which ingest data was safe to delete from Kafka.

  • Fix several cases where Humio might attempt to write a message to Kafka larger than what Kafka will allow.

  • Fixes the issue where Humio could behave incompatibly with Kafka versions prior to 2.3.0 if KAFKA_MANAGED_BY_HUMIO was true.

  • Refuse to boot if the booting node would cause violations of the “Minimum previous Humio version” as listed in the release notes.

  • Fixes an issue which caused free-text-search to not work correctly for large (>64KB) events.

  • Fixes a bug where unit:convert couldn’t handle numbers in scientific notation.

  • Rename a few scheduler threads so they reflect whether they’re associated with streaming queries (“streaming-scheduler”) or not (“normal-scheduler”)

  • Fixes an issue where Humio might try to get admin access to Kafka when KAFKA_MANAGED_BY_HUMIO was false.

  • If KAFKA_MANAGED_BY_HUMIO is true, Humio will ensure unclean leader election is disabled on the global-events topic.

  • The Humio-search-all view will no longer be removed if CREATE_HUMIO_SEARCH_ALL is set to false. The view will instead become possible to delete manually via the admin UI.

  • Reduce the number of merge target updates Humio will write to global on digest leader reassignment or reboot.

  • Fixes a bug causing join() to not work after an aggregating function.

  • Fixes a bug causing sort()/head()/tail() to work incorrectly after other aggregating functions.

  • Fixes a bug causing the sub-queries of join() etc. to not see events with an @ingesttimestamp occurring later than the search time interval.

  • Fixes an issue causing Humio to fail to upload files to bucket storage in rare cases.

  • Switch from JDK to BouncyCastle provider for AES decrypt to reduce memory usage.

  • Changed default TLS ciphers and protocols accepted by Humio, see TLS Configuration reference page.