Humio Server 1.31.0 Preview (2021-09-27)

VersionTypeRelease DateEnd of SupportUpgrades FromData MigrationConfig. Changes
ValueJAR Checksum

We now distribute Humio as a tarball in addition to the fat jar format we've previously used. We will continue to distribute the fat jar for the time being. The tarball includes a launcher script, which will set a number of JVM arguments for users automatically. We believe this will help users configure Humio for good performance out of the box. For more information, see Launcher Script.

Search performance via hashfilter-first on segments in buckets

Some searches, including regex and literal string matches, now allow searching without fetching the actual segment files from the bucket, in case the segment is only present in the bucket and not on any local disk. Humio now fetches the hash filter file and uses that to decide if the segment file may have a match before downloading the segment file in this case.

Humio packages can now carry scheduled searches, all types of actions, and files with lookup data (either CSV or JSON formatted). Additionally, we have improved the UI for managing packages, to make it easier to find the package you are looking for. This also marks the point where packages are brought out of beta.

Improvements, new features and functionality

  • UI Changes

    • Updated design for Package Marketplace and Installed Packages to make them easier to use and more consistent.

    • Removed the pop-up link to edit an alert or scheduled search when on the form page. This link is only relevant when creating an entity from the search page via a dialog.

    • The signup path was removed, together with the corresponding pages.

    • Fixed some styling issue on Query Quotas page.

    • Identity provider pages style update.

    • The left navigation menu hides, and can be opened again, for mobile devices, on organization settings pages and repository settings pages.

    • Updated the style of the email action template and made the wording used dependent on whether an alert or scheduled search was triggered.

    • Breadcrumbs are aligned across all pages and show the package name with a link when viewing or editing an asset from a package.

    • Cluster management pages style updates.

  • GraphQL API

    • Deprecates the installPackageFromRegistry and updatePackageFromRegistry GraphQL mutations in favor of installPackageFromRegistryV2 and updatePackageFromRegistryV2.

    • Deprecates the copyFile GraphQL mutation, as it is no longer used. If you use this mutation, please let us know.

    • Deprecates the package field on the SearchDomain GraphQL type, in favor of packageV2. The new field has a simpler and more correct return type.

    • Added information about the use of preview fields in the result from calling the GraphQL API. The information will be in the field extensions.preview and will be a list of objects with a name and reason field.

    • Added a GraphQL mutation "cancelDeleteEvents" that allows cancelling a previously submitted deletion. Cancellation is best-effort, and events that have already been deleted will not be restored.

    • The GraphQL DateTime type now supports non-UTC time. Timestamps like 2021-07-18T14:13:09.517+02.0 are now legal, and will be converted to UTC time internally.

    • When using the GraphQL field allowedViewActions, the two previously deprecated actions ChangeAlertsAndNotifiers and ReadEvents are no longer returned. Look for their replacements ChangeTriggersAndActions and ReadContents instead.

    • The fileName, displayName, and location GraphQL fields on the UploadedFileSnapshot type are deprecated in favor of the new nameAndPath field.

    • Deprecates the two GraphQL fields id and contentHash on the File type. The two fields are considered unused, so no alternatives are provided. If you rely on them, please let us know.

    • Extended 'Relative' field type for schema files to include support for the value 'now'.

    • The name, displayName, and location GraphQL fields on the File type are deprecated in favor of the new nameAndPath field.

  • Configuration

    • Added compatibility mode for using IBM Cloud Object Storage as bucket storage via S3_STORAGE_IBM_COMPAT

    • On a node configured as USING_EPHEMERAL_DISKS=true allow the local disk management deleting files even if a query may need them later, as the system is able to re-fetch the files from bucket storage when required. This improves the situation when there are active queries that in total have requested access to more segments than the local disk can hold.

    • The Scheduled Searches feature is no longer in beta and can be used by all users without enabling it first

  • Other

    • Added a Data subprocessors page under account.

    • Package installation error messages are now much more readable.

    • Scheduled search "schedule" is explained using human readable text such as "At 9.30 on Tuesdays".

    • Fixed an issue with using the browser back button while "advanced editing" the query text of a scheduled search or an alert would hide the blue bar that allows saving the query.

    • Added focus states to text field, selection and text area components.

    • Added explicit distribution information for elastic bulk API for elasticsearch API compatibility.

    • Added support for importing packages with CSV and JSON files. Exporting packages with files is not fully supported yet, but will be in a future release.

    • Warnings when running scheduled searches now show up as errors in the scheduled search overview page if [SCHEDULED_SEARCH_DESPITE_WARNINGS]( is set to 'false' (the default).

    • Allow launching using JDK-16.

    • Improved handling of local disk space relative to `LOCAL_STORAGE_MIN_AGE_DAYS`: When the local disk would overflow by respecting that config, Humio can now delete the oldest local segments that are present in bucket storage, even when they are within that time range.

    • When selecting actions for alerts or scheduled searches, the actions are now grouped by the package they were imported from.

    • Improved audit log for organization creation.

    • Scheduled search "schedule" field is now validated, showing accurate help for each part of the crontab expression.

    • The test action functionality no longer uses alert terminology, as actions can be invoked from both alerts and scheduled searches. Also, it is now possible to also test the scheduled search specific message templates using it.

    • Improved error handling when running scheduled searches, so that a failed schedules search will be retried as long as it is within the [backfill limit](

    • Humio docker images is now based on the Alpine linux.

    • Added Dark Mode for Query Monitor page.

    • Added support for including dashboard and alert labels when exporting a package.

    • You can now export and import packages containing any of the action types: Webhook, Email, Humio Repo, Pager Duty, Slack, Slack multi channel, Ops Genie and Victor Ops.

    • Improved performance in IP database lookups for the functions ipLocation(), asn() and worldMap().

    • Improved search for users page.

    • Added maximum width to tabs on the Group page, so they do not keep expanding forever.

    • Added loading and error states to the page where user selects to create a new repository or view.

Bug Fixes

  • Other

    • Fixed an issue where the job responsible for deleting segment files off nodes was not deleting as many segments as it should.

    • The DiskSpaceJob now removes newly written backfilled segments off the local disk before it chooses to remove non-backfilled segments.

    • When a search is able to filter out segments based on the hash filter files, and a segment file is not present locally on any node, fetch only the hash filter at first, evaluate that, and only if required, fetch the segment file. This speeds up searches that target segments only present in bucket storage and that have search filters that generate hash filter checks, such as regex and literal text comparisons.

    • Fixed an issue that caused some errors to be hidden behind a message about "internal error".

    • When accessing Humio through a URL with either a repository or view name in it and using an ingest token, it is now checked that the view on the token matches the repository or view in the URL, and a 403 Forbidden status is returned, if not.

    • Fixed a bug which potentially have caused alerts to not re-fire after the throttle period for field-based throttling had passed.

    • Reworded a confusing error message when using the top() function with a limit parameter exceeding the limits configured with TOP_K_MAX_MAP_SIZE_HISTORICAL or TOP_K_MAX_MAP_SIZE_LIVE.

    • Fixed an issue where it was possible to submit queries to the Delete Events API that were not valid for that API. Only pure filtering queries are allowed.

    • Amended an internal limit on how many segments can be fetched from bucket storage concurrently. The old limit was based on the number of running queries. The new limit is 32.

    • Fixed thread safety for a variable involved in fetching from bucket storage for queries.

    • Fixed an issue where Humio attempted to fetch global from other nodes before TLS was initialized.

    • Introduced a check for compatibility for packages and humio versions.

    • Fixed an issue where new groups added to a repository got a query prefix that disallowed search. The default is now to allow search with the queryprefix *.

    • Fixed an issue where certain problems highlighted the first word in a query, not the location of the problem.

    • Fixed an issue where Shift+Enter would select the current completion rather than adding a newline.

    • Fixed an issue that could cause UploadedFileSyncJob to crash if an uploaded file went missing.

    • The simple and advanced permission model has been merged, thus allowing users who were using the simple permission model to create their own permission roles and groups, create groups with default roles, and all other features that were previously only available in advanced permissions mode.

    • Updated Elastic ingest endpoint to accept 'create' operations in addition to 'index' operations. Both operation types result in the same ingest behavior. This update was added as Fluent-Bit v1.8.3 began using the 'create' operation rather than 'index' for ingest.

    • Global snapshots are now uploaded to bucket storage more often when there are a lot of updates to it, leading to shorter replay times on startup.

    • Fixed an issue where Humio would create auxiliary files (hash files) for segments unnecessarily when moving segments between nodes.

    • Fixed a bug causing the disk space job to use an expensive code path even when a cheaper one was available.

    • Updated Slack action for messaging multiple channels, so it propogates errors when triggered. Previously errors were ignored.

    • Fixed an issue that could cause cluster nodes to crash when growing the number of digest partitions.

    • Fixed an issue where, looking at GraphiQL, the dropdown from the navigation menu was partially hidden.

    • Fixed an issue where metrics of type gauge with a double value were not reported to the humio-metrics repository, but only to the humio repository.

    • Truncate long user names on the Users page.

    • Split package export page into dialog with multiple steps.

    • Creating a new dashboard now opens it after creation.

    • Fixed an issue where top() with max= can yield the same key multiple times (ei. ...| top([queryId, query], max=totalSize)).

    • Fixed an issue with the split() function which caused incorrect (usually, too few) query results in some cases where the output fields were refered to later in the query.

    • Updated the examples on how to use the match query function in the inline documentation.

    • Fixed an issue where the global consistency check job would fail to perform the consistency check, instead logging lines like "Global dump requested but global had expired". This line can still occur, but only when the consistency check takes too long.

    • Fixed an issue where the job responsible for deleting segment files off nodes was not running as often as expected.

    • Fixed an issue where the DiskSpaceJob could continue tracking segments if they were deleted from global, but the files were still present locally.

    • Fixed an issue where the DiskSpaceJob could mark segments accessed slightly out of order during boot.

    • Fixed a bug on queries that triggered an error while executing due to the input (such as a regex that exceeds limits on execution time) could result in the client getting 404 as status on poll, where it should get .0.

    • The DiskSpaceJob no longer initializes based off of the segment last-modified timestamp, this only happens if no access order snapshot is stored locally. If a snapshot is present, we trust that.

    • Fixed a bug where a 404 Not Found status on an internal endpoint would be incorrectly reported as an 401 Unauthorized.

    • Security when viewing installed packages and packages on the marketplace are now less strict. Permissions are still required for installing and uninstalling packages.

    • Fixed a bug where a hidden field named "#humioAutoShard" would sometimes show up in the field list.

    • Cloning an asset now redirects you to the edit page for the asset for all assets.

    • Updated dependencies with security fixes.

    • Removed an old Cloud Signups page. The page is not necessary since organizations were implemented for the Cloud environments.

    • Updated the new asset dialog button text so that it will say 'Continue' when an asset will not be created directly.

    • Fixed an issue that caused some metrics of type gauge to be reported with a wrong value.

    • Fixed an issue where the {time_zone} message template for actions would show a full description of the scheduled search instead of only the time zone.

    • Fixed an issue where the query scheduler would spend too much time "shelving" queries, and not enough on getting them executed, leading to little progress on queries.

    • Fixed an issue where Humio would create a broken hash file for the merge result when merging minisegments that did not originally have hash files.