Humio Server 1.19.0 Preview (2021-01-14)
Version | Type | Release Date | End of Support | Upgrades From | JDK Compatibility | Data Migration | Config. Changes |
---|---|---|---|---|---|---|---|
1.19.0 | Preview | 2021-01-14 | 2021-01-28 | 1.16.0 | 11 | No | Yes |
JAR Checksum | Value |
---|---|
MD5 | 63d03b5a7d362d1d9a5dfcb5a7d6fcea |
SHA1 | 7ed7776a690ff76afd4ff77ac585a28ef7ee1b2c |
SHA256 | 532dd54bc612b6f771a142899277430469a85c3a431a7824105c1ab69d21974e |
SHA512 | 9edc286d2409cdf36496cc9a7c69ab525ac3207006f1ce3aa194bd17e59e4601f676dbdec0c71cfecd197364b45d83398e5566cebbed82e0a10e1d19ae2e91eb |
Important Information about Upgrading
Beginning with version 1.17.0, if your current version of Humio is not directly able to upgrade to the new version, you will get an error if you attempt to start up the incompatible version. The 1.19.0 release is only compatible with Humio release 1.16.0 and newer. This means that you will have to ensure that you have upgraded at least to 1.16.0 before trying to upgrade to 1.19.0. In case you need to do a rollback, this can also ONLY happen back to 1.16.0 or newer. Rolling directly back to an earlier release can result in data loss.
Deprecation
Items that have been deprecated and may be removed in a future release.
New config
MAXMIND_IP_LOCATION_EDITION_ID
for selecting the maxmind edition of the IP location database. DeprecatesMAXMIND_EDITION_ID
, but old config will continue to work.
Improvements, new features and functionality
Other
Stateless Ingest-only nodes: A node that the rest of the cluster does not know exists, but is capable of ingesting events into the ingest queue. Enable using
NODE_ROLES=ingestonly
.Custom ingest tokens making it possible for root users to create ingest tokens with a custom string.
Bug Fixes
Configuration
New config
AUTO_UPDATE_MAXMIND
for enabling/disabling updating of all maxmind databases. DeprecatesAUTO_UPDATE_IP_LOCATION_DB
, but old config will continue to work.New config
QUERY_QUOTA_EXCEEDED_PENALTY
with value 50 by default. When set >= 1.0 then this throttles queries from users that are over their quota by this factor rather than stopping their queries. Set to 0 to disable and revert to stopping queries.
Functions
New function
hash()
for computing hashes of fields. Seehash()
reference page.Make the query functions
window()
andseries()
be enabled by default. They can be disabled by setting the configuration optionsWINDOW_ENABLED
andSERIES_ENABLED
tofalse
, respectively.Fixed an issue causing queries using
kvParse()
to filter out too much in specific circumstances when filtering on a field assigned beforekvParse()
.Added a new function for retrieving the ASN number for a given IP address, see
asn()
reference page.Fixed an issue with the
cidr()
function that would make some IPv4 subnets accept IPv6 addresses and some strings that were not valid IP addresses.Fixed an issue where unit-conversion (by timechart) did not take effect through
groupBy()
andwindow()
.Fixed an issue causing queries using
kvParse()
to be executed incorrectly in certain circumstances whenkvParse()
assigned fields starting with a non-alphanumeric character.
Other
Updated the permission checks when polling queries. This will results in dashboard links "created by users who are either deleted or lost permissions to the view" to get unauthorized. To list all dashboard links, run this graphql query as root:
logscalequery { searchDomains {dashboards { readOnlyTokens { createdBy name token } } } }
Raised the limit for note widget text length to .00
Fixed an issue that could cause node id assignment to fail when running on ephemeral disks and using Zookeeper for node id assignment. Nodes in this configuration will now try to pick a new id if their old id has been acquired by another node.
Custom made saved queries, alerts and dashboards in the humio repository searching for events of the kinds metrics, requests or nonsensitive may need to be modified. This is described in more detail in LogScale Internal Logging.
Fixed an issue where segment merge occasionally reported BrokenSegmentException when merging, while the segments where not broken.
In the GraphQL API, the value
ChangeAlertsAndNotifiers
on thePermission
enum has been deprecated and will be removed in a later release. It has been replaced by theChangeTriggersAndActions
value. The same is true for theViewAction
enum. On theViewPermissionsType
type, theadministerAlerts
field has been deprecated and will be removed in a later release. It has been replaced by theadministerTriggersAndActions
field.The Humio Repository action (formerly notifier) now replaces a prefix '#' character in field names with @tag. so that
#source
becomes@tag.source
. This is done to make them searchable in Humio. You can change the name by creating a custom parser. See Falcon LogScale Repository.Fixed an issue causing Humio to retain deleted minisegments in global for longer than expected.
Add an error message to the event if the user is trying to redirect it to another repo using #repo, and the target repo is invalid.
When a host dies and Humio reassigns digest, it will warn if a fallback host is picked that is in the same zone as existing replicas. Eliminate warning if falling back to a host in the null zone.
New feature "Scheduled Searches" making it possible to run queries on a schedule and trigger actions (formerly notifiers) upon query results. See Scheduled Searches.
Fixed logic for when the organization owner panel should be shown in the User's Danger zone.
Introduction of the new log file
humio-requests.log
. Also the log format for the fileshumio-metrics.log
andhumio-nonsensitive.log
has changed as described above. See Log LogScale to LogScale.Upgraded Log4j2 from 2.13.3 to 2.14.0.
New validation when creating an ingest token using the API that the parser, if specified, actually exists in the repository.
Reduced the number of writes to global on restart, due to merge targets not being properly reused.
unit
on timechart (and bucket) now works also when the function within uses nesting and anonymous pipelines.Fixed an issue which caused free-text-search to not work correctly for large (>64KB) events.
New filter function
test()
.Fixed an issue causing the secondary storage transfer job to select and queue too many segments for transfer at once. The job will now stop and recalculate the bulk to transfer periodically.
The function
parseCEF()
now deals with extension fields with labels, i.e.cs1=Value cs1Label=Key
becomescef.label.Key=Value
.Fixed an issue with updating user profile, in some situations save failed.
Fixed a bug where fullscreen mode could end up blank
Improved app loading logic.
Fixed a rare issue where a node that was previously assigned digest could write a segment to global, even though it was no longer assigned the associated partition.
Added config option for Auth0 based sign on method:
AUTH_ALLOW_SIGNUP
defaults to true. The config is forwarded to the auth0 configuration for the lock widget setting: allowSignUpFixed bug where repeating queries would not validate in alerts.
Fixed an rare issue where the digest coordinator would assign digest fewer hosts than configured.
New feature "Event forwarding" making it possible to forward events during ingest out of Humio to a Kafka server. See Event Forwarding documentation. Currently only available for on-prem customers.
The built-in
json-for-notifier
parser used by the Humio Repository action (formerly notifier) is deprecated and will be removed in a later release. It has been replaced by an identical parser with the namejson-for-action
, see json-for-action.The transfer job will delete primary copies shortly after transferring the segments to secondary storage. The copies would previously only be deleted once a full bulk had been moved.
Introduced humio insights package that is installed per default on startup on the humio repository.
Raised the parser test character length to .00.
For ingest using a URL with a repository name in it, Humio now fails ingest if the repository in the URL does not match the repository of the ingest token. Previously, it would just use the repository of the ingest token.
Fixed timeout issue in S3 Archiving
The built-in
bro-json
parser is deprecated and will be removed in a later release. It has been replaced by an identical parser with the namezeek-json
, see zeek-json.Improve handling of broken local cache files
In the GraphQL API, on the
Alert
type, thenotifiers
field has been deprecated and will be removed in a later release. It has been replaced by theactions
field.API Changes (Non-Documented API): Getting Alert by ID has been moved to a field on the SearchDomain type.
Fixed an issue where canceling queries could produce a spurious error log.
Fixed an issue where the filter and groupBy buttons on the search page would not restart the search automatically
Reduce contention on the query scheduler input queue. It was previously possible for large queries to prevent each other from starting, leading to timeouts.
Added mutation to update the runAsUser for a read only dashboard token.
Kafka client inside Humio has been bumped from 2.4.1 to 2.6.0.
Humio will only allow using Zookeeper for node id assignment (
ZOOKEEPER_URL_FOR_NODE_UUID
) when configured for ephemeral disks (USING_EPHEMERAL_DISKS
). When using persistent disks, there is no need for the extra complexity added by Zookeeper.New ingest endpoint
/api/v1/ingest/raw
for ingesting singular webcalls as events. See Ingest API - Raw Data documentation.The names of the metadata fields added by the Humio Repository action (formerly notifier) has been changed to accomodate that it can now also be used from scheduled searches. See Falcon LogScale Repository.
Removed config
IDLE_POLL_TIME_BEFORE_DASHBOARD_QUERY_IS_CANCELLED_MINUTES
. Queries on dashboards now have the same life cycle as other queries.Notifiers have been renamed to Actions throughout the UI and in log statements. The REST APIs have not been changed and all message templates can still be used.
Made cluster nodes log their own version as well as the versions of all other nodes. This makes it easier to tell which versions are running in the cluster.
Fixed crash in CleanupDatasourceFilesJob when examining a file size fails due to that file being deleted concurrently.
API Changes (Non-Documented API):
getFileContent
has been moved to a field on the SearchDomain type.No longer overwrite the humio parser in the humio repository on startup.
The configuration option
HTTP_PROXY_ALLOW_NOTIFIERS_NOT_USE
has been renamed toHTTP_PROXY_ALLOW_ACTIONS_NOT_USE
. The old name will continue to work.The configuration option
IP_FILTER_NOTIFIERS
has been renamed toIP_FILTER_ACTIONS
. The old name will continue to work.Cluster management stats now shows segments as underreplicated if they are replicated to enough hosts, but are not present on all configured hosts.
Added timeout for TCP ingest listeners. By default the connection is closed if no data is received after 5 minutes. This can be changed by setting
TCP_INGEST_MAX_TIMEOUT_SECONDS
. See Ingest Listeners.API Changes (Non-Documented API): Queries and Mutations for Parser now expects an
id
field in place of aname
field, when fetching and updating parsers.Renamed
LOG4J_CONFIGURATION
environment variable toHUMIO_LOG4J_CONFIGURATION
. See Configuration Settings.Fixed an issue where the segment rewrite job handling event deletion might rewrite segments sooner than configured.
Humio no longer deletes an existing humio-search-all view if the
CREATE_HUMIO_SEARCH_ALL
environment variable is false. The view instead becomes deleteable via the admin page.