Humio Server 1.51.0 LTS (2022-08-15)
Version? | Type? | Release Date? | Availability? | End of Support | Security Updates | Upgrades From? | Config. Changes? |
---|---|---|---|---|---|---|---|
1.51.0 | LTS | 2022-08-15 | Cloud | 2023-08-31 | No | 1.30.0 | No |
JAR Checksum | Value |
---|---|
MD5 | af0041ec27647291086073dc83628bc2 |
SHA1 | 210e924a2863c3b28d08659241c0a283482c12b8 |
SHA256 | fddcb2184d76d5cfefb2f6a02705f66764d8972fb2e664ef7115353344d05680 |
SHA512 | 2b230a836e18d13a282ea28be23130897876e6cbe0bfa30a71a3795a7704255cbdf1c19eadc094cd970e58e0234c540a6b469266765d736fc6435c0a01bbda1e |
Docker Image | SHA256 Checksum |
---|---|
humio | c5db5fac0b03adf9039c31ef4ba69c49356b230b01e0eab0b16c477768dd52af |
humio-core | 36dbe1d90534d2ca72bdadb4992397aa60d2315565ce7a3c7c272a48617cf759 |
kafka | 92de8c0d092fe5c04cc10e076a9bb183c8f3136b9813f3ae74b423ae090cb0d8 |
zookeeper | 85620182a8c5f91426e0d138ca21947975c99a16fbb00f9e7d0c0ca7a8e94d2a |
Download: https://repo.humio.com/repository/maven-releases/com/humio/server/1.51.0/server-1.51.0.tar.gz
Bug fixes and updates.
Removed
Items that have been removed as of this release.
API
The deprecated REST API for actions has been removed, except for the endpoint for testing an action.
The deprecated REST API for parsers has been removed.
Deprecation
Items that have been deprecated and may be removed in a future release.
Deprecated enabledFeatures query. Use the new featureFlags query instead.
New features and improvements
Falcon Data Replicator
FDR polling is now turned on by default. Whether FDR polling should be turned on or off on a node can be configured using the
ENABLE_FDR_POLLING_ON_NODE
configuration variable.If an S3 file is found to be incorrectly formatted during FDR ingest, it will not be ingested completely, but an attempt is made to ingest the remaining S3 files of the SQS message.
If an S3 file cannot be found during FDR ingest, it will not be ingested, but an attempt is made to ingest the remaining S3 files of the SQS message.
Added environment variable
FDR_USE_PROXY
which makes the fdr job use the proxy settings specified with:HTTP_PROXY_*
environment variables.
UI Changes
The design of the Time Selector has been updated, and it now features an button on the dashboard page. See Time Interval Settings.
Field columns now support multiple formatting options. See Formatting Columns for details.
Add missing accessibility features to the login page.
Toggle switches anywhere in the UI are now tabbable and can be accessed using the keyboard.
In lists of users, with user avatars containing user initials, the current user would sometimes appear to have an opening parenthesis as their last initial.
The Live checkbox is now no longer checked automatically when changing the value of the time window in the Time Selector. See Changing Time Interval for details.
If Humio fails to start because the cluster is being upgraded, a dedicated message will show when launching the UI.
The Save As... button is now always displayed on the Search page, see it described at Saving Searches.
Improved keyboard accessibility for creating repositories and views.
New styling of errors on search and dashboard pages.
Adds an icon and a hint to a disabled side navigation menu item that tells the user the reason for it being disabled.
When editing an email action in the UI and adding multiple recipients, it is now possible to add a space after the comma in the comma-separated list of recipients.
Documentation
All documentation links have been updated after the documentation site has been restructured. Please contact support, if you experience any broken links.
Automation and Alerts
Fixed a bug where an alert with name longer than 50 characters could not be edited.
GraphQL API
Added preview fields
isClusterBeingUpdated
andminimumNodeVersion
to the GraphQL Cluster object type.Added a new dynamic configuration flag
QueryResultRowCountLimit
that globally limits how many results (events) a query can return. This flag can be set by administrators through GraphQL. See Limits & Standards for more details.The GQL API mutation updateDashboard has been updated to take a new argument updateFrequency which can currently only be
NEVER
orREALTIME
, which correspond respectively to "dashboard where queries are never updated after first completion" and "dashboard where query results are updated indefinitely".Expose a new GraphQL type with feature flag descriptions and whether they are experimental.
Added a GraphQL mutation for testing an action. It is still in preview, but it will replace the equivalent REST endpoint soon.
Improved error messaging of GraphQL queries and mutations for alerts, scheduled searches and actions in cases where a given repository or view cannot be found.
Configuration
Detect need for higher autoshard count by monitoring ingest request flow in the cluster. Dynamically increase the number of autoshards for each datasource to keep flow on each resulting shard below approximately 2MB/s. New dynamic configuration for this that sets the target maximum rate of ingest for each shard of a datasource:
TargetMaxRateForDatasource
. Default value is 2000000 (2 MB).Added a new environment variable
GLOB_MATCH_LIMIT
which sets the maximum number of rows for csv_file inmatch(..., file=csv_file, glob=true)
function. PreviouslyMAX_STATE_SIZE
was used to determine this limit. The default value of this variable is 20000. If you've changed the value ofMAX_STATE_SIZE
, we recommend that you also changeGLOB_MATCH_LIMIT
to the same value for a seamless upgrade.Default value of configuration variable
S3_ARCHIVING_WORKERCOUNT
raised from1
to(vCPU/4)
.Added a new dynamic configuration
GroupDefaultLimit
. This can be done through GraphQL. See Limits & Standards for details. If you've changed the value ofMAX_STATE_LIMIT
, we recommend that you also changeGroupDefaultLimit
andGroupMaxLimit
to the same value for a seamless upgrade, seegroupBy()
for details.Introduced new dynamic configuration
LiveQueryMemoryLimit
. It can be set using GraphQL. See Limits & Standards for details.Introduced new dynamic configuration
JoinRowLimit
. It can be set using GraphQL and can be used as an alternative to the environment variableMAX_JOIN_LIMIT
. If theJoinRowLimit
is set, then its value will be used instead ofMAX_JOIN_LIMIT
. If it is not set, thenMAX_JOIN_LIMIT
will be used.Adds a new metric for measuring the merge latency, which is defined as the latency between the last minisegment being written in a sequence with the same merge target, and those minisegments being merged. The metric name is
segment-merge-latency-ms
.Introduced new dynamic configuration
StateRowLimit
. It can be set using GraphQL. See Limits & Standards for details.Improve the error message if Humio is configured to use bucket storage, but the credentials for the bucket are not configured.
Change default value for configuration
AUTOSHARDING_MAX
from 16 to 128.Add environment variable
EULA_URL
to specificy url for terms and conditions.New configurations
BUCKET_STORAGE_SSE_COMPATIBLE
that makes bucket storage not verify checksums of raw objects after uploading to an S3. This option is turned on automatically is KMS is enabled (seeS3_STORAGE_KMS_KEY_ARN
) but is available directly here for use with other S3 compatible providers where verfying even content length does not work.Mini segments usually get merged if their event timestamps span more than
MAX_HOURS_SEGMENT_OPEN
. Mini segments created as part of backfilling did not follow this rule, but will now get merged if their ingest timestamps span more thanMAX_HOURS_SEGMENT_OPEN
.Added a link to humio-activity repository for debugging IDP configurations to the page for setting up the same.
Bucket storage now has support for a new format for the keys (file names) for the files placed in the bucket. When the new format is applied, the listing of files only happens for the prefixes "tmp/" and "globalsnapshots/". This helps products such a "HCP". The new format is applied only to buckets created after the dynamic configuration
BucketStorageKeySchemeVersion
has been set to "2". Existing cluster can start using the new format for new files by setting this dynamic configuration. The change will take effect after restarting the cluster. When creating a new Humio cluster, the new format is the default. The new format is supported only on Humio version 1.41+.Introduced new dynamic configuration
GroupMaxLimit
. It can be set using GraphQL. See Limits & Standards for details.Support for KMS on S3 bucket for Bucket Storage. Specify full ARN of the key. The
key_id
is persisted in the internalBucketEntity
so that a later change of the ID of the key to use for uploads will make Humio still refer the old keyID when downloading files uploaded using the previous key. Setting a new value for the target key results in a fresh internal bucket entity to track which files used kms and which did not. For simplicity it is recommended to not mix KMS and non-KMS configurations on the same S3 bucket.New configuration variable
S3_STORAGE_KMS_KEY_ARN
that specifies the KMS key to use.New configuration variable
S3_STORAGE_2_KMS_KEY_ARN
for 2nd bucket key.New configuration variable
S3_RECOVER_FROM_KMS_KEY_ARN
for recovery bucket key.
New file format for files uploaded to bucket storage that allows files larger than 2GB to be written to bucket storage. This may be turned on by setting the dynamic configuration
BucketStorageWriteVersion
to3
. When creating a new Humio clusters, the new format is the default. The new format is supported only on Humio version 1.41+.Adds a new logger job that logs the age of an unmerged miniSegment if the age exceeds the threshold set by the env variable
MINI_SEGMENT_MAX_MERGE_DELAY_MS_BEFORE_WARNING
. The default value ofMINI_SEGMENT_MAX_MERGE_DELAY_MS_BEFORE_WARNING
is 2 xMAX_HOURS_SEGMENT_OPEN
.MAX_HOURS_SEGMENT_OPEN
defaults to 24 hours. The error log produced looks like:Oldest unmerged miniSegment is older than the threshold thresholdMs={value} miniSegmentAgeMs={value} segment={value}
.Introduced new dynamic configuration
QueryMemoryLimit
. It can be set using GraphQL. See alsoLiveQueryMemoryLimit
for live queries. For more details, see Limits & Standards.
Dashboards and Widgets
Applied stylistic changes for the Inspect Panel used in Widget Editor.
Dashboards can now be configured to not update after the initial search has completed. This mode is mainly meant to be used when a dashboard is interactive and not for wall-mounted monitors that should update continually. The feature can be accessed from the Dashboard properties panel when a dashboard is put in edit-mode. See Working in Edit Mode.
Bar Chart
widget:The Y-axis can now start at smaller values than 1 for logarithmic scales, when the data contain small enough values.
It now has an
Auto
setting for the Input Data Format property, see Wide or Long Input Format for details.Now works with bucket query results.
Added empty states for all widget types that will be rendered when there are no results.
When importing existing dashboard with a static Shared time, recent changes in the time selection would make those dashboards live.
The
Pie Chart
widget now uses the first column for the series as a fall back option.The Dashboard page now displays the current cluster status.
Note
widget:Default background color is now
Auto
.Introduced the text color configuration option.
Sorting of
Pie Chart
widget categories, descending by value. Categories grouped asOthers
will always be last.Single Value
widget:Missing buckets are now shown as gaps on the sparkline.
Isolated data points are now visualised as dots on the sparkline.
The widget legend column width is now based on the custom series title (if specified) instead of the original series name.
Introducing the
Heatmap
widget that visualizes aggregated data as a colorised grid.The
Normalize
option for theWorld Map
widget has been replaced by a third magnitude mode namedNone
, which results in fixed size and opacity for all marks.Table
widgets will now break lines for newline characters in columns.Better handling of dashboard connections issues during restarts and upgrades.
Pie Chart
widget now uses the first column for the series as a fall back option.Single Value
widget new configuration: deprecated fielduse-colorised-thresholds
in favor ofcolor-method
.Single Value
widget Editor: the configuration optionEnable Thresholds
is being replaced by an option calledMethod
under the Colors section.
Log Collector
The Log Collector download page has been enabled for on-prem deployments.
Functions
Added validation to the
field
andkey
parameters of thejoin()
function, so empty lists will be rejected with a meaningful error message.The
groupBy()
function now acceptsmax
as value for thelimit
parameter, which sets the limit to the largest allowed value (as configured by the dynamic configurationGroupMaxLimit
).Improved the phrasing of the warning shown when
groupBy()
exceeds the max or default limit.Added validation to the
field
parameter of thekvParse()
function, so empty lists will be rejected with a meaningful error message.
Other
All users will not have access to the audit log or search all view by default anymore. Access can be granted with permissions.
Bump the version of the Monaco code editor.
Streaming queries that fail to validate now return a message of why validation failed.
Fix a bug causing Humio's digest coordinator to allow nodes to take over digest without catching up to the current leader. This could cause the new leader to replay more data from Kafka than necessary.
Fixed an issue where query auto-completion sometimes wouldn't show the documentation for the suggested functions.
Adds a new metric for the temp disk usage. The metric name is
temp-disk-usage-bytes
and denotes how many bytes are used.Added a log message with the maximum state size seen by the live part of live queries.
Include the requester in logs from QuerySessions when a live query is restarted or cancelled.
The audit log system repository on Cloud has been replaced with a view, so that dashboards etc. can be created on top of audit log data.
Make
BucketStorageUploadJob
only log at info level rather than error if a segment upload fails because the segment has been removed from the host. This can happen if node X tries to upload a segment, but node Y beats it to the punch. Node X may then choose to remove its copy before the upload completes.When unregistering a node from a cluster, return a validation error if it is still alive. Hosts should be shut down before attempting to remove them from the cluster. This validation can be skipped using the same
accept-data-loss
parameter that also disables other validations for the unregistration endpoint.Added detection and handling of all queries being blocked during Humio upgrades.
Added a log of the approximate query result size before transmission to the frontend, captured by the
approximateResultBeforeSerialization
key.Add flag whether a feature is experimental.
Added a log line for when a query exceeds its allotted memory quota.
The referrer meta tag for Humio has been changed from no-referrer to same-origin.
Compute next set of Prometheus metrics only in a single thread concurrently. If more requests arrive, then the next request gets the previous response.
Make a number of improvements to the digest partition coordinator. The coordinator now tries harder to avoid assigning digest to nodes that are not caught up on fetching segments from the other nodes. It also does a better job unassigning digest from dead nodes in edge cases.
Fix an unhandled IO exception from
TempDirUsageJob
. The consequence of the uncaught exception was only noise in the error log.Added a new action type that creates a CSV file from the query result and uploads it to Humio to be used with the
match()
query function. See Action Type: Upload File.Java in the docker images no longer has the
cap_net_bind_service
capability and thus Humio cannot bind directly to privileged ports when running as a non-root user.Add warning when a multitenancy user is changing data retention on an unlimited repository.
Improved performance of NDJSON format in S3 Archiving.
Fix a bug that could cause Humio to spuriously log errors warning about segments not being merged for datasources doing backfilling.
Humio now logs digest partition assignments regularly. The logs can be found using the query
class=*DigestLeadershipLoggerJob*
.All feature flags now contains a textual description about what features are hidden behind the flag.
Adds a logger job for cluster management stats it log the stats every 2 minutes, which makes them searchable in Humio.
The logs belong to the class
c.h.c.ClusterManagementStatsLoggerJob
, logs for all segments containsglobalSegmentStats
log about singular segments starts withsegmentStats
.Remove remains of default groups and roles. The concept was replaced with UserRoles.
Fixed in this release
Security
Update Netty to address CVE-2022-24823.
Bump javax.el to address CVE-2021-28170.
Falcon Data Replicator
FDR Ingest will no longer fail on events that are larger than the maximum allowed event size. Instead, such messages will be truncated.
UI Changes
Prevent the UI showing errors for smaller connection issues while restarting.
Websocket connections are now kept open when transitioning pages, and are used more efficiently for syntax highlighting.
Fixed an issue where some warnings would show twice.
Intermediate network issues are not reported immediately as an error in the UI.
Cloud: Updated the layout for license key page.
Fix the dropdown menus closing too early on the home page.
Fixed a bug where the "=" and "/=" buttons did not appear on cells in the event list where they should.
When viewing the events behind e.g. a Time Chart, the events will now only display with the
@timestamp
and@rawstring
columns.
GraphQL API
Fix the
assets
GraphQL query in organizations with views that are not 1-to-1 linked.
Configuration
Fixed a bug that could result in merging small ("undersized") segments even if the resulting segment would then have a wider than desired time span. The goal it to not produce segments that span more than the 10% of the retention setting for time for the repository. If no time-based retention is configured on the repository, then 3 times the value of configuration variable
MAX_HOURS_SEGMENT_OPEN
is applied as limit. For default settings, that results in 72 hours.Fixed an issue where event forwarding still showed as beta.
Fixed an issue where delete events from a minisegment could result in the merge of those minisegments into the resulting target segment never got executed.
Index in block needs reading from blockwriter before adding each item.
Fixed a bug where the @id field of events in live query were off by one.
Dashboards and Widgets
The theme toggle on a shared dashboard was moved to the header panel and no longer overlaps with any widgets.
The
Time Chart
widget regression line is no longer affected by the interpolation setting.
Functions
Fixed a bug where using
eval
as an argument to a function would result in a confusing error message.Fixed a bug where
ioc:lookup()
would sometimes give incorrect results when negated.Revised some of the error messages and warnings regarding
join()
andselfJoin()
.Fixed a bug where the
writeJson()
function would write any field starting with a case-insensitiveinf
orinfinity
prefix as a null value in the resulting JSON.
Other
Make streaming queries search segments newest-to-oldest rather than oldest-to-newest. Streaming queries do not ensure the order of exported events anyway, and searching newest-to-oldest is more efficient.
Fix a bug where changing a role for a user under a repository would trigger infinite network requests.
Centralise decision for names of files in bucket, allow more than one variant.
Improved hover messages for strings.
Fixed an issue where query auto-completion would sometimes delete existing parentheses.
If a segment is deleted or otherwise disappears from global while Humio is attempting to upload it to bucket storage, the upload will now be dropped with an info-level log, rather than requeued with an error log.
Fixes the placement of a confirmation dialog when attempting to change retention.
Humio will now clean up its tmp directories by deleting all "humiotmp" directories in the data directory when terminating gracefully.
Fix a regression in the launcher script causing
JVM_LOG_DIR
to not be evaluated relative to the Humio base install path. All paths in the launcher script should now be relative to the base install path, which is the directory containing the bin folder.Fix a bug that could cause merge targets to be cached indefinitely if the associated minis had their mergeTarget unset. The effect was a minor memory leak.
Fix a bug that could cause Humio to attempt to merge minisegments from one datasource into a segment in another datasource, causing an error to be thrown.
When configuring thread priorities, Humio will no longer attempt to call the native setpriority function. It will instead only call the Java API for setting thread priority.
Fixed an issue where JSON parsing on ingest and in the query language was inefficient for large JSON objects.
Fix performance issue for users with access to many views.
Improve file path handling in DiskSpaceJob to eliminate edge cases where the job might not have been able to tell if a file was on primary or secondary storage.
Fix type in
Unregisters node
text on cluster admin UI.Fix a bug that could cause a
NumberFormatException
to be thrown fromZookeeperStatsClient
.Fixed an issue where event forwarder properties were not properly validated.
Reduced the timeout used when testing event forwarders in order to get a better error when timeouts happen.
Fix a bug causing digesters to continue digesting even if the local disk is full. The digester will now pause digesting and error log if this occurs.
Fix response entities not being discarded in error cases for the
proxyqueryjobs
endpoint, which could cause warnings in the log.Update
org.json:json
to address a vulnerability that could cause stack overflows.Fix an issue causing the event forwarding feature to incorrectly reject topic names that contained a dash
(-)
.Fix an issue that could rarely cause exceptions to be thrown from
Segments.originalBytesWritten
, causing noise in the log.Fix an issue causing Humio to create a large number of temporary directories in the data directory.
Bump woodstox to address SNYK-JAVA-COMFASTERXMLWOODSTOX-2928754.
Some errors messages wrongly pointed to the beginning of the query.
Kafka upgrades to 3.2.0 in the Docker images and in the Humio dependencies.
Fixed an issue where strings like
Nana
andInformation
could be interpreted asNaN
(not-a-number) andinfinity
, respectively.Fixed a bug where multiline comments weren't always highlighted correctly.