Falcon LogScale 1.124.3 LTS (2024-05-14)

Version?Type?Release Date?Availability?End of SupportSecurity UpdatesUpgrades From?Downgrades To?Config. Changes?
1.124.3LTS2024-05-14

Cloud

On-Prem

2025-03-01No1.70.01.112.0No

Download: https://repo.humio.com/repository/maven-releases/com/humio/server/1.124.3/server-1.124.3.tar.gz

Hide file hashes

Show file hashes

These notes include entries from the following previous releases: 1.124.1, 1.124.2

Bug fixes and updates.

Breaking Changes

The following items create a breaking change in the behavior, response or operation of this release.

  • Functions

    • The default accuracy of the percentile() function has been adjusted. This means that any query that does not explicitly set the accuracy may see a change in reported percentile. Specifically, the percentile() function may now deviate by up to one 100th of the true percentile, meaning that if a given percentile has a true value of 1000, percentile() may report a percentile in the range of [990; 1010].

      On the flip side, percentile() now uses less memory by default, which should allow for additional series or groups when this function is used with either timeChart() or groupBy() and the default accuracy is used.

Advance Warning

The following items are due to change in a future release.

  • Installation and Deployment

    • We intend to drop support for Java 17, making Java 21 the minimum. We plan to make this change in March 2024.

    • We aim to stop publishing the jar distribution of LogScale (e.g. server-1.117.jar) as of LogScale version 1.130.0.

      Users deploying via Docker images are not affected. Users deploying on bare metal should ensure they deploy the tar artifact, and not the jar artifact.

      A migration guide for bare metal deployments is available at How-To: Migrating from server.jar to Launcher Startup.

Removed

Items that have been removed as of this release.

GraphQL API

  • Removed the Asset interface type in GraphQL that Alert, Dashboard, Parser, SavedQuery and ViewInteraction datatypes implemented. It was not used as a type for any field. All fields from the Asset interface type are still present in the implementing types.

Configuration

  • The DEFAULT_PARTITION_COUNT configuration parameter has been removed, as it was unused by the system due to earlier changes to partition handling.

Deprecation

Items that have been deprecated and may be removed in a future release.

  • In the GraphQL API, the name argument to the parser field on the Repository datatype has been deprecated and will be removed in version 1.136 of LogScale.

  • We are deprecating the humio/kafka and humio/zookeeper Docker images due to low use. The planned final release for these images will be with LogScale 1.148.0.

    Better alternatives are available going forward. We recommend the following:

    • If your cluster is deployed on Kubernetes: STRIMZI

    • If your cluster is deployed to AWS: MSK

    If you still require humio/kafka or humio/zookeeper for needs that cannot be covered by these alternatives, please contact Support and share your concerns.

  • The assetType GraphQL field on Alert, Dashboard, Parser, SavedQuery and ViewInteraction datatypes has been deprecated and will be removed in version 1.136 of LogScale.

  • In the GraphQL API, the ChangeTriggersAndAction enum value for both the Permission and ViewAction enum is now deprecated and will be removed in version 1.136 of LogScale.

  • The humio Docker image is deprecated in favor of humio-core. humio is no longer considered suitable for production use, as it runs Kafka and ZooKeeper on the same host as LogScale, which our deployment guidelines no longer recommend. The final release of humio Docker image will be in version 1.130.0.

    The new humio-single-node-demo image is an all-in-one container suitable for quick and easy demonstration setups, but which is entirely unsupported for production use.

    For more information, see Installing Using Containers.

  • The QUERY_COORDINATOR environment variable is deprecated. To control whether a node should be allowed to be a query coordinator, use the query node task instead. Node tasks can be assigned and unassigned at runtime using the assignTasks() and unassignTasks() GraphQL mutations respectively, or controlled using the INITIAL_DISABLED_NODE_TASKS environment variable.

    For more information, see INITIAL_DISABLED_NODE_TASKS.

Behavior Changes

Scripts or environment which make use of these tools should be checked and updated for the new configuration:

  • Storage

    • We have adjusted the code that calculates where to start reading from the ingest queue to be more conservative. It will no longer allow for skipping past segments that are not fully replicated when later segments on the same datasource are fully replicated. This fixes a very rare edge case that could cause data loss on clusters using ephemeral disks. Due to the changed behavior, any segment failing to properly replicate will now cause LogScale to stop deleting data from the affected Kafka partition. Cluster administrators are strongly encouraged to monitor this case, by keeping under observation Kafka's disk usage.

  • Ingestion

    • We have reverted the behavior of blocking heavy queries in case of high ingest, and returned to the behavior of only stopping the query, due to issues caused by the blockage. Heavy queries causing ingest delay will be handled differently in a future version release.

Upgrades

Changes that may occur or be required during an upgrade.

  • Installation and Deployment

    • Kafka client library has been upgraded to 3.6.1. Some minor changes have been made to serializers used by LogScale to reduce memory copying.

New features and improvements

  • User Interface

    • When Manage Users, it is now possible to filter users based also on their assigned roles (for example, type admin in the Users search field).

    • Time zone data has been updated to IANA 2024a and has been trimmed to +/- 5 years from the release date of IANA 2024a.

    • Time zone data has been updated to IANA 2023d.

    • Deletion of a file that is actively used by live queries will now stop those queries.

      For more information, see Export a lookup file.

    • Multi-Cluster Search — early adopter release for Self-hosted LogScale.

      • Keep the data close to the source, search from single UI

      • Search across multiple LogScale clusters in a single view

      • Support key functionalities like alerts & dashboards

      The functionality is limited to LogScale self-hosted versions at this point.

      For more information, see LogScale Multi-Cluster Search.

    • The Field Aliasing feature is introduced. Implementing Field Aliasing in your workflow simplifies data correlation from various sources. With this feature, users can give alternative names — aliases — to fields created at parse time, across a view, or the entire organization. It makes data interpretation more intuitive and provides analysts with a smoother search experience.

      For more information, see Field Aliasing.

  • Automation and Triggers

    • The following changes affects the UI for Legacy alerts:

      • A minimum time window of 1 minute is introduced, since anything smaller will not produce reliable results. Any existing standard alert with a time window smaller than 1 minute will not run, instead an error notification will be shown.

      • It is no longer possible to specify the time window and the throttle period in milliseconds. Any existing standard alerts with a time window or throttle period specified in milliseconds will have it rounded to the nearest second.

      • When saving the alert, the query window is automatically changed to the largest unit in the Relative Time Syntax that can represent it. For example 24h is changed to 1d and 60s is changed to 1m.

    • The ChangeTriggersAndActions permission is now replaced by two new permissions:

      • ChangeTriggers permission is needed to edit alerts or scheduled searches.

      • ChangeActions permission is needed to edit actions as well as viewing them. Viewing the name and type of actions when editing triggers is still possible without this permission.

      Any user with the legacy ChangeTriggersAndActions permissions will by default have both. It is possible to remove one of them for more granular access controls.

    • A slow-query logging has been added when an alert is slow to start due to the query not having finished the historical part.

  • GraphQL API

    • Added limits for GraphQL queries on the total number of selected fields and fragments. Defaults are 1000 for authenticated and 150 for unauthenticated users.

      Cluster administrators can adjust these limits with the GraphQLSelectionSizeLimit and UnauthenticatedGraphQLSelectionSizeLimit dynamic configurations.

  • Storage

  • Configuration

    • The meaning of S3_STORAGE_CONCURRENCY and GCP_STORAGE_CONCURRENCY configuration variables has slightly changed. The settings are used for throttling downloads and uploads for bucket storage. Previously, a setting of S3_STORAGE_CONCURRENCY=10 for example, meant that LogScale would allow 10 concurrent uploads, and 10 concurrent downloads. Now, it means that LogScale will allow a total of 10 transfers at a time, disregarding the transfer direction.

    • New dynamic configurations have been added:

    • Ingest rate monitoring for autosharding improved. For clusters with more than 10 nodes, only a subset of the nodes will be reporting their ingest rate for any given datasource, and the total rate for each datasource estimated based on that. The dynamic configuration TargetMaxRateForDatasource still sets the threshold for sharding; however, once the rate is exceeded, it is no longer needed to be twice the TargetMaxRateForDatasource configuration before shards are added.

  • Ingestion

    • Introducing Ingest Feeds, a new pull-based ingest source that ingests logs stored in AWS S3. The files within the AWS S3 bucket can be Gzip compressed and we currently support newline delimited files and the JSON object format in which CloudTrail logs are stored in. Ingest Feeds require some configuration setup on the AWS side to get started.

      This feature is part of a gradual rollout process and may not be available on your cloud instance, but will be available to all customers in the following weeks.

      For more information, see Ingest Data from AWS S3.

    • The limits on the size of parser test cases when exporting as templates or packages has been increased.

    • The amount of logging produced by DigestLeadershipLoggerJob has been reduced in clusters with many ingest queue partitions.

  • Dashboards and Widgets

    • A series of improvements has been added to the dashboard layout experience:

      • New widgets will be added in the topmost available space

      • When you drag widgets up, all widgets in the same column will move together

      • Improved experience when swapping the order of widgets (horizontally or vertically)

  • Log Collector

    • Groups have been added to Fleet Management for the LogScale Collector. This feature makes it possible to define dynamic groups using a filter based upon a subset of the LogScale Query Language Syntax. New Collectors enrolled into the fleet will automatically be configured based upon the groups filters they match, eliminating the need for manually assigning a configuration to every new LogScale Collector. Groups also allow you to combine multiple reusable configuration snippets.

      Additionally the management of instances has been simplified and merged into this new feature, and therefore the Assigned Instances page has been removed to favor use of the Group functions.

      For more information, see Manage Groups.

  • Queries

    • The worker-level prioritization of queries has been changed. The new prioritization will attempt to divide time evenly between all users, and divide the time given to each user evenly among that user's queries.

    • Live query cost metrics corrections:

      • livequeries-rate metric has changed from long to double.

      • livequeries-rate-canceled-due-to-digest-delay metric has changed from long to double.

      For more information, see Node-Level Metrics.

  • Functions

    • The new array:length() function has been introduced. It finds the length of an array by counting the number of array entries.

      For more information, see array:length().

Fixed in this release

  • User Interface

    • When hovering over a query function in the query editor, the link to the function documentation now always points to the latest version of the page.

  • Automation and Triggers

    • After updating Scheduled searches where the action was failing, they would constantly fail with a None.get error until they were disabled and enabled again, or the LogScale cluster was restarted. This issue is now fixed.

  • Storage

    • Fixed an issue that could cause repositories undeleted using the mechanism described at Restoring a Repository or View to be only partially restored. Some deleted datasources within the repositories could erroneously be skipped during restoration.

      For more information, see Restoring a Repository or View.

  • Dashboards and Widgets

    • Users were prevented from exporting results of queries containing multi value parameters. This issue is now fixed.

  • Queries

    • Queries in some cases would be killed as if they were blocked even though they did not match the criteria of the block. This issue is now fixed.

    • Fixed a bug in which the second poll inside the cluster could be delayed by upwards of 10 seconds. This fix ensures that the time between polls will never be later than the start time of the query, this means that early polls will not be delayed too much, enabling faster query responses.

  • Functions

    • selectLast() has been fixed for an issue that could cause this query function to miss events in certain cases.

  • Other

    • It was not possible to create a new repository with a time retention greater than 365 days. Now, the UI limit is the one that is set on the customer organization.

      Input validation on fields when creating new repositories is now also improved.

Improvement

  • Storage

    • Allowed reassignment of digest that assigns partitions unevenly to hosts. This is to support clusters where hosts are not evenly sized, and so an even partition assignment is not expected.

  • Configuration

  • Ingestion

    • The cancelling mechanism for specific costly queries has been improved to solve cases where those queries got restarted anyway: the query with the exact match on the query string is now blocked for 5 minutes. This will free enough CPU for ingest to catch up and avoid blocking queries for too long.