Falcon LogScale 1.124.3 LTS (2024-05-14)
Version? | Type? | Release Date? | Availability? | End of Support | Security Updates | Upgrades From? | Config. Changes? |
---|---|---|---|---|---|---|---|
1.124.3 | LTS | 2024-05-14 | Cloud | 2025-03-01 | No | 1.70.0 | No |
Hide file hashes
TAR Checksum | Value |
---|---|
MD5 | 730dd2226aa23d325b7fecc7f7ee4138 |
SHA1 | 3fdde9ee0728eace9808f845504c9f584d0da81f |
SHA256 | fd5d88be54cc487db542d4c3dad3072913edad50540e0fe2e14487ee26525d0b |
SHA512 | 22377e496e6cd0abd4aac0c6fba41d35596703246d98d0e2dcd8ddb026127a6fce4a2fa4bad6e46682f0686508b58c5c13ad23db8ad3554356b157aeb6c95e0e |
Docker Image | Included JDK | SHA256 Checksum |
---|---|---|
humio | 21 | 84fb05447e3776cace395a8799a17476aca125bc7b4ee50d515a4e3aa89d3282 |
humio-core | 21 | 81b999f222d55ca6c36a05fbb1237444a6de91997eef7520cd208d16a7d29618 |
kafka | 21 | b375c5ce0bbfbc3dea1fa2fbaa2be5cc66b2c59dd38710c1b09acdbce176a40f |
zookeeper | 21 | e054e07d2ba316b0c9e59699b420f37bac2057e39516f5a7263d46c13628dc26 |
Download: https://repo.humio.com/repository/maven-releases/com/humio/server/1.124.3/server-1.124.3.tar.gz
These notes include entries from the following previous releases: 1.124.1, 1.124.2
Bug fixes and updates.
Breaking Changes
The following items create a breaking change in the behavior, response or operation of this release.
Functions
The default accuracy of the
percentile()
function has been adjusted. This means that any query that does not explicitly set the accuracy may see a change in reported percentile. Specifically, thepercentile()
function may now deviate by up to one 100th of the true percentile, meaning that if a given percentile has a true value of 1000,percentile()
may report a percentile in the range of[990; 1010]
.On the flip side,
percentile()
now uses less memory by default, which should allow for additional series or groups when this function is used with eithertimeChart()
orgroupBy()
and the default accuracy is used.
Advance Warning
The following items are due to change in a future release.
Installation and Deployment
We aim to stop publishing the
jar
distribution of LogScale (e.g.server-1.117.jar
) as of LogScale version 1.130.0.Users deploying via Docker images are not affected. Users deploying on bare metal should ensure they deploy the
tar
artifact, and not thejar
artifact.A migration guide for bare metal deployments is available at How-To: Migrating from server.jar to Launcher Startup.
We intend to drop support for Java 17, making Java 21 the minimum. We plan to make this change in March 2024.
Removed
Items that have been removed as of this release.
GraphQL API
Removed the
Asset
interface type in GraphQL thatAlert
,Dashboard
,Parser
,SavedQuery
andViewInteraction
datatypes implemented. It was not used as a type for any field. All fields from theAsset
interface type are still present in the implementing types.Configuration
The
DEFAULT_PARTITION_COUNT
configuration parameter has been removed, as it was unused by the system due to earlier changes to partition handling.
Deprecation
Items that have been deprecated and may be removed in a future release.
The assetType GraphQL field on
Alert
,Dashboard
,Parser
,SavedQuery
andViewInteraction
datatypes has been deprecated and will be removed in version 1.136 of LogScale.The
humio
Docker image is deprecated in favor ofhumio-core
.humio
is no longer considered suitable for production use, as it runs Kafka and ZooKeeper on the same host as LogScale, which our deployment guidelines no longer recommend. The final release ofhumio
Docker image will be in version 1.130.0.The new
humio-single-node-demo
image is an all-in-one container suitable for quick and easy demonstration setups, but which is entirely unsupported for production use.For more information, see Installing Using Containers.
In the GraphQL API, the
ChangeTriggersAndAction
enum value for both thePermission
andViewAction
enum is now deprecated and will be removed in version 1.136 of LogScale.The
QUERY_COORDINATOR
environment variable is deprecated. To control whether a node should be allowed to be a query coordinator, use thequery
node task instead. Node tasks can be assigned and unassigned at runtime using the assignTasks() and unassignTasks() GraphQL mutations respectively, or controlled using theINITIAL_DISABLED_NODE_TASKS
environment variable.For more information, see
INITIAL_DISABLED_NODE_TASK
.We are deprecating the
humio/kafka
andhumio/zookeeper
Docker images due to low use. The planned final release for these images will be with LogScale 1.148.0.Better alternatives are available going forward. We recommend the following:
If you still require
humio/kafka
orhumio/zookeeper
for needs that cannot be covered by these alternatives, please contact Support and share your concerns.In the GraphQL API, the name argument to the parser field on the
Repository
datatype has been deprecated and will be removed in version 1.136 of LogScale.
Behavior Changes
Scripts or environment which make use of these tools should be checked and updated for the new configuration:
Storage
We have adjusted the code that calculates where to start reading from the ingest queue to be more conservative. It will no longer allow for skipping past segments that are not fully replicated when later segments on the same datasource are fully replicated. This fixes a very rare edge case that could cause data loss on clusters using ephemeral disks. Due to the changed behavior, any segment failing to properly replicate will now cause LogScale to stop deleting data from the affected Kafka partition. Cluster administrators are strongly encouraged to monitor this case, by keeping under observation Kafka's disk usage.
Ingestion
We have reverted the behavior of blocking heavy queries in case of high ingest, and returned to the behavior of only stopping the query, due to issues caused by the blockage. Heavy queries causing ingest delay will be handled differently in a future version release.
Upgrades
Changes that may occur or be required during an upgrade.
Installation and Deployment
Kafka client library has been upgraded to 3.6.1. Some minor changes have been made to serializers used by LogScale to reduce memory copying.
New features and improvements
UI Changes
Time zone data has been updated to IANA 2024a and has been trimmed to +/- 5 years from the release date of IANA 2024a.
Time zone data has been updated to IANA 2023d.
Deletion of a file that is actively used by live queries will now stop those queries.
For more information, see Exporting or Deleting a File.
Multi-Cluster Search — early adopter release for Self-hosted LogScale.
Keep the data close to the source, search from single UI
Search across multiple LogScale clusters in a single view
Support key functionalities like alerts & dashboards
The functionality is limited to LogScale self-hosted versions at this point.
For more information, see LogScale Multi-Cluster Search.
When Manage Users, it is now possible to filter users based also on their assigned roles (for example, type
admin
in the Users search field).The Field Aliasing feature is introduced. Implementing Field Aliasing in your workflow simplifies data correlation from various sources. With this feature, users can give alternative names — aliases — to fields created at parse time, across a view, or the entire organization. It makes data interpretation more intuitive and provides analysts with a smoother search experience.
For more information, see Field Aliasing.
Automation and Alerts
The following changes affects the UI for Standard Alerts:
A minimum time window of 1 minute is introduced, since anything smaller will not produce reliable results. Any existing standard alert with a time window smaller than 1 minute will not run, instead an error notification will be shown.
It is no longer possible to specify the time window and the throttle period in milliseconds. Any existing standard alerts with a time window or throttle period specified in milliseconds will have it rounded to the nearest second.
When saving the alert, the query window is automatically changed to the largest unit in the Relative Time Syntax that can represent it. For example
24h
is changed to1d
and60s
is changed to1m
.
The
ChangeTriggersAndActions
permission is now replaced by two new permissions:ChangeTriggers
permission is needed to edit alerts or scheduled searches.ChangeActions
permission is needed to edit actions as well as viewing them. Viewing the name and type of actions when editing triggers is still possible without this permission.
Any user with the legacy
ChangeTriggersAndActions
permissions will by default have both. It is possible to remove one of them for more granular access controls.A slow-query logging has been added when an alert is slow to start due to the query not having finished the historical part.
GraphQL API
Added limits for GraphQL queries on the total number of selected fields and fragments. Defaults are
1000
for authenticated and150
for unauthenticated users.Cluster administrators can adjust these limits with the
GraphQLSelectionSizeLimit
andUnauthenticatedGraphQLSelectionSizeLimit
dynamic configurations.
Storage
The following validation constraints are added on boot:
LOCAL_STORAGE_PERCENTAGE
is less thanSECONDARY_STORAGE_MAX_FILL_PERCENTAGE
on nodes with secondary storage configured.LOCAL_STORAGE_PERCENTAGE
is less thanPRIMARY_STORAGE_MAX_FILL_PERCENTAGE
on nodes without secondary storage configured.
Nodes will crash on boot if these constraints are violated.
We have changed how LogScale handles being temporarily bottlenecked by bucket storage. Uploads are now prioritized ahead of downloads, which reduces the impact on ingest work.
Configuration
The meaning of
S3_STORAGE_CONCURRENCY
andGCP_STORAGE_CONCURRENCY
configuration variables has slightly changed. The settings are used for throttling downloads and uploads for bucket storage. Previously, a setting ofS3_STORAGE_CONCURRENCY=10
for example, meant that LogScale would allow 10 concurrent uploads, and 10 concurrent downloads. Now, it means that LogScale will allow a total of 10 transfers at a time, disregarding the transfer direction.New dynamic configurations have been added:
defaultDigestReplicationFactor
dynamic configuration defaults to2
if the value is not explicitly set and there is more than 1 node in the cluster performing digest.If necessary, a different default can be explicitly set using the
DEFAULT_DIGEST_REPLICATION_FACTOR
environment variable.defaultSegmentReplicationFactor
dynamic configuration defaults to2
if the value is not explicitly set, unless there is only 1 node in the cluster storing segments, or ifUSING_EPHEMERAL_DISKS
environment variable is set totrue
.If necessary, a different default can be explicitly set using the
DEFAULT_SEGMENT_REPLICATION_FACTOR
environment variable.
Ingest rate monitoring for autosharding improved. For clusters with more than 10 nodes, only a subset of the nodes will be reporting their ingest rate for any given datasource, and the total rate for each datasource estimated based on that. The dynamic configuration
TargetMaxRateForDatasource
still sets the threshold for sharding; however, once the rate is exceeded, it is no longer needed to be twice theTargetMaxRateForDatasource
configuration before shards are added.
Dashboards and Widgets
A series of improvements has been added to the dashboard layout experience:
New widgets will be added in the topmost available space
When you drag widgets up, all widgets in the same column will move together
Improved experience when swapping the order of widgets (horizontally or vertically)
Ingestion
Introducing Ingest Feeds, a new pull-based ingest source that ingests logs stored in AWS S3. The files within the AWS S3 bucket can be Gzip compressed and we currently support newline delimited files and the JSON object format in which CloudTrail logs are stored in. Ingest Feeds require some configuration setup on the AWS side to get started.
This feature is part of a gradual rollout process and may not be available on your cloud instance, but will be available to all customers in the following weeks.
For more information, see Ingest Data from AWS S3.
The limits on the size of parser test cases when exporting as templates or packages has been increased.
The amount of logging produced by
DigestLeadershipLoggerJob
has been reduced in clusters with many ingest queue partitions.
Log Collector
Groups have been added to Fleet Management for the LogScale Collector. This feature makes it possible to define dynamic groups using a filter based upon a subset of the LogScale Query Language Syntax. New Collectors enrolled into the fleet will automatically be configured based upon the groups filters they match, eliminating the need for manually assigning a configuration to every new LogScale Collector. Groups also allow you to combine multiple reusable configuration snippets.
Additionally the management of instances has been simplified and merged into this new feature, and therefore the Assigned Instances page has been removed to favor use of the Group functions.
For more information, see Manage Groups.
Queries
The worker-level prioritization of queries has been changed. The new prioritization will attempt to divide time evenly between all users, and divide the time given to each user evenly among that user's queries.
Live query cost metrics corrections:
livequeries-rate
metric has changed from long to double.livequeries-rate-canceled-due-to-digest-delay
metric has changed from long to double.
For more information, see Node-Level Metrics.
Functions
The new
array:length()
function has been introduced. It finds the length of an array by counting the number of array entries.For more information, see
array:length()
.
Fixed in this release
UI Changes
When hovering over a query function in the query editor, the link to the function documentation now always points to the latest version of the page.
Automation and Alerts
After updating Scheduled searches where the action was failing, they would constantly fail with a None.get error until they were disabled and enabled again, or the LogScale cluster was restarted. This issue is now fixed.
Storage
Fixed an issue that could cause repositories undeleted using the mechanism described at Restoring a Repository or View to be only partially restored. Some deleted datasources within the repositories could erroneously be skipped during restoration.
For more information, see Restoring a Repository or View.
Dashboards and Widgets
Users were prevented from exporting results of queries containing multi value parameters. This issue is now fixed.
Queries
Queries in some cases would be killed as if they were blocked even though they did not match the criteria of the block. This issue is now fixed.
Fixed a bug in which the second poll inside the cluster could be delayed by upwards of 10 seconds. This fix ensures that the time between polls will never be later than the start time of the query, this means that early polls will not be delayed too much, enabling faster query responses.
Functions
selectLast()
has been fixed for an issue that could cause this query function to miss events in certain cases.
Other
It was not possible to create a new repository with a time retention greater than 365 days. Now, the UI limit is the one that is set on the customer organization.
Input validation on fields when creating new repositories is now also improved.
Improvement
Storage
Allowed reassignment of digest that assigns partitions unevenly to hosts. This is to support clusters where hosts are not evenly sized, and so an even partition assignment is not expected.
Configuration
The default limit for uploading CSV Lookup Files set by
MaxCsvFileUploadSizeBytes
dynamic configuration has been increased from100MB
to200MB
. IfMAX_FILEUPLOAD_SIZE
is set, its value will be the default for bothMaxCsvFileUploadSizeBytes
andMaxJsonFileUploadSizeBytes
.
Ingestion
The cancelling mechanism for specific costly queries has been improved to solve cases where those queries got restarted anyway: the query with the exact match on the query string is now blocked for 5 minutes. This will free enough CPU for ingest to catch up and avoid blocking queries for too long.