Falcon LogScale 1.165.1 LTS (2024-12-17)

Version?Type?Release Date?Availability?End of Support

Security

Updates

Upgrades

From?

Config.

Changes?
1.165.1LTS2024-12-17

Cloud

On-Prem

2025-12-31Yes1.112No

Hide file hashes

Show file hashes

Download

Bug fixes and updates.

Removed

Items that have been removed as of this release.

GraphQL API

  • Removed the following deprecated fields from the Cluster GraphQL type:

    • ingestPartitionsWarnings

    • suggestedIngestPartitions

    • suggestedIngestPartitions

    • storagePartitions

    • storagePartitionsWarnings

    • suggestedStoragePartitions

Configuration

  • The dynamic configuration and related GraphQL API AstDepthLimit has been removed.

  • The UNSAFE_ALLOW_FEDERATED_CIDR, UNSAFE_ALLOW_FEDERATED_MATCH, and ALLOW_MULTI_CLUSTER_TABLE_SYNCHRONIZATION environment variables have been removed as they now react as if they are always enabled.

Deprecation

Items that have been deprecated and may be removed in a future release.

  • The lastScheduledSearch field from the ScheduledSearch datatype is now deprecated and planned for removal in LogScale version 1.202. The new lastExecuted and lastTriggered fields have been added to the ScheduledSearch datatype to replace lastScheduledSearch.

Upgrades

Changes that may occur or be required during an upgrade.

  • Installation and Deployment

    • The JDK has been upgraded to 23.0.1

New features and improvements

  • Security

    • Users can now view actions in restricted read-only mode when they have the Data read access permission on the repository or view.

    • Users can now see and use saved queries without needing the CreateSavedQueries and the UpdateSavedQueries permissions.

    • Users can now see actions in restricted read-only mode when they have the ReadAccess permission on the repository or view.

  • Installation and Deployment

  • UI Changes

    • PDF Render Service now supports proxy communication between service and LogScale. Adding the environment variable http_proxy or https_proxy to the PDF render service environment will add a proxy agent to all requests from the service to LogScale.

    • Documentation is now displayed on hover in the LogScale query editor within Falcon. The full syntax usage and a link to the documentation is now visible for any keyword in a query.

    • The Files page now features a new table view with enhanced search and filtering, making it easier to find and manage your files. You can now import multiple files at once.

      For more information, see Lookup Files.

    • When Saving Queries, saved queries now appear in sorted order and are also searchable.

    • Users with the ReadAccess permission on the repository or view can now view scheduled reports in read-only mode.

    • Files grouped by package are now displayed back again on the Files page including the Package Name column, which was temporarily unavailable after the recent page overhaul.

    • A custom dialog now helps users save their widget changes on the Dashboard page before continuing on the Search page.

  • Automation and Alerts

    • In the activity logs, the exception field now only contains the name of the exception class, as the remainder of what used to be there is already present in the exceptionMessage field.

    • Three alert messages were deprecated and replaced with new, more accurate alert messages.

      • For Legacy Alerts: The query result is currently incomplete. The alert will not be polled in this loop replaces Starting the query for the alert has not finished. The alert will not be polled in this loop.

      • For Filter Alerts and Aggregate Alerts: The query result is currently incomplete. The alert will not be polled in this run replaces Starting the alert query has not finished. The alert will not be polled in this run in some situations where it is more correct.

      • The alert message was updated for filter and aggregate alerts in some cases where the live query was stopped due to the alert being behind.

      For more information, see Monitoring Alert Execution through the humio-activity Repository.

    • The queryStart and queryEnd fields has been added for two aggregate alerts log lines:

      • Alert found results, but no actions were invoked since the alert is throttled

      • Alert found no results and will not trigger

      and removed for three others as they did not contain the correct value:

      • Alert is behind. Will stop live query and start running historic queries to catch up

      • Alert query took too long to start and the result are now too old. LogScale will stop the live query and start running historic queries to catch up

      • Running a historic query to catch up took too long and the result is now outside the retry limit. LogScale will skip this data and start a query for events within the retry limit

    • The Alerts page now shows the following UI changes:

      • A new column Last modified is added in the Alerts overview to display when the alert was last updated and by whom.

      • The same above column is added either in the alert properties side panel and in the Search page.

      • The Package column is no longer displayed as default on the Alerts overview page.

      For more information, see Creating an Alert from the Alerts Overview.

  • GraphQL API

    • The disableFieldAliasSchemaOnViews GraphQL mutation has been added. This mutation allows you to disable a schema on multiple views or repositories at once, instead of running multiple disableFieldAliasSchemaOnView mutations.

      For more information, see disableFieldAliasSchemaOnViews() .

    • New yamlTemplate fields have been created for Dashboard and SavedQuery datatypes. They now replace the deprecated templateYaml fields.

      For more information, see Dashboard , SavedQuery .

    • GraphQL introspection queries now require authentication. Setting the configuration parameter API_EXPLORER_ENABLED to false will still reject all introspection queries.

    • Added the permissionType field to the Group GraphQL type. This field identifies the level of permissions the group has (view, organization or system).

    • Added the following mutations:

      These mutations extend the functionality of the previous versions (without the V2 suffix) by returning additional information about the token such as the id, name, permissions, expiry and IP filters.

  • Storage

    • WriteNewSegmentFileFormat feature flag is now removed and the feature enabled by default to improve compression of segment files.

    • The amount of autoshard increase requests allowed has been reduced, to reduce pressure on global traffic from these requests.

  • API

    • Implemented support for returning a result over 1GB in size on the /api/v1/globalsubset/clustervhost endpoint. There is now a limit on the size of 8GB of the returned result.

  • Configuration

  • Ingestion

    • Query resources will now also account for reading segment files in addition to scanning files. This will enable better control of CPU resources between search and the data pipeline operations (ingest, digest, storage).

    • Increased a timeout for loading new CSV files used in parsers to reduce the likelihood of having the parser fail.

    • The way query resources are handled with respect to ingest occupancy has changed. If the maximum occupancy over all the ingest readers is less than the limit set (90 % by default), LogScale will not reduce resources for queries. The new configuration variable INGEST_OCCUPANCY_QUERY_PERMIT_LIMIT now allows to change such default limit of 90 % to adjust how busy ingest readers should be in order to limit query resources.

    • The toolbar of the Parser editor has been modified to be more in-line with the design of the LogScale layout. You can now find Duplicate, Settings and Export buttons under the ellipsis menu.

      For more information, see Parsing Data.

    • Added logging when a parser fails to build and ingest defaults to ingesting without parsing. The log lines start with Failed compiling parser.

  • Dashboards and Widgets

    • Numbers in the Table widget can now be displayed with trailing zeros to maintain a consistent number of decimal places.

    • When configuring series for a widget, suggestions for series are now available in a dropdown list, rather than having to type the series out.

    • The Bar Chart widget can now be configured in the style panel with a horizontal or vertical orientation.

  • Log Collector

  • Queries

    • LogScale Regular Expression Engine V2 is now optimized to support character match within a single line, e.g. /.*/s.

    • Ad-hoc tables feature is introduced for easier joins. Use the defineTable() function to define temporary lookup tables. Then, join them with the results of the primary query using the match() function. The feature offers several benefits:

      • Intuitive approach that now allows for writing join-like queries in the order of execution

      • Step-by-step workflow to create complex, nested joins easily.

      • Workflow that is consistent to the model used when working with Lookup Files

      • Easy troubleshooting while building queries, using the readFile()function

      • Expanded join use cases, providing support for:

        • inner joins with match(... strict=true)

        • left joins with match(... strict=false)

        • right joins with readFile() | match(... strict=false)

      • Join capabilities in LogScale Multi-Cluster Search environments (Self-Hosted users only)

      When match() or similar functions are used, additional tabs from the files and/or tables used in the primary query now appear in order in Search next to the Results tab. The tab names are prefixed by \"Table: \" to make it more clear what they refer to.

      For more information, see Using Ad-hoc Tables.

    • Changed the internal submit endpoint such that the requests logs correct information on whether the request is internal or not.

  • Functions

    • Improvements in the sort(), head(), and tail() functions: the error message when entering an incorrect value in the limit parameter now mentions both the minimum and the maximum configured value for the limit.

    • Introducing the new query function array:rename(). This function renames all consecutive entries of an array starting at index 0.

      For more information, see array:rename().

    • A new parameter trim has been added to the parseCsv() function to ignore whitespace before and after values. In particular, it allows quotes to appear after whitespace. This is a non-standard extension useful for parsing data created by sources that do not adhere to the CSV standard.

    • The following new functions have been added:

    • bitfield:extractFlags() can now handle unsigned 64 bit input. It can also handle larger integers, but only the lowest 64 bits will be extracted.

    • The wildcard() function has an additional parameter: includeEverythingOnAsterisk. When this parameter is set to true, and pattern is set to *, the function will also match events that are missing the field specified in the field parameter.

      For more information, see wildcard().

    • The following query functions limits have now their minimum value set to 1. In particular:

    • The new query functions crypto:sha1() and crypto:sha256() have been added. These functions compute a cryptographic SHA-hashing of the given fields and output a hex string as the result.

Fixed in this release

  • Security

    • OIDC authentication would fail if certain characters in the state variable were not properly URL-encoded when redirecting back to LogScale. This issue has been fixed.

  • UI Changes

    • Event List has been fixed as it would not take sorting from query API into consideration when sorting events based on UI configuration.

    • The red border appearing in the Table widget when invalid changes are made to a dashboard interaction is now fixed as it would not display correctly.

    • Dragging would stop working on the Dashboard page in cases where invalid changes were made and saved to a widget and the user would then click Continue editing. This issue has been fixed and the dragging now works correctly also in this case.

  • Automation and Alerts

    • Fixed an issue where the Action overview page would not load if it contained a large number of actions.

  • GraphQL API

    • role.users query has been fixed as it would return duplicate users in some cases.

  • Storage

    • Mini-segments would not be prioritized correctly when fetching them from bucket storage. This issue has now been fixed.

    • Segments were not being fetched on an owner node. This issue could lead to temporary under-replication and keeping events in Kafka.

    • Resolved a defect that could lead to corrupted JSON messages on the internal Kafka queue.

    • NullPointerException error occurring since version 1.156.0 when closing segment readers during redactEvent processing has now been fixed.

    • Several issues have been fixed, which could cause LogScale to replay either too much, or too little data from Kafka if segments with topOffsets were deleted at inopportune times. LogScale will now delay deleting newly written segments, even if they violate retention, until the topOffsets field has been cleared, which indicates that the segments cannot be replayed from Kafka later. Segment bytes being held onto in this way are logged by the RetentionJob as part of the periodic logging.

    • An extremely rare data loss issue has been fixed: file corruption on a digester could cause the cluster to delete all copies of the affected segments, even if some copies were not corrupt. When a digester detects a corrupt recently-written segment file during bootup, it will no longer delete that segment from Global. It will instead only remove the local file copy. If the segment needs to be deleted in Global because it's being replayed from Kafka, the new digest leader will handle that as part of taking over the partition.

    • Recently ingested data could be lost when the cluster has bucket storage enabled, USING_EPHEMERAL_DISKS is set to false, and a recently ingested segment only exists in bucket storage. This issue has now been fixed.

    • LogScale could spuriously log Found mini segment without replacedBy and a merge target that already exists errors when a repository is undeleted. This issue has been fixed.

  • API

    • An issue has been fixed in the computation of the digestFlow property of the query response. The information contained there would be stale in cases where the query started from a cached state or there were digest leadership changes (for example, in case of node restarts).

      For more information, see Polling a Query Job.

  • Ingestion

    • Parser Assertions have been fixed as some would be marked as passing, even though they should be failing.

    • An erronous array gap detection has been fixed as it would detect gaps where there were none.

    • An error is no longer returned when running parser tests without test cases.

    • An issue has been fixed that could cause the starting position for digest to get stuck in rare cases.

  • Dashboards and Widgets

    • Long values rendered in the Single Value widget would overflow the widget container. This issue has now been fixed.

    • Dashboard parameter values were mistakenly not used by saved queries in scenarios with parameter naming overlap and no saved query arguments provided.

  • Queries

    • Backtracking checks are now added to the optimized instructions for (?s).*? in the LogScale Regular Expression Engine V2. This prevents regexes of this type from getting stuck in infinite loops which are ultimately detrimental to a cluster's health.

    • Fixed an issue which could cause live query results from some workers being temporarily represented in the final result twice. The situation was transient and could only occur during digester changes.

    • Fixed an issue where a query would fail to start in some cases when the query cache was available. The user would see the error Recent events overlap span excluded from query using historicStartMin.

    • Stopping alerts and scheduled searches could create a Could not cancel alert query entry in the activity logs. This issue has now been fixed. The queries were still correctly stopped previously, but this bug led to incorrect logging in the activity log.

    • The query scheduler has been fixed for an issue that could cause queries to get stuck in rare cases.

  • Functions

    • In defineTable(), start and end parameters did not work correctly when the primary query's end time was a relative timestamp: the sub-query's time was relative to now, and it has now been fixed to be relative to the primary query's end time.

    • Error messages produced by the match() function could reference the wrong file. This issue has now been fixed.

  • Other

    • Query result highlighting would crash cluster nodes when getting filter matches for some regexes. This issue has been fixed.

Known Issues

  • Functions

    • A known issue in the implementation of the defineTable() function means it is not possible to transfer generated tables larger than 128MB. The user receives an error if the generated table exceeds that size.

    • The match() function misses some matching rows when matching on multiple rows in glob mode. This happens in cases where there are rows with different glob patterns matching on the same event. For example, using a file example.csv:

      Raw Events
      column1,column2
      ab*,one
      a*,two
      a*,three

      and the query:

      logscale
      match(example.csv, field=column1, mode=glob, nrows=3)

      An event with the field column1=abc will only match on the last two rows.

    • The match() function misses some matching rows when matching on multiple rows in cidr mode. This happens in cases where there are rows with different subnets matching the same event. For example, using a file example.csv:

      Raw Events
      subnet,value
      1.2.3.4/24,monkey
      1.2.3.4/25,horse

      and the query:

      logscale
      match(example.csv, field=subnet, mode=cidr, nrows=3)

      An input event with ip = 1.2.3.10 will only output: 

      ip,value
1.2.3.10,horse

      whereas the correct output should actually be: 

      ip,value
1.2.3.10,horse
1.2.3.10,monkey

Improvement

  • UI Changes

    • Improving the information messages that are displayed in the query editor when errors with lookup files used in queries occur.

    • Improving the warnings given when performing multi-cluster searches across clusters running on different LogScale versions.

  • API

  • Queries

    • Worker query prioritization is improved in specific cases where a query starts off highly resource-consuming but becomes more efficient as it progresses. In such cases, the scheduler could severely penalize the query, leading to it being unfairly deprioritized.

    • Queries that refer to fields in the event are now more efficient due to an improvement made in the query engine.