Lookup Files Operations with match()
When using lookup files with the match() function, consider the
following:
Lookup files use server memory proportional to the size of the file on disk, at least as much and typically more. If you have a 1Gb lookup file it will take up at least 1Gb of memory on some, potentially all, hosts within the cluster. This requirement should be taken into account when uploading and sizing the nodes within the cluster.
From LogScale v1.108 on, content of the file is shared among all queries that uses
match(), that is, the included columns that are common amongmatch()functions can be reused among queries.From 1.117 version on, whenever a file is updated, live queries and alert queries that use that file will seamlessly continue to run with the new updated file, thus making little difference if you have many small files to update or one large file. Since the file is swapped while the query is running, this also means that events can be queried with different versions of the file.
From LogScale v1.90, if you have large lookup files, wrap the uses of
match()in saved queries rather than use them directly across multiple different queries to ensure you don't accidentally pass slightly different argument in different queries. However, due to an improved reuse of files introduced in LogScale v1.108, this practice is no longer necessary starting from that version.