Lookup Files
Security Requirements and Controls
Change files
permissionData read access
permission
Lookup files are used to add additional context to data, enabling you to attach or replace text from events recorded in a repository when searched.
To add a lookup file, you create or import a CSV (comma-separated value) file and upload it to the repository.
These files can be used together with query functions to provide lookups
and matching using the match()
function.
The feature also works with the readFile()
function
for reading a file which is used as data input for your query.
The following operations are available:
For information on how Lookup files interact with the rest of the system, see Lookup Files Operations.
Creating a File
Click
→ → .Specify a name for the file and then select either
to create an empty file to populate or to use a template from a previously installed package.Click
.If you've created an empty file, click
to add rows and columns.Click
to save the changes.
If you have many changes to make, editing a data table through the
Files
interface page can be tedious:
click and
then edit the table in a spreadsheet program or a simple text editor.
Note
Files larger than 100 MB cannot be viewed in the UI.
Figure 33. Create New CSV File
When a file is referenced in a query, a tab is shown in the
Search
page bearing the same name of the
file. This file tab will display the file content as a
Table
widget. Alternatively, if the file
cannot be queried, a download link will be presented instead. For
example, executing the query:
groupBy([status])
| match(file="status_codes.csv", column="code", field="status", include=name)
will show a file table named status_codes.csv:
Figure 34. File Tab in Search View
Uploading Files
Go to the
interface → → .Drag and drop your file or browse for the file to upload.
You can upload a CSV file containing text like what you see below, which is essentially a lookup table that you can use for labels or value lookups.
yamluserid,ip,username,region 1,"212.12.31.23","pete","EU" 2,"212.12.31.231","bob","EU" 3,"98.12.31.21","anders","EU" 4,"121.12.31.23","jeff","US" 5,"82.12.31.23","ted","AU" 6,"62.12.31.23","annie","US" 7,"122.12.31.23","joe","CH" 8,"112.11.11.21","alice","CH" 9,"212.112.131.22","admin","RU" 10,"212.12.31.23","wendy","EU"
Once it has been uploaded, it will look like what you see in figure below.
Figure 35. Import CSV File
You would use such a data table together with the
match()
functions to add labels to the results of a search. Notice that the values are in quotes, except for the ones for userid, which are integers. See the Lookup API reference page for more information on this topic.Edit the data in the file editor table as you wish, and click
to add rows and columns. Clicking the tiny information icon next to the file name displays metadata info about the file (created by, time it was created, etc.)Once you have finished editing, click
, or click if you wish to download the edited file.
Exporting or Deleting a File
Files can be managed by clicking the menu icon next to each file. You can either export or delete a file:
Figure 36. Manage CSV Files
Warning
Deleting a file that is actively used by live queries will stop those queries.
Lookup Files Operations
When using Lookup files and match()
functionality,
consider the following:
Lookup files use server memory proportional to the size of the file on disk; at least as much and typically more. If you have a 1Gb lookup file it will take up at least 1Gb of memory on some, potentially all, hosts within the cluster. This requirement should be taken into account when uploading and sizing the nodes within the cluster.
From LogScale v1.108 on, content of the file is shared among all queries that uses
match()
, that is, the included columns that are common amongmatch()
functions can be reused among queries.From 1.117 version on, whenever a file is updated, live queries and alert queries that use that file will seamlessly continue to run with the new updated file, thus making little difference if you have many small files to update or one large file. Since the file is swapped while the query is running, this also means that events can be queried with different versions of the file.
From LogScale v1.90, if you have large lookup files, wrap the uses of
match()
in saved queries rather than use them directly across multiple different queries to ensure you don't accidentally pass slightly different argument in different queries. However, due to an improved reuse of files introduced in LogScale v1.108, this practice is no longer necessary starting from that version.