Lookup Files
Permission Requirements
Change files
permissionData read access
permission
Lookup files enable you to attach or replace text from events recorded in a repository when searched.
To do add a lookup file, you create and import a CSV (comma-separated
value) file and uploading them to the repository. These files can be used
together with query functions to provide lookups and matching using the
match()
.
The following operations are available:
For information on how Lookup Files interact with the rest of the system, see ???.
Creating a File
Click
Files
→ → .Specify a name for the file and then select either
to create an empty file to populate or to use a template from a previously installed package.Click
to add rows and columns.Click
to save the changes.If you have many changes to make, editing a data table through the
Files
interface page can be tedious: click and then edit the table in a spreadsheet program or a simple text editor.

Figure 56. Create New CSV File
Uploading a File
Click
Files
→ → .Browse for the file to upload and click
.You can upload a CSV file containing text like what you see below, which is essentially a lookup table that you can use for labels or value lookups.
yamluserid,ip,username,region 1,"212.12.31.23","pete","EU" 2,"212.12.31.231","bob","EU" 3,"98.12.31.21","anders","EU" 4,"121.12.31.23","jeff","US" 5,"82.12.31.23","ted","AU" 6,"62.12.31.23","annie","US" 7,"122.12.31.23","joe","CH" 8,"112.11.11.21","alice","CH" 9,"212.112.131.22","admin","RU" 10,"212.12.31.23","wendy","EU"
Once it has been uploaded, it will look like what you see in figure below.
Figure 57. Upload CSV File
You would use such a data table together with the
lookup()
andmatch()
functions to add labels to the results of a search. Notice that the values are in quotes, except for the ones for userid, which are integers. See the Lookup API reference page for more information on this topic.Edit the data in the table as you wish, and click
to add rows and columns.Once you have finished editing, click
.
Lookup Files Operation
When using the Lookup Files
and
match()
functionality, consider the following:
Lookup files use server memory proportional to the size of the file on disk; at least as much and typically more. If you have a 1Gb lookup file it will take up at least 1Gb of memory on some, potentially all, hosts within the cluster. This requirement should be taken into account when uploading and sizing the nodes within the cluster.
Before LogScale v1.89.0 maintains a copy of a file for each different query that uses it. Therefore, if you have 2 different queries using a 1G file then that will occupy at least 2G of memory.
LogScale v1.90.0 maintains a copy of the file for each choice of arguments to match (from 1.90 on). Therefore you can have different queries that use the same match without the file being copied but if the arguments to match are different, say they use the same file but different values for field, then the file will be duplicated in memory.
LogScale restarts all queries that use a file whenever the file is updated. If you don't update your files it makes little difference if you have many small or one large. If you do update your files then prefer to have fewer bigger files since many updates to small files leads to many query restarts.
From 1.90 on, if you have large lookup files, wrapping the uses of
match()
in saved queries rather than use them directly across multiple different queries. That way you're sure you don't accidentally pass slightly different argument in different queries.