Lookup Files

Security Requirements and Controls

Lookup files enable you to attach or replace text from events recorded in a repository when searched.

To add a lookup file, you create and import a CSV (comma-separated value) file and upload it to the repository. These files can be used together with query functions to provide lookups and matching using the match().

The following operations are available:

For information on how Lookup files interact with the rest of the system, see Lookup Files Operations.

Creating a File

  1. Click Files → + New File → Create New.

  2. Specify a name for the file and then select either + Empty File to create an empty file to populate or From Package to use a template from a previously installed package.

  3. Click Create file.

  4. If you've created an empty file, click + to add rows and columns.

  5. Click Save to save the changes.

If you have many changes to make, editing a data table through the Files interface page can be tedious: click Export and then edit the table in a spreadsheet program or a simple text editor.

Note

Files larger than 100 MB cannot be viewed in the UI.

Create New CSV File

Figure 32. Create New CSV File


Uploading a File

  1. Click Files → + New File → Import File.

  2. Browse for the file to upload and click Open.

    You can upload a CSV file containing text like what you see below, which is essentially a lookup table that you can use for labels or value lookups.

    yaml
    userid,ip,username,region
1,"212.12.31.23","pete","EU"
2,"212.12.31.231","bob","EU"
3,"98.12.31.21","anders","EU"
4,"121.12.31.23","jeff","US"
5,"82.12.31.23","ted","AU"
6,"62.12.31.23","annie","US"
7,"122.12.31.23","joe","CH"
8,"112.11.11.21","alice","CH"
9,"212.112.131.22","admin","RU"
10,"212.12.31.23","wendy","EU"

    Once it has been uploaded, it will look like what you see in figure below.

    Import CSV File

    Figure 33. Import CSV File


    You would use such a data table together with the match() functions to add labels to the results of a search. Notice that the values are in quotes, except for the ones for userid, which are integers. See the Lookup API reference page for more information on this topic.

  3. Edit the data in the table as you wish, and click + to add rows and columns.

  4. Once you have finished editing, click Save, or click Export if you wish to download the edited file.

Exporting or Deleting a File

Files can be managed by clicking the menu icon next to each file. You can either export or delete a file:

Manage CSV Files

Figure 34. Manage CSV Files


Lookup Files Operations

When using Lookup files and match() functionality, consider the following:

  • Lookup files use server memory proportional to the size of the file on disk; at least as much and typically more. If you have a 1Gb lookup file it will take up at least 1Gb of memory on some, potentially all, hosts within the cluster. This requirement should be taken into account when uploading and sizing the nodes within the cluster.

  • Up to version 1.89.0, LogScale maintains a copy of a file for each different query that uses it. Therefore, if you have 2 different queries using a 1G file then that will occupy at least 2G of memory.

  • Up to version 1.116, LogScale restarts all live queries and alert queries that use a file whenever the file is updated. If you don't update your files it makes little difference if you have many small or one large. If you do update your files then prefer to have fewer bigger files since many updates to small files leads to many query restarts.