Using Lookup Files

Lookup files allow LogScale to use a previously uploaded file as the basis of the data to be used in the join. These lookups files can be used when there is a defined set of data.

The primary use case is to upload a CSV (of JSON) file that contains the lookup information. The file could come from another source including static lookup files, reference files, system data (for example the system services or network files), or data exported or generated from other applications and systems.

For example, using a file that has been uploaded that contains a defined set of protocols, lookups can be used to match protocols found in a query and showing the protocol name or port number.

To use lookup files in queries, the file must be uploaded to LogScale. This can be achieved through the UI using Lookup Files or GraphQL. Alternatively, the process can use information from a query and then automatically upload the file to be used in a lookup. See Using Lookups with Auto-updated Files for more information.

LogScale supports the following lookup methods:

  • Using match() to lookup data in query and extend the fields in a result set with the lookup information.

  • Using readFile() as the file input to a query. The advantage of using the readFile() function instead of the match() function, is that the lookup will not be matched against data.

Using Lookups with Auto-updated Files

A typical use case with readFile() is to use a scheduled search or alert with the Upload File action to automatically update the contents of a scheduled search to a file to use as the basis for a join.

In the following diagram, a scheduled search is used to generate results that are uploaded as a file as part of the action:

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% graph LR r["Lookup Repository"] ss("Scheduled Search") uf[["Upload File"]] pq["Query"] j("readFile()") r --> ss ss --> uf uf --> j pq <-->j

For example, this could be used in the following situation:

  • A scheduled search periodically queries a repository to identify a list of users to be investigated.

  • When the scheduled search creates an output list, the data is uploaded as a file using the Upload File action.

  • A dashboard query executes a Scheduled Search that uses the uploaded file to search for suspicious usernames in a repository of logins.

Using the Scheduled Search method enables a join between a regularly executed query and a dataset. The two datasets could be across different repositories or the same.