Event Data Sources

All queries operate on a stream of events in list format and typically in time order, where each event contains one or more fields. However, different sources of data exist for the purposes of querying the data, and the different source of data implies a different basis for the query.

AttributeRepositoryView defineTable() readFile() createEvents()
Source Data Stored (segment) data One or more repository (local or remote) Query repository or view Lookup file From supplied content
Filtering None Filter expression Subquery None None
Time Selection Query time selector (UI or API) Query time selector (UI or API) Relative to query time selector None None

The following sources of data for querying are available:

  • Repository

    When reading data from an existing repository, the data will be composed of the raw event data, in time-series order. Time specification of the data is defined by the Time Interval Selector.

  • View

    A view is a filtered combination of repositories, designed either to combine the data from multiple repositories or filter data from one or more repositories. The benefit of a view is that it can be configured to show a very limited set of data, and also be limited to a set or group of users.

    Views can be composed of different components:

    • Event data from one or more repositories, which is combined into a single stream of events according to the event timestamp.

    • A repository with a filter expression applied. Filter expressions on views limit the data to particular types, groups, users and/or other items based on the query data. The query filter is applied before the event stream is presented to the query mechanism. In the following example, two separate repositories use a filter to select a subset of data which is then presented in the view Host to be queried:

      graph LR; A["Metrics Repo"] B["Security Repo"] C["Host View"] A --#type=host--> C B --#type=hostname--> C
      graph LR; A["Metrics Repo"] B["Security Repo"] C["Host View"] A --#type=host--> C B --#type=hostname--> C

    • Views or repositories for other clusters when using multi-cluster search.

    One or a combination of these sources is possible in a view. For example, a filter on a remote multi-cluster repository or filters on some but not all repositories.

    Time selection of the data from a view is limited by the Time Interval Selector.

  • defineTable() Function

    The defineTable() function enables the execution of a query on a repository or view and the use of the result as the basis of the subsequent query. This can be useful for creating a subset of data, or joining data from multiple subqueries together without creating a view.

    The following example creates a new set of events from a repository, filtering based on the domain with each event only including the host and ipaddr fields:

    logscale
    defineTable(query={domain="crowdstrike.com"}, name="tablename", include=[host,ipaddr])

    defineTable() can be useful when data needs to be filtered or combined with other data, or when creating a dataset that is limited with a different time selection than the original source data.

  • readFile() Function

    The readFile() function composes data previously loaded into the lookup files system as JSON or CSV data and generates the content as a stream of events. The columns of the CSV or keys of the JSON file are translated into the fields of the events.

    While the data from readFile() cannot be time limited, the uploaded file can be strictly controlled.

  • match() Function

    The match() function is a form of join that looks up data and matches it to the incoming event set, with the effect being to update an existing event stream with values matched in the lookup file, or combining data with events.

  • createEvents() Function

    The createEvents() function can be used to create some sample data typically for testing or validating different values.

Different data sources can be used in combination with each other, so it's possible to use defineTable() on a view that also uses match() to lookup data.

When used in combination, the execution priority for each data source of a query or filter is important. In sequence, data will be recovered in order from the following sources:

  1. Views, including any filters on each repository

  2. createEvents()

  3. defineTable()

  4. match(), readFile()

When executing a query within these parameters, consider that different parts of the query will execute before others.