Data Representation in LogScale

LogScale operates as an event management platform, where data is divided into events and each event represents a discrete piece of data. Information sources include log management and security operations, and each event represents the information generated by a single line or even from source data.

Events:

  • Are stored within a repository, and each repository represents a collection of events. They can be used either to organize the events by their type, source, or other value.

  • Consist of fields, and can contain information added during ingestion (when data is added), during an optional parsing step (configured during ingestion), and/or generated during the query process.

  • Include original raw string event data by default and any fields created during parsing and ingestion. This allows for some values to be identified during ingestion, and other values to be extracted only during querying if required.

  • Has fixed fields when reading data from a repository at the source level, including a timestamp- this is typically extracted from the source event data, and is accompanied with an ingest time stamp, representing the date/time when the data was added to the repository.

    During a query, however, the content and format of each event can change.

  • Are nominally unstructured. Aside from the metadata fields timestamp and raw string, the data does not need to have a specific format or structure.

This flexible, unstructured format has an impact on the way the data is queried. During a query:

  • Data can be queried by examining the original data string

  • Data can be queried by using any fields generated during parsing

  • The event field structure is flexible; new fields can be extracted, created, modified, and deleted during query execution.

    For example, additional fields can be extracted from the original event text, or a new field created via calculation or another operation.

  • Fields can be removed, merged, or completely replaced with a different structure.

    For example, using an aggregation where events are summarised by counting events, the entire event set will get replaced with the new aggregated structured.