Event Fields
In LogScale, you can search either the raw data of events or the fields extracted from the event when the data is parsed.
An event is the smallest fragment of data in LogScale. Events are the basis of all storage and queries. Collections of events are stored in repositories. When querying data, the query is executed on a sequence of those events across a given time range.
Each event may have a different set of fields, a different schema, and this is valid. For example, in the HTTP log example there is no authentication or user, but there could be lines in the log that contain that information. LogScale does not use or require a fixed schema for storing the data, and you do not to define the data structure, validation or indexes before the data can be ingested.
A single repository may therefore contain multiple source log data consisting of different formats, and events. This flexibility enables you to query multiple log files simultaneously and the query language provides a powerful mechanism for filtering and formatting the data.
LogScale also stores the original @rawstring of the log line. This is because the parser may pick specific fields of information but not handle all the different formats. Because we don't want to worry about these differences or lose the data, that original log line is stored and can be queried and processed when searching.
There are different kinds of fields coming from the events:
Metadata fields using the prefix @ contain additional information about the event such as the time when the data was ingested, or the source file or host of the data. All events will have these default fields, for example @id or @timestamp. See Metadata Fields for the complete list of these fields.
Tag fields using the prefix # define how events are physically stored and indexed. See Tag Fields for the list of these fields.
User fields is any field that is not a tag field or metadata field, and contains the information provided and/or parsed from the original source data.
Event fields can be viewed and managed from the LogScale User Interface. See Search Data.
For more information on how to query event fields in LogScale and discover what you can achieve with query writings, see Query Management and Common Queries.
Metadata Fields
Each event has some metadata attached to it on ingestion; all metadata fields (also called system fields) start with @ to make them easy to identify. All events will contain the following metadata fields by default.
| Metadata Field | Description |
|---|---|
| @rawstring | The original text of the event. As it keeps the original data on ingestion, this field allows you to do free-text searching across all logs and to extract virtual fields in queries. |
| @id | A unique identifier for the event. Can be used to refer to and re-find specific events. |
| @timestamp |
Timestamp in milliseconds since the epoch
(1st Jan 1970, 00:00) of the
ingested event, for example
2022-11-22 09:50:20.100 if
the event has an identifiable timestamp. Timestamps are in UTC.
See also Parsing Timestamps for more
details.
|
| @timezone | The timezone the event originated in, if known. This is often set when the event's timestamp is parsed. |
| @ingesttimestamp | The timestamp of when the event was ingested. The value is milliseconds-since-epoch. Timestamps are in UTC. |
| @timestamp.nanos |
Extended precision of timestamp below millisecond. For example,
295000.
Timestamps are in UTC. See also
Parsing Timestamps for more details.
|
Tag Fields
Each event has some tagged data attached to it on ingestion; all tag fields start with # to make them easy to identify. All events will contain the following tag fields by default.