Part of our Foundational Concepts series:

The data stored in LogScale are called events. An event is a piece of data and an associated timestamp.

Examples of events include:

  • Log Lines

  • Metric Data

  • Analytics Data

However, any piece of data with an associated timestamp can be thought of as an event.

When data is sent to LogScale the associated parser converts the data into an event. For example, suppose LogScale received this log line:

[2018-10-11 22:00:10] INFO - User Logged In. userId=97110

The result might be turned into an event data containing the following fields:

Field Value
@rawstring [2018-10-11 22:00:10] INFO - User Logged In. userId=97110
@id 3gqidgqi_uwgdwqu121duqgdw2iqwud_721gqwdugqdwu1
@timestamp 2018-10-11 22:00:10
@timezone Z
#repo server-logs
#type my-parser
loglevel INFO
message User Logged In
userId 97110

Field Types

There are three types of fields:

Metadata Fields

Each event has some metadata attached to it on ingestion; all events will have an @id, @timestamp, @timezone, and @rawstring field.

Notice that all metadata fields start with @ to make them easy to identify.

The two most important are @timestamp and @rawstring and will be described in detail below.

Tag Fields

Tag fields define how events are physically stored and indexed. They are also used for speeding up queries.

Users can associate custom tags as part of the parsing and ingestion process but their use is usually very limited. The only built-in tags are #repo and #type and both are described in detail below.

Usually the client sending data to LogScale will be configured to include #host and #source tags that contain the hostname and file that the event was read from.

User Fields

Any field that is not a tag or metadata is a user field. They are extracted at ingest by a parser or at query time by a regular expression. User fields are usually the interesting part of an event, containing application specific information.

Field @rawstring

LogScale represents the original text of the event in the @rawstring attribute.

One of the greatest strengths of LogScale is that it keeps the original data and nothing is thrown away at ingest. This allows you to do free-text searching across all logs and to extract virtual fields at query time for parts of the data you did not even know would be important.

You can read more about free-text search and extracting fields in the search documentation.

Field @timestamp

The timestamp of an event is represented in the @timestamp field. This field defines where the event is stored in LogScale's database and is what defines whether an event is included in search results when searching a time range.

The timestamp needs special treatment when Example: Parsing Log Lines during ingestion.

Field @ingesttimestamp

The timestamp of when an event was ingested is represented in the @ingesttimestamp field. The value is milliseconds-since-epoch. Searches can restrict the data they search using this timestamp. Currently this can be done by including @ingesttimestamp > X AND @ingesttimestamp < Y early in the search pipeline, where X and Y are the bounds as milliseconds since 1/1/1970.

Field #repo

All events have a special #repo tag that denotes the repository that the event is stored in. This is useful in cross-repository searches when using views.

Field #type

The type field is the name of the Parsing Data used to ingest the data.

Part of our Foundational Concepts series: