Event Stream

When querying in LogScale, understanding the sequence and structure of the data and how this affects the tools and methods used to query and process the information is important.

The primary structure for all query processing is the Event Stream, a collection of events composed of individual events.

When data is ingested into LogScale, the information is stored with the following attributes:

  • All data is stored with a timestamp in the @timestamp field, which is derived either from the time of the original event, or the time the data was ingested.

  • The date and time the data was ingested into LogScale is stored in the @ingesttimestamp field.

  • Individual elements of the incoming data can be extracted and stored in a named field, like hostname or login.

  • Additional fields can be added to accomplish tasks like classifying data by type or defining the source of the data.

  • The original event text is also stored in full in the @rawstring field.

When ingesting the line from log file, each row will be converted to a single event, and each event will include one or more fields extracted from the original row, along with the timestamp and original raw event data.

In the following example, notice that some fields will be common to each event, like the hostname (LON=SRV01 or MAIL01). Some specific types of rows of data are also common, like COMMAND or PWD. The fields extracted from a given event are defined by the parser, and LogScale supports a number of standardized parsers for extracting fields from common events:

Raw Events
2025-08-13T16:05:49.453Z DEV-TEST01 audit[33456]: file_operations={"paths": ["/home/user/documents/report2025.pdf", "/var/log/system/error_23.log", "/etc/config/settings.conf"], "operations": ["read", "write", "delete"], "user_ids": [123, 456, 789]}
2025-08-13T16:05:50.079Z LON-SRV01 falcon_sensor[NYC-SRV01]: process_tree={"parent_pid": 234, "child_pids": [1234, 5678, 9012], "cmd_lines": ["/usr/bin/grep", "/usr/sbin/useradd", "/bin/ls"]}
2025-08-13T16:05:50.453Z SYD-SRV01 httpd[62145]: 192.168.4.198 - - [2025-03-13:17:30:15 +0200] "GET /downloads/software/installer.exe HTTP/1.1" 401 7891
2025-08-13T16:05:51.974Z TYO-SRV01 CRON[28934]: (svc_sql) CMD (/usr/bin/find)
2025-08-13T16:05:52.975Z MAIL01 kernel: [Hardware Error]: CPU 0: Machine Check Exception
2025-08-13T16:05:52.976Z DNS01 sudo[49123]: bakerm : TTY=pts/0 ; PWD=/home/blackj ; USER=root ; COMMAND=/sbin/ifconfig

For more information about parsers, see Parse Data.

Important

Because LogScale always stores the original raw event string, even if the field is not extracted during ingestion, the data can be extracted into a field during querying. The parsing language is the same as the query language, so all of the tools and functions are available to you in both environments.

The flexibility of the data and the ability to process and extract information from the original raw event string allows LogScale to store data without having to know the format of the information.

When querying the data, the format and nature of the data will affect the methods used to extract the information. Most queries will follow one or more of the following steps:

  • Filter the data (search) either by looking at the raw text or specifying values from specific fields

  • Aggregate and/or summarize the information to build a simplified view of the event data

  • Format the event data; converting numbers or formatting strings to match the required output format

The structure of individual events is described in Event Structure.