Field Naming and Standards
There are no required or standard fields built into the LogScale data structure. However, there are conventions and standards to follow that can help to identify different fields, field types, and field naming conventions.
Field names are determined by the data that is ingested and parsed by LogScale. Field names are flexible and support the following characters:
Any alphanumeric character, upper or lower case.
Most punctuation characters, except as noted below.
The space character; if a space is used in the name, the field name must be enclosed in double quotes, for example
"host name".
Characters not followed in a field name:
$This character is reserved for identifying saved searches.
()Parentheses are reserved for identifying functions.
Double quotes (
") unless escaped:\"
In addition to these naming conventions, the following field name and structure conventions are followed:
Fields may be declared as a given type according to a single character prefix, as described in Field Types.
Some common fields are created during ingestion and used by LogScale during querying. For more information, see Standard Fields.
All other fields within LogScale are entirely driven by the user. For standardization and common query processing, there are standards and systems used to harmonize the structure of the data ingested. For more information, see Event Fields Standardization.
Field Types
There are different kinds of fields coming from the events:
| Prefix | Type | Description |
|---|---|---|
@
| Metadata fields | Contain metadata about each event extracted during ingestion. These are typically created by LogScale automatically as data is ingest. For example @id or @timestamp. See Standard Fields for the list of these fields. |
#
| Tag fields | Used to identify event types and influence how data is grouped and stored Tag Fields for the list of these fields. |
_
| Temporary fields | Fields with a leading underscore are temporary fields. These can be generated by functions (the default field prefix for functions creating new fields is the underscore), or when creating or assembling fields during processes. |
| Â | User fields | Any field that is not a tag field or metadata field. |
Note that any field or field type can be created at any time, and metadata and tag fields can be deleted or overwritten during parsing and querying. These are conventions rather than restrictions.
Standard Fields
Each event has some metadata attached to it on ingestion; all metadata fields start with @ to make them easy to identify. All events will contain the following metadata fields by default.
| Metadata Field | Description |
|---|---|
| @rawstring | The original text of the event. Because it keeps the original data on ingestion, this field allows you to do free-text searching across all logs and to extract virtual fields in queries. |
| @id | A unique identifier for the event. Can be used to refer to and re-find specific events. |
| @timestamp |
Timestamp in milliseconds since the epoch
(1st Jan 1970, 00:00) of the
ingested event, for example
2022-11-22 09:50:20.100 if
the event has an identifiable timestamp. Timestamps are in
Coordinated Universal Time (UTC). See also
Parsing Timestamps for more details.
|
| @timezone | The timezone the event originated in, if known. This is often set when the event's timestamp is parsed. |
| @ingesttimestamp | The timestamp of when the event was ingested. The value is milliseconds-since-epoch. Timestamps are in UTC. |
| @timestamp.nanos |
Extended precision of timestamp below millisecond. For example,
295000. Timestamps are in
UTC. See also Parsing Timestamps for
more details.
|
Each event may also have fields with tagged data attached added during ingestion. The list of tag fields is dependent on the parser used when ingesting the data, but typically will include the following fields:
Event Fields Standardization
LogScale is designed to work with unstructured data while still allowing for the data to be queried, formatted, and extracted. While this flexibility allows for different log files and information to be ingested, when querying data in common ways, the use of a common standard for naming of fields that contain the same information is critical.
This flexibility can make querying data with common values or fields challenging. Standardizing or agreeing on a data structure and set of common fields that contain information for security-related queries can simplify the querying process. For example, when searching for a hostname or IP address, if the information is not in the same field name, then multiple fields will need to be searched.
A number of options are available to improve standardization:
Built-in Parsers
Some formats of data, such as syslog or HTTP logs, can be ingested using one of the standard parsers built into LogScale. These follow a standard to allow searching and correlation of fields containing similar data, such as IP addresses.
For more information, see Built-in Parsers.
Integrations and Packages
LogScale supports a set of packages and integrations that will parse and process data from many third-party log sources. These packages and integrations follow the CrowdStrike Parsing Standard (CPS), which defines a standardized set of fields to ensure consistent data.
For more information on integrations, see Package Marketplace.
CrowdStrike Parsing Standard (CPS)
The CPS defines a standard set of fields for storing common information, and then sets a mapping from the source fields from ingested data, into the identified common field within the CPS. The basic structure is modeled on the Elastic Common Schema, with customizations for LogScale.
For more information on CPS, see CrowdStrike Parsing Standard (CPS) 1.2.
Field Aliasing
The field aliasing system enables you to create alternative names, aliases, for existing fields so that multiple fields can be queried using a common alias.
See Field Aliasing for more information.