CrowdStrike Parsing Standard (CPS) 1.0
Following CrowdStrike Parsing Standard (CPS) helps you ingest data in a way that simplifies writing queries that combine data across different data sources. This page provides you with an introduction on how to write parsers that conform to this standard, focusing on normalization.
The majority of parsers on the Marketplace Integrations follow this standard. For overall guidance on writing parsers in LogScale, see Creating a Parser and Developer Guidelines for Packages.
What is normalization in LogScale?
Extracting fields and tags during the parsing stage is crucial for search performance. Mapping different vendor field names (for example timestamp or IP) during the parsing stage to a unified field name helps you simplify your queries. Instead of switching between different fields that contain common information, for example ip_source and source-ip depending on the log format, you will be able to use consistent field names across differet vendors and different log formats.
Why CrowdStrike Parsing Standard (CPS)
The Crowdstrike Parsing Standard builds on the Elastic Common Schema (ECS) 8.x. ECS isn't specific to any data store, which gives us a lot of flexibility. It's a mature and proven common schema for metrics, logs, traces and resources, managed by the OpenTelemetry community which shares our interest in the convergence of observability and security. See ECS Categorization fields for more detail on ECS fields.
CPS differs from ECS in a number of ways that build on the specifics of LogScale core architecture. For example, parsers following CPS make all fields in a log event available as actual LogScale fields, even if they don't match a field in ECS. See Variations to the ECS for more details on the differences between ECS and CPS.
For any given data source, when developing a parser, you can decide which domain specific fields are applicable to the data.
To create CPS compliant Parsers:
Use CPS Compliant Parser tags:
#Cps.version
#Vendor
#ecs.version
#event.dataset
#event.kind
#event.module
#event.outcome
#observer.type
Your parsers must populate the fields described in CPS Compliant Parser Fields
Manage any fields which do not match the ECS fields, see Managing Non ECS Fields.
Add any fields you need to the standard, see Adding Fields to the Standard.
CPS Compliant Parser Fields
The fields in the table below define the CPS compliant fields expected to be created during parsing:
Fields | Description | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Event categorization fields (kind, type, category, outcome) |
| |||||||||
ecs.version |
This field contains the version of ECS that is being followed by the parser. | |||||||||
Cps.version |
| |||||||||
Parser.version |
| |||||||||
Vendor |
| |||||||||
event.module | contains roughly the name of the product or service that the event belongs to. Existing event.module values is reused whenever appropriate. | |||||||||
event.dataset |
|
Managing Non ECS Fields
CPS compliant parsers strive to make all fields in a log event available as actual LogScale fields, even if they don't match a field in ECS:
If Fields from the event which do not exist in ECS, their name is prefixed with the string literal Vendor .
This gives the ECS fields the
root
namespace
, while vendor specific fields can always be found with the Vendor. prefix.If a field can exist as both an ECS field and a vendor specific field, the following logic applies:
If the value of both fields is byte-for-byte the same, it only preserve the ECS field, and discards the vendor specific field.
If the value of the fields differ, both fields are preserved. For example, an ECS field may require its value be lower cased, but the original log has mixed casing. In that case, the vendor specific field contains the original, mixed case value.
Adding Fields to the Standard
When adding new fields to this standard, all fields which are not taken directly from ECS must have a capital letter from the point where they differ from the schema.
Using capital letters for field names follows the guidance from ECS on how to add event fields outside the schema.
Example of fully custom field: Parser.version is similar to ecs.version, but the Parser
namespace
is our own, so must start with a capital letter.Example of extending ECS with custom field: observer.Fictional_field where observer is an existing namespace in the schema, but Fictional_field is our own field inside that namespace.