Basic Concepts

The most important ingestion concepts.

Events and Log Data

The fundamental unit of data in LogScale is an event.

An event is a single, timestamped record that represents something that happened at a specific point in time. For example, a user logging in, a firewall blocking a connection, or an application producing an error.

Events arrive in LogScale as raw log data. This raw data can take many forms depending on the source:

  • Unstructured text - Free-form log lines produced by many applications and operating systems, where the meaning of each part of the line is not explicitly labeled.

  • Semi-structured text - Formats such as Syslog or Common Event Format (CEF), which follow a defined structure but may contain free-form content within that structure.

  • Structured data - Machine-readable formats such as JSON or XML, where each piece of information is explicitly labeled with a key and a value.

Regardless of the format in which data arrives, LogScale uses parsers to interpret and structure it into a consistent set of fields that can be searched and analyzed. See Parsers for more information.

Repositories and Views

In LogScale, data is stored in a repository.

A repository is a dedicated storage container for a collection of events. When you ingest data, you direct it to a specific repository, where it is indexed and made available for search and analysis.

You can have multiple repositories in LogScale, which allows you to organize your data logically. For example, by data source type, business unit, etc.

A view is a virtual 'layer' on top of one or more repositories. Views allow you to query data across multiple repositories as if they were a single data set, but without physically merging or changing the underlying data. Views are useful when you want to:

  • Search across data from multiple sources in a single query.

  • Apply role-based access controls to restrict what data different users or teams can see.

  • Present a filtered or transformed subset of data to a specific audience.

When you are setting up data ingestion, you will need to decide which repository your data should be directed to. This is configured as part of the ingest token setup. See Ingest Tokens for more information.

Ingest Tokens

An ingest token is a credential that authenticates data being sent to LogScale.

Every data source that sends data to LogScale must present a valid ingest token. Without a token, LogScale will not accept the incoming data.

Ingest tokens serve two important purposes:

  • Authentication - The token proves that the sender is authorized to write data to LogScale.

  • Routing - Each token is associated with a specific repository, so LogScale knows where to store the incoming data.

It is recommended practice to create a separate ingest token for each data source. This approach provides several benefits:

  • You can identify which source sent which data, making auditing and troubleshooting easier.

  • You can revoke or rotate the token for one source without affecting any other sources.

  • You can apply different parser assignments to different tokens, so that data from each source is processed appropriately.

Ingest tokens are managed at the repository level within LogScale.

Parsers

A parser is a component in LogScale that interprets raw incoming log data and transforms it into a structured set of fields and values.

Parsers are a critical part of the ingestion pipeline because they determine how your data is organized and what you can do with it once it is stored.

When raw log data arrives at LogScale, it is initially just a string of text. The parser reads that text, identifies meaningful pieces of information within it, and extracts them as named fields. For example, a parser processing a web server access log might extract fields such as:

  • timestamp - When the request was made

  • client_ip - The IP address of the client

  • http_method - The HTTP method used (GET, POST, and so on)

  • status_code - The HTTP response code returned by the server

Once these fields are extracted, they can be searched, filtered, and aggregated in LogScale queries.

LogScale supports several types of parser:

  • Built-in parsers - LogScale includes a set of built-in parsers for common log formats and protocols, such as JSON and Syslog.

  • Package parsers - Many packages available in the LogScale Package Marketplace include parsers specifically designed for a particular data source, such as a firewall vendor or a cloud platform.

  • Custom parsers - If no suitable parser exists for your data source, you can write your own using the LogScale query language and regular expressions.

A parser is assigned to an ingest token, so that all data arriving on that token is automatically processed by the correct parser.

Fields and Tags

Once a parser has processed an incoming event, the data is represented as a collection of fields.

A field is a key-value pair that represents a single piece of information extracted from the event. For example, status_code=404 or username=jsmith.

Fields are the primary way you interact with your data in LogScale. When you write a query, you filter, search, and aggregate based on field names and values.

There are two categories of field in LogScale:

  • Parsed fields - Fields that are extracted from the raw event content by the parser at ingest time.

  • Metadata fields - Fields that LogScale adds automatically, such as @timestamp (the time the event was received) and @rawstring (the original, unmodified log line).

Tags

Tags are a special category of field in LogScale that are used to partition and organize data within a repository.

Unlike regular parsed fields, tags are indexed at storage time and are used by LogScale to determine how data is physically organized on disk.

Because tags influence how data is stored, they have a significant impact on query performance. Filtering on a tag in a query allows LogScale to quickly locate only the relevant segments of data, rather than scanning everything in the repository.

Common examples of fields that are well suited to being used as tags include:

  • The type or source of the log data (for example, #type=firewall).

  • The host or system that generated the event (for example, #host=webserver01).

Tags should be chosen carefully. Because they affect data storage, using too many tags or tags with a very high number of distinct values can have a negative impact on storage efficiency and performance.

Data Collection and Transport

Before data can be ingested by LogScale, it must be collected from its source and transported to the LogScale ingest endpoint.

The method you use to do this depends on the nature of your data source and your environment. LogScale supports several collection and transport mechanisms:

  • Falcon LogScale Collector - A lightweight agent that you install on a host system. The Collector monitors log files and other data sources on that host and forwards events to LogScale in real time. It is well suited to collecting data from servers, workstations, and other host-based sources.

  • Syslog - A widely supported protocol for transmitting log messages over a network. Many network devices, firewalls, routers, and infrastructure components can send log data via Syslog without requiring any additional software. LogScale can receive Syslog data directly or via a Syslog forwarder.

  • HTTP API - LogScale provides an HTTP-based ingest API that allows applications and services to send data programmatically. This is a flexible option for custom integrations and for sources that can make outbound HTTP requests.

  • Integration Packages - Some packages in the LogScale Package Marketplace include pre-configured collection and transport components that handle the ingestion setup for a specific data source automatically.

The choice of collection and transport method does not affect how data is stored or queried in LogScale — it only affects how data gets there. You can use different methods for different data sources within the same LogScale environment.

Packages and the Marketplace

The LogScale Package Marketplace is a library of pre-built integration packages that can simplify and accelerate the process of getting data in from common sources. A package is a self-contained bundle of content that is designed to work with a specific data source or use case.

Packages can include any combination of the following components:

  • Parsers - Pre-built parsers that understand the log format produced by the data source.

  • Dashboards - Pre-built visualizations that present key metrics and activity from the data source in a structured, at-a-glance format. Dashboards can include charts, graphs, tables, and counters that update in real time as new data arrives.

  • Alerts - Pre-configured alert rules for common threat scenarios and operational conditions relevant to that source.

  • Saved Queries - Useful queries that you can run immediately against your data without having to write them from scratch.

Using a package where one is available is generally the fastest way to get value from a new data source, because the parser and supporting content are already configured for you. However, packages are optional. You can always configure ingestion manually if you prefer, or if no package exists for your data source.

Summary

With these concepts in mind, you are ready to move on to the practical steps of getting your data into LogScale. See the Getting Data Into LogScale - Overview for a step-by-step guide to the ingestion process.

Key Ingestion Terms

Understanding the following terms is essential for successfully getting data into LogScale.

Table: Data ingestion terms and concepts

Term Description
Backfilling The process of ingesting historical log data into LogScale. Backfilling requires careful planning because LogScale is optimized for real-time streaming data. Large backfills can impact cluster performance if not properly throttled.
Beats Lightweight data shippers from Elastic (Filebeat, Metricbeat, Winlogbeat). LogScale supports ingesting data from Beats agents through the Elastic Bulk API compatibility layer.
Dashboards Ready-made visualizations that give you immediate visibility into the data from that source.
Elastic Bulk API An HTTP API protocol compatible with Elasticsearch's bulk ingest format. LogScale provides Elastic Bulk API endpoints to simplify migration from Elasticsearch or integrate with tools that support this protocol.
Event A single log entry or record stored in LogScale. Each event has a timestamp, a raw text representation, and zero or more extracted fields. Events are the fundamental unit of data in LogScale.
Field A key-value pair extracted from an event by a parser. Fields make log data searchable and enable filtering, aggregation, and correlation. Examples include status_code, user_agent, or bytes_sent.
__flc_name__ CrowdStrike's lightweight agent for collecting and forwarding logs to LogScale. __flc_name__ can tail files, collect Windows events, run commands, and enrich data before forwarding. It is the recommended collection agent for most use cases.
HEC (HTTP Event Collector) A widely-supported HTTP API protocol for sending events to LogScale. Originally designed for Splunk, HEC is supported by many log shippers and applications. LogScale provides HEC-compatible endpoints for easy integration.
Ingest Rate The volume of data being ingested per unit time, typically measured in GB/day or events/second. Understanding your ingest rate is critical for capacity planning and licensing.
Ingest Token A repository-specific authentication token that authorizes data ingestion and routes incoming events to the correct repository. Each ingest token can optionally assign a default parser to incoming data.
Package A pre-built bundle of parsers, dashboards, alerts, and saved queries for a specific data source or use case.
Parser A set of rules written in LogScale query language that extracts fields from raw log lines, identifies timestamps, and optionally assigns tags. Parsers run at ingest time and transform unstructured logs into structured, searchable events.
Parser Assignment The mechanism that determines which parser processes incoming events. Parsers can be assigned via ingest token defaults, or LogScale can automatically select parsers based on pattern matching against the raw log text.
Repository A storage container for log data with its own retention settings, access controls, and parsers. All data must be ingested into a specific repository. Repositories provide logical separation between different data sets or teams.
Retention How long LogScale stores data before automatically deleting it. Retention is configured per repository and is based on data age and storage size limits. Longer retention increases storage costs.
Syslog A standard protocol for sending log messages over UDP or TCP. LogScale can receive syslog data directly and parse it using built-in or custom parsers. Syslog is commonly used for network devices, firewalls, and Unix systems.
Tag A special field that LogScale uses to physically organize events into data sources. Tags should be low-cardinality fields like host, environment, or service that efficiently partition your data for query performance.
Timestamp The time associated with an event. LogScale uses timestamps to order events chronologically and to enforce retention policies. Parsers extract timestamps from log data or use ingest time as a fallback.

Table:

TermDescription
Parser Editor

An integrated editing environment in the LogScale web interface for writing and testing parser scripts. The Parser Editor displays the parser script on the left panel and test cases on the right panel, allowing you to run tests against sample log events and inspect the resulting parsed fields.

Persisted Aggregation

A scheduled query that summarizes data from a source repository and writes results to a dedicated persisted aggregation repository. Results are stored as events and can be queried directly, independently of the source data. In LogScale, persisted aggregations transform the platform from a log search tool into a hybrid analytics engine by pre-calculating and caching aggregated results such as count, average, sum, and percentiles on a fixed schedule. This eliminates the need for compute-intensive real-time aggregations, significantly reducing query latency and resource costs while enabling long-term retention of aggregated data at reduced storage costs. Persisted aggregations are particularly valuable for high-frequency analytic workloads such as dashboards and baseline analysis where query speed and resource efficiency are priorities.

Persisted Aggregation Backfill

A one-time historical data generation process that runs when a persisted aggregation is first created. When you specify a backfill duration, LogScale generates aggregated results for past intervals as if the aggregation had been running historically. For example, specifying a 30-day backfill duration creates results for the previous 30 days of data. Unlike scheduled search backfill limits, persisted aggregation backfill is a one-time operation that executes sequentially from the earliest interval forward. Mutable references such as lookup files and saved queries are resolved using their current state at backfill execution time, not their historical state. Backfills follow the same considerations as backfilling any data in LogScale. If persisted aggregation execution lags too far beyond current time, the status may show as backfilling.

Persisted Aggregation Repository

A special repository dedicated to storing pre-computed results from persisted aggregations. Unlike standard repositories that store raw event data, persisted aggregation repositories only accept writes from LogScale persisted aggregations and cannot receive log data through standard ingest methods. These repositories store aggregated metrics instead of raw event data, and their results can be retrieved using the readPersistedAggregation() function. Persisted aggregation repositories have restricted configuration options with the repository type locked to Default and usage tags automatically categorized separately for monitoring purposes, ensuring proper separation between metric storage and raw log data.

Repository

Organized collection of data where LogScale stores different sources of data such as events, system logs and metrics.

Scheduled Search Backfill

The maximum time window that a scheduled search queries backward from the current time on each execution. For example, a scheduled search with a 24-hour backfill limit queries the last 24 hours of data every time it runs. This setting determines how much historical data is included in each scheduled execution and helps control query resource consumption. The backfill limit applies to every execution of the scheduled search, not just the first run.