Getting Data In

Before you begin sending data to LogScale, it helps to understand the core concepts behind how data ingestion works. This section explains the key ideas, components, and terminology you will encounter when getting data into LogScale, so that you can make informed decisions about how to set up and manage your data pipelines.

What Is Data Ingestion?

Data ingestion is the process of collecting data from one or more sources and delivering it to LogScale for storage, processing, and analysis. The data sources can be almost anything that generates log or event data.

Here are some common data sources that produce log data:

  • Servers

  • Applications

  • Network devices

  • Cloud services

  • Security tools

Once data arrives in LogScale, it is parsed, structured, and stored so that it can be searched and analyzed in real time or retrospectively. The ingestion pipeline (from source to storage) is the foundation of everything you do in LogScale.

At a high level, the ingestion pipeline looks like this:

flowchart LR A([Data Source]) --> B[Collection /\nTransport] B --> C[LogScale\nIngest Endpoint] C --> D[Parser] D --> E[Repository /\nView] E --> F([Search &\nAnalysis])
flowchart LR A([Data Source]) --> B[Collection /\nTransport] B --> C[LogScale\nIngest Endpoint] C --> D[Parser] D --> E[Repository /\nView] E --> F([Search &\nAnalysis])

Each stage of this pipeline is described in more detail in the sections that follow.

Core Steps

Getting data into Falcon LogScale consists of three core steps:

  • Configure Ingest: Set up data sources and ingest methods

  • Parse Data: Define how your data should be processed

  • Validate Data Flow: Ensure data flows correctly into Falcon LogScale

Falcon LogScale takes each raw log line and parses the content before writing the information into a repository. While Falcon LogScale can query raw log lines, parsing extracts key information into specific fields for more efficient filtering, correlation, and summarization.

Data ingestion in Falcon LogScale is optimized for real-time data. Follow the guidelines provided to manage historical data efficiently.

graph LR A([Configure Ingest]) --> B([Parse Data]) --> C([Validate Data Flow]) %% Styling style A fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:10px style B fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:10px style C fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:10px
graph LR A([Configure Ingest]) --> B([Parse Data]) --> C([Validate Data Flow]) %% Styling style A fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:10px style B fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:10px style C fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:10px

Figure 1. Getting Data In process


How Concepts Connect

These concepts are interdependent. Parser design influences tag selection. Tag selection determines how data sources are created. Data source structure governs query performance at scale. A thorough understanding of this chain — and the implications of each decision within it — is the foundation of a well-functioning Falcon LogScale deployment.