How to Get Data in

Overview

Falcon LogScale is only as valuable as the data inside it. Getting data in efficiently and correctly is the foundation everything else is built on — your queries, dashboards, alerts, and investigations all depend on data arriving in a form that Falcon LogScale can organize and search effectively.

Falcon LogScale supports a wide range of ingest methods including the HEC (HTTP Event Collector) API, Syslog, Beats, Elastic Bulk API, and direct file uploads. You can send structured data like JSON or CSV, semi-structured logs, or unstructured text. The challenge is not whether Falcon LogScale can accept your data — it is making deliberate choices about parsers, tags, and repository structure to ensure the data is useful when it arrives.

Basic Flow

At its simplest, getting data into Falcon LogScale follows a straightforward path:

graph LR A([Data Source]) --> B([TransportMechanism]) B --> C([IngestToken]) C --> D([Parser]) D --> E([Repository]) %% Styling style A fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px style B fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px style C fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px style D fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px style E fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px
graph LR A([Data Source]) --> B([TransportMechanism]) B --> C([IngestToken]) C --> D([Parser]) D --> E([Repository]) %% Styling style A fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px style B fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px style C fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px style D fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px style E fill:#3498DB,stroke:#2C3E50,color:#ffffff,font-size:12px

Figure 2. Data ingest flow


  • Data Source — the system generating logs or events (application servers, cloud services, network devices, containers)

  • Transport Mechanism — how data reaches Falcon LogScale. Common options include __flc_name__, Filebeat, Logstash, the HEC API, Syslog, or the Elastic Bulk API. Each has different performance characteristics and deployment requirements.

  • Ingest Token — a repository-specific token that authenticates the connection and routes incoming data to the correct repository. Each ingest token is associated with a single repository and optionally assigns a default parser.

  • Parser — a set of rules written in Falcon LogScale query language that extracts fields from raw log lines, assigns timestamps, and optionally assigns tags. Built-in parsers handle common formats like JSON, accesslog, and CEF. Custom parsers handle application-specific formats.

  • Repository — the storage container where parsed events are organized into data sources based on tag values. Repositories have their own retention settings, access controls, and query scope.

Each step in this chain is a decision point. The choices you make affect how useful and performant your data will be.

Start with Intentionality

The most common ingest mistakes are not technical — they result from moving too fast. Sending data before understanding its structure, skipping parser testing, or choosing high-cardinality fields as tags can create problems that are difficult to fix later. Changing tags after ingestion requires reingesting all data. Poorly designed parsers that miss critical fields or extract timestamps incorrectly affect every query downstream.

Taking time to assess your data, test parsers in a development repository, and validate tag cardinality before production ingestion pays dividends at every stage that follows. Falcon LogScale provides built-in tools like the Parser Tester and Data Sources view to help validate your configuration before committing to it.

The sections that follow walk through each part of the ingest process — from evaluating your data and choosing the right transport mechanism to writing parsers, assigning tags, and validating that events arrive correctly in your repository.