What Data Can LogScale Ingest?

Falcon LogScale is designed to ingest and analyze data from almost any source, making it a highly versatile platform for both security and observability use cases. Whether you are collecting logs from cloud platforms, host operating systems, containerized workloads, or streaming pipelines, LogScale provides a range of flexible ingestion methods to suit your infrastructure and requirements.

LogScale uses a 'schema-on-read' approach where, rather than requiring data to conform to a fixed structure at ingest time, LogScale allows fields to be extracted and interpreted at query time. This means that even unstructured or raw data can be ingested immediately and explored without needing to define a schema upfront.

Data can be ingested into LogScale through four main methods:

  • Ingest Feeds - Cloud-native integrations that enable direct ingestion from cloud platform services such as AWS S3 and Azure Event Hub. Ingest feeds are configured within LogScale and handle authentication and data retrieval automatically

  • LogScale Collector - A lightweight agent installed on host machines that collects and forwards log data to LogScale. The Collector supports a wide range of host-based log sources including Windows Event Logs, syslog, and file-based logs, and can also be deployed as a Kubernetes DaemonSet for container environments

  • APIs - Programmatic ingestion endpoints that allow data to be sent directly to LogScale via HTTP. LogScale supports several API formats including its own native API, the ElasticSearch Bulk API, and the Splunk HTTP Event Collector (HEC) format, making it straightforward to integrate with existing pipelines or migrate from other platforms

  • Third-Party Tools - External data collection, forwarding, and pipeline tools that send data to LogScale. This includes tools such as Apache Kafka, Fluentd, Fluent Bit, and Vector, which are commonly used to collect, transform, and route data before ingestion into LogScale

Where a parser is required or recommended, LogScale's Marketplace provides a wide range of pre-built parsers and packages that can be applied at ingest time to extract structured fields and normalize data for consistent querying and analysis.

The following sections detail the range of data sources that LogScale supports, organized by category. Where available, links to parsers on the LogScale Marketplace are provided.

Cloud Platform Data

Source Description Methods Parser Required?
AWS S3 Bucket Data Ingests log files and event data stored in Amazon S3 buckets. Useful for batch or archived data ingestion, including stored log files from other AWS services Ingest feeds Optional
AWS CloudTrail Ingests AWS API call history and account activity logs, providing visibility into user actions and resource changes across AWS infrastructure Ingest feeds Yes
Azure Event Hub Data Ingests streaming event data from Microsoft Azure Event Hub, enabling real-time processing of large volumes of Azure-sourced telemetry and log data Ingest feeds Optional
Azure Activity Logs Ingests Azure subscription-level logs that track operations performed on resources, useful for auditing and monitoring Azure infrastructure changes Ingest feeds Yes
GCP Audit Logs Ingests Google Cloud Platform audit logs, capturing administrative activity and data access events across GCP services Ingest feeds Yes
Falcon Data Replicator (FDR) Ingests CrowdStrike Falcon platform event data delivered via S3, providing rich endpoint telemetry and security event data directly from the Falcon sensor Ingest feeds Yes

Host and OS Logs

Source Types of Data Methods Parser Required?
Windows Event Logs Collects Windows system, security, and application event logs directly from Windows hosts, providing visibility into OS-level activity and security events LogScale Collector Yes
Syslog Data Receives standard syslog messages from network devices, servers, and applications over UDP/TCP, one of the most widely used log formats across Linux/Unix systems and network devices LogScale Collector Optional
Journal Logs Collects structured log data from the systemd journal on Linux systems, capturing detailed system and service-level events LogScale Collector Optional
Linux Var Logs Ingests traditional Linux log files stored in /var/log, including system logs, authentication logs, and application-specific log files LogScale Collector Yes
Unified Logs (macOS) Collects log data from the Apple Unified Logging system on macOS, capturing system and application events from Apple devices LogScale Collector Yes
Container/Kubernetes Logs Ingests log data from containerized workloads and Kubernetes clusters, including pod logs, container stdout/stderr streams, and Kubernetes audit logs LogScaleCollector / Third -party tools Optional

File-Based Data

Source Types of Data Methods Parser Required?
Files Ingests data directly from log files and text files on disk, supporting a wide range of file formats and enabling collection from applications that write logs to local storage LogScale Collector Optional
Raw Ingests unprocessed raw data as-is, without any pre-processing or formatting requirements, providing maximum flexibility for custom data sources APIs No
Unstructured Data Ingests free-form text data that does not follow a predefined schema, with LogScale's parser framework able to extract structure and meaning at query time LogScale Collector / APIs No

Security Data

Source Types of Data Methods Parser Required
Security Data Ingests a broad range of security-focused event data including alerts, threat intelligence, and security tool outputs, enabling centralized security monitoring and analysis LogScale Collector / APIs / Third-Party Tools Optional
OCSF Ingests security event data conforming to the OCSF standard, an open-source schema framework that normalizes security data across different vendors and tools for consistent analysis APIs / Third-Party Tools Optional

API and Structured Formats

Source Types of Data Methods Parser Required?
ElasticSearch Bulk Ingest API Supports ingestion via the ElasticSearch Bulk API format, enabling easy migration from ElasticSearch-based pipelines and compatibility with tools that support ES output APIs Optional
HEC - HTTP Event Collector Ingests data via Splunk-compatible HTTP Event Collector endpoint, enabling straightforward migration from Splunk and compatibility with the wide range of tools that support HEC output APIs Optional
HEC Raw Ingests raw data via the HEC endpoint without requiring JSON formatting, providing a simpler alternative to structured HEC ingestion for plain text or unformatted data APIs Optional
JSON Raw Ingests raw JSON-formatted data directly, supporting both single JSON objects and JSON arrays without requiring a specific schema or structure APIs No
Structured Data Ingests data that conforms to a predefined schema or format, such as CSV or key-value pairs, enabling efficient parsing and field extraction at ingest time LogScale Collector / APIs Optional
OpenTelemetry Data Ingests telemetry data conforming to the OpenTelemetry standard, including traces, metrics, and logs from instrumented applications and infrastructure APIs / Third-party tools No
Metrics Data Ingests time-series metrics data from infrastructure, applications, and services, enabling performance monitoring and capacity planning alongside log data APIs / Third-party tools Optional

Streaming and Pipeline Data

Source Types of Data Methods Parser Required?
Apache Kafka Ingests high-volume streaming event data from Apache Kafka topics, enabling real-time processing of data from distributed systems and event-driven architectures Third-party tools Optional
Fluentd / Fluent Bit Ingests log data forwarded via Fluentd or its lightweight counterpart Fluent Bit, widely used open-source log collection and forwarding agents that support a large number of input and output plugins Third-party tools Optional
Vector Ingests data from Vector, a high-performance open-source observability data pipeline tool, supporting transformation and routing of logs, metrics, and traces before ingestion into LogScale Third-party tools Optional