Manage Data Ingest

This section covers various tools and configurations that can be used manage data ingest and ingested data. The following topics are included:

  • Backfilling Data โ€” Guidelines for ingesting historical or non-current data while maintaining optimal query performance. See Backfilling Data.

    You can use different methods to ingest data depending on your requirements; OS, Log format and so on:

    • Ingesting FDR Data

      Falcon LogScale can ingest Falcon Data Replicator (FDR) data into LogScale without having to configure log shippers. See Ingest FDR Data.

    • Ingesting AWS S3 Bucket Data

      Falcon LogScale can ingest and parse AWS S3 bucket data. See Ingest Data from AWS S3.

      S3 Ingest was introduced in v1.120.0

    • Ingesting Azure Feeds

      Falcon LogScale can ingest and parse Azure data. See Ingest Data from Azure Event Hubs.

    • Log Shippers

      Falcon LogScale is able to ingest data from a wide range of log shippers. Log shippers use the Ingest API to send one or more logs to LogScale. A log shipper can handle multiple logs, multiple log types, manage the log storage on disk, and pre-process the logs before sending them to LogScale. Log shippers are covered in more details in Log Shippers.

    • Listeners

      Ingest listeners are a great way of shipping data to LogScale through raw sockets, using either UDP or TCP. See Ingest Listeners.

    • LogScale Ingest Tokens

      A repository can have one or more ingest tokens associated with it. Ingest tokens are used with the Ingest API to enable data to be routed to the right repository, and to associate a parser. See Ingest Tokens.

    • LogScale API

      The Ingest API can be used directly or through one of LogScale's APIs or software libraries. See the Ingest API reference page for more information.

  • Disabling Ingestion โ€” How to temporarily pause data ingestion to a repository for troubleshooting or maintenance. See Disabling Ingestion.

  • Event Forwarding โ€” Forward ingested events to external systems like Kafka for additional processing while maintaining centralized logging. See Event Forwarding.

  • Ingesting FDR Data โ€” Falcon LogScale can ingest Falcon Data Replicator (FDR) data without configuring log shippers. See Ingest FDR Data.

  • Ingest Listeners โ€” Ship data to LogScale through raw TCP or UDP sockets for formats like rsyslog, StatsD, and GELF. See Ingest Listeners.

  • Ingest Tokens โ€” Tokens provide authentication for data ingestion and route data to the appropriate repository with the correct parser. See Ingest Tokens.

  • Ingest Feeds โ€” Configure and manage data ingest feeds for your repositories. See Ingest Feeds.

For information about ingest methods and quickly getting data into LogScale, see Use Case: Ingesting Application Logs.

LogScale is optimized for live streaming of events in real time. If you ship data that are not live, you must observe some basic rules so that the resulting events are stored in LogScale as efficiently as if they had been received live. See Backfilling Data.