Manage Data Ingest

This section covers various tools and configurations that can be used manage data ingest and ingested data. The following topics are included:

  • Backfilling Data โ€” Guidelines for ingesting historical or non-current data while maintaining optimal query performance. See Backfilling Data.

    You can use different methods to ingest data depending on your requirements; OS, Log format and so on:

  • Disabling Ingestion โ€” How to temporarily pause data ingestion to a repository for troubleshooting or maintenance. See Disabling Ingestion.

  • Event Forwarding โ€” Forward ingested events to external systems like Kafka for additional processing while maintaining centralized logging. See Event Forwarding.

  • Ingesting FDR Data โ€” Falcon LogScale can ingest Falcon Data Replicator (FDR) data without configuring log shippers. See Ingest FDR Data.

  • Ingest Listeners โ€” Ship data to LogScale through raw TCP or UDP sockets for formats like rsyslog, StatsD, and GELF. See Ingest Listeners.

  • Ingest Tokens โ€” Tokens provide authentication for data ingestion and route data to the appropriate repository with the correct parser. See Ingest Tokens.

  • Ingest Feeds โ€” Configure and manage data ingest feeds for your repositories. See Ingest Feeds.

For information about ingest methods and quickly getting data into LogScale, see Use Case: Ingesting Application Logs.

LogScale is optimized for live streaming of events in real time. If you ship data that are not live, you must observe some basic rules so that the resulting events are stored in LogScale as efficiently as if they had been received live. See Backfilling Data.