Getting Data into LogScale

Learn ingestion concepts and how to configure sources, parse data, and send logs.

Path 1

GDI Fundamentals

Learn the basics of data ingestion

Learn GDI basics, data ingestion concepts, and data flows.

  • What is GDI?
  • What is a Repository?
  • Identify Data Sources
  • What Ingest Methods are available?
  • Data Parsing and Timestamps
  • Monitor Ingest
Start Here Path 2

Ingest and Parse

Set up ingest methods and parsing techniques

Configure repositories, select ingest methods, and build parsers.

  • Generate An Ingest Token and verify data retention policies
  • Choose an Ingest Method
  • Configure Data Ingest
  • Parse Data
Get Started Path 3

Cloud-Native Ingestion

AWS, Azure, and Google Cloud Platform

Ingest logs and metrics from cloud platforms with native integrations. Ideal for SecOps, DevOps, and SRE teams.

  • AWS CloudTrail, CloudWatch and S3 Integration
  • Azure Event Hubs and Blob Storage
  • Google Cloud Logging and Pub/Sub
  • Cloud-Native Parsing and Enrichment
Get Started Path 4

On-Premises and Traditional Systems

Syslog, Windows Events, and network devices

Collect logs from syslog sources, Windows servers, network devices, and databases. Ideal for ITOps teams.

  • Syslog Configuration for Network Devices
  • Windows Event Log Collection
  • Database and Application Server Logs
  • Legacy System Integration
Get Started Path 5

Container and Kubernetes Logs

Docker, Kubernetes, and container orchestration

Deploy LogScale Collector in container environments with Helm charts and DaemonSets. Ideal for DevOps and SRE teams.

  • Kubernetes DaemonSet Deployment
  • Docker Container Log Collection
  • Pod and Node Log Aggregation
  • Container Metadata Enrichment
Get Started Path 6

Security and SIEM Integration

EDR, firewalls, and threat intelligence

Integrate EDR platforms, SIEM connectors, firewalls, and threat intelligence feeds. Ideal for SecOps and security analysts.

  • CrowdStrike Falcon EDR Integration
  • SIEM Connector Setup (Splunk, QRadar, Sentinel)
  • Firewall and IDS/IPS Logs (Palo Alto, Cisco, Fortinet)
  • Threat Intelligence Feed Integration
Get Started

Path 1: GDI Fundamentals

Learn the fundamentals of data ingestion. These concepts make configuring ingest and parsing straightforward.

1

What is GDI?

Understand the fundamentals of getting data into LogScale

GDI Overview

guideintermediate

Introduction to getting data into LogScale

GDI Concepts

guideintermediate

Core concepts and terminology for data ingestion
2

What is a Repository?

Learn how repositories organize and store your data

Repository Overview

guideintermediate

Understand how repositories work in LogScale
3

Identify Data Sources

Understand what types of data you can send to LogScale

Data Source Types

guideintermediate

Logs, metrics, events, and supported data types

Assess Your Data

guideintermediate

Evaluate volume, format, and structure of your data

Supported Formats

guideintermediate

JSON, CSV, syslog, and other supported formats
4

What Ingest Methods are Available?

Compare different methods to send data to LogScale

Ingest Methods Overview

guideintermediate

Compare all available ingestion methods

Ingest API

guideintermediate

Send data directly through HTTP endpoints

Log Collectors

guideintermediate

Use agents to collect and forward logs

Pre-built Integrations

guideintermediate

Integrations for common platforms and services
5

Data Parsing and Timestamps

Learn how LogScale parses data and handles timestamps

Parser Overview

guideintermediate

How parsers extract fields from raw data

Timestamp Parsing

guideintermediate

How LogScale identifies and parses timestamps

Built-in Parsers

guideintermediate

Pre-configured parsers for common data formats

Custom Parsers

guideintermediate

Create Custom parsers
6

Monitor Ingestion and Data Impact

Track ingested data and understand how the ingest process impacts data

Usage Management

guideintermediate

Track and manage your ingestion volume

Optimize ingest

guideintermediate

Strategies to reduce ingestion volume and costs

How is Data Impacted

guideintermediate

How data is impacted by the ingest process

Path 2: Implement Ingestion and Parsing

Master GDI capabilities with this hands-on progression through repository setup, data ingestion, and parsing configuration.

1

Generate Ingest Token and Verify Data Retention

Create authentication tokens and confirm retention policies

Ingest Token Management

guideintermediate

Manage and secure ingest tokens

Verify Retention

guideintermediate

Confirm retention policies are properly configured
2

Choose an Ingest Method

Select and configure the best ingestion method for your data source

Ingest Methods Overview

guideintermediate

Complete reference of all ingestion methods

Ingest API

guideintermediate

HTTP endpoints for direct data ingestion

Log Collectors

guideintermediate

Agent-based log collection and forwarding

Integrations

guideintermediate

Pre-built integrations for common data sources
3

Configure Data Ingest

Set up and configure your data ingestion pipeline

Configure Ingestion

guideintermediate

Set up your data ingestion pipeline

Getting Started with Data ingest

guideintermediate

Understand the procedures involved in data ingest using different methods.
4

Parse Data

Create or Choose and apply parsers to extract fields from your data

Create or Choose Parsers

guideintermediate

Build custom parsers for your data format or use built in parsers

Assign Parsers

guideintermediate

Assign parsers to repositories and ingest tokens

Workflow 3: On-Premises and Traditional Systems

Collect logs from syslog sources, Windows servers, network devices, and databases.

1

Syslog Configuration

Configure syslog for network devices, firewalls, and legacy systems

Syslog

guideintermediate

Step-by-step guide for syslog ingestion (Setup effort: Low to Medium)

Cisco Network Devices

guideintermediate

Configure syslog on Cisco routers and switches

Palo Alto Firewalls

guideintermediate

Forward Palo Alto firewall logs via syslog

Fortinet FortiGate

guideintermediate

Configure FortiGate firewall syslog forwarding

Syslog Parser Configuration

guideintermediate

Parse standard and custom syslog formats
2

Windows Event Logs

Collect Windows Event Logs from servers and workstations

Windows Event Collector

guideintermediate

Step-by-step guide for Windows Event Collector (Setup effort: Medium)

WMI-Based Collection

guideintermediate

Use WMI for Windows Event Log collection

LogScale Collector for Windows

guideintermediate

Deploy collector agent on Windows systems

Windows Event Parsers

guideintermediate

Parse Windows Security, Application, and System logs
3

Database and Application Servers

Ingest logs from databases, web servers, and application platforms

Database Logs (JDBC)

guideintermediate

Step-by-step guide for database log collection via JDBC (Setup effort: High)

SNMP Traps

guideintermediate

Step-by-step guide for SNMP trap ingestion (Setup effort: Medium)

Apache Web Server Logs

guideintermediate

Configure Apache access and error log ingestion

NGINX Logs

guideintermediate

Ingest NGINX access and error logs

Microsoft IIS Logs

guideintermediate

Collect IIS web server logs
4

Legacy System Best Practices

Optimize traditional system log collection and troubleshooting

Syslog Best Practices

guideintermediate

Performance tuning and reliability tips

Configuring Syslog Source

guideadvanced

Debug common syslog connection and parsing problems

Create Custom Parsers

guideadvanced

Build parsers for proprietary log formats

Workflow 4: Container and Kubernetes Logs

Deploy LogScale Collector in container environments. Collect logs from Docker containers and Kubernetes pods.

1

Kubernetes DaemonSet Deployment

Deploy LogScale Collector to Kubernetes using Helm charts

Falcon LogScale Collector

guideintermediate

Step-by-step guide for deploying the LogScale Collector (Setup effort: Low to Medium)

Deploy via Helm Chart

guideintermediate

Install LogScale Collector using Helm

DaemonSet Configuration

guideintermediate

Configure DaemonSet for cluster-wide collection

RBAC Configuration

guideintermediate

Set up service accounts and permissions
2

Pod and Container Log Collection

Collect logs from pods, containers, and container runtime

Pod Log Collection

guideintermediate

Automatically collect logs from all pods

Pod Annotations for Parsing

guideintermediate

Use annotations to assign parsers

Multi-line Log Handling

guideintermediate

Configure multi-line log parsing for stack traces

JSON Log Parsing

guideintermediate

Parse structured JSON logs from containers
3

Docker Container Logs

Collect logs from Docker containers on standalone hosts

Docker Collector

guideintermediate

Step-by-step guide for Docker log collection (Setup effort: Medium)

Kafka

guideintermediate

Step-by-step guide for Kafka ingestion (Setup effort: High)

Docker Compose Configuration

guideintermediate

Add LogScale Collector to Docker Compose stacks

Docker Swarm Deployment

guideintermediate

Deploy collector in Docker Swarm mode
4

Metadata Enrichment and Best Practices

Enrich logs with container and Kubernetes metadata

Kubernetes Metadata Enrichment

guideintermediate

Add pod, namespace, and label metadata to logs

Log Filtering and Exclusions

guideintermediate

Filter noisy logs and exclude unwanted sources

Kubernetes Best Practices

guideintermediate

Performance tuning and resource optimization

Workflow 5: Security and SIEM Integration

Integrate EDR platforms, SIEM connectors, firewalls, and threat intelligence feeds for security monitoring.

1

CrowdStrike Falcon EDR Integration

Ingest detections, events, and telemetry from Falcon platform

Crowdstream

guideintermediate

Step-by-step guide for Crowdstream ingestion (Setup effort: Medium to High)

Falcon Detections

guideintermediate

Ingest Falcon intel indicators and detection streams

Falcon Spotlight Vulnerabilities

guideintermediate

Import vulnerability assessment data

Falcon Device Package

guideintermediate

Pre-built dashboards and queries for Falcon devices
2

SIEM Connector Setup

Configure bidirectional integration with existing SIEM platforms

Splunk Integration

guideintermediate

Bidirectional data exchange with Splunk

IBM QRadar Integration

guideintermediate

Forward logs to and from QRadar

Microsoft Sentinel

guideintermediate

Integrate with Azure Sentinel SIEM

Generic SIEM Connector

guideintermediate

Configure custom SIEM integrations
3

Firewall and Network Security Logs

Ingest logs from firewalls, IDS/IPS, and network security devices

Palo Alto Networks

guideintermediate

Ingest Palo Alto firewall logs

Cisco ASA Firewalls

guideintermediate

Configure Cisco ASA log forwarding

Fortinet FortiGate

guideintermediate

Ingest FortiGate security logs

Check Point Firewalls

guideintermediate

Forward Check Point logs to LogScale

CEF and LEEF Parsing

referenceintermediate

Parse Common Event Format and LEEF security logs
4

Threat Intelligence and Security Packages

Integrate threat feeds and install pre-built security packages

Threat Intelligence Feeds

guideintermediate

Ingest IOCs from threat intelligence platforms

Firewall Adversaries Packages

guideintermediate

Browse pre-built security tool packages

Security Ingestion Best Practices

guideintermediate

Optimize security log collection and retention

Workflow 6: Custom and Application Integration

Build custom integrations with the Ingest API. Configure log shippers, develop parsers, and instrument applications.

1

Ingest API Setup

Use the REST API to send data directly to LogScale

HTTP / HTTPS API

apiintermediate

Step-by-step guide for HTTP/HTTPS API ingestion (Setup effort: Medium)

API Authentication

apiintermediate

Authenticate with ingest tokens

Send Structured Data

apiintermediate

Ingest JSON, NDJSON, and structured logs

Send Unstructured Data

apiintermediate

Ingest raw text logs via API

Bulk Ingestion

apiintermediate

Optimize throughput with batch ingestion
2

Log Shipper Configuration

Configure Filebeat, Fluentd, Logstash, and other forwarders

Filebeat

guideintermediate

Step-by-step guide for Filebeat configuration (Setup effort: Medium)

Fluentd

guideintermediate

Step-by-step guide for Fluentd integration (Setup effort: High)

Logstash

guideintermediate

Step-by-step guide for Logstash pipeline configuration (Setup effort: High)

LogScale Collector

guideintermediate

Native lightweight collector agent

Rsyslog Forwarding

guideintermediate

Configure rsyslog to forward to LogScale
3

Custom Parser Development

Build and test parsers for proprietary or unique log formats

Create Custom Parsers

guideintermediate

Build parsers using LogScale parsing language

Regular Expression Parsing

referenceintermediate

Extract fields using regex patterns

JSON Path Extraction

guideintermediate

Parse JSON logs with JSON path expressions

Parsers Validation

guideintermediate

Validate parser logic with sample data

Managing Parsers

guideintermediate

Performance optimization and maintainability
4

Application Log Instrumentation

Instrument applications to send logs directly to LogScale

SDK and Client Libraries

apiadvanced

Use official SDKs for popular languages

API Logging Best Practices

apiintermediate

Structure logs for optimal analysis

Structured JSON Logging

apiintermediate

Emit structured JSON logs from applications