Getting Started with Storage, Archiving, and Data Lifecycle

Manage costs, compliance, and data protection with archiving and retention

Workflow 1

Configure Archive Storage

Set up S3, Azure, or GCS archiving for long-term data storage

Follow this guided journey to configure cloud storage archiving for your LogScale deployment. In this path, you will:

  • Understand archiving concepts and storage backends
  • Configure S3, Azure, or GCS archiving
  • Set up archiving file formats and layout
  • Monitor archiving operations
  • Troubleshoot archiving issues
Start Here Workflow 2

Optimize Storage and Costs

Configure retention policies and monitor storage usage

Learn how to manage data retention and optimize storage costs for your LogScale deployment. In this path, you will learn about:

  • Time-based and size-based retention policies
  • Storage monitoring and capacity planning
  • Performance tuning for long retention
  • Cost optimization strategies
Start Here Workflow 3

Backup and Restore Strategies

Implement data protection and disaster recovery procedures

Master backup and restore procedures to protect your LogScale data. Configure comprehensive data protection and test restore procedures. In this path, you will learn how to:

  • Configure archiving for data protection
  • Understand restore procedures from archive storage
  • Verify data integrity and immutability
Get Started

Workflow 1: Configure Archive Storage

Follow this guided journey to configure cloud storage archiving for long-term data storage. Each step builds on the previous one to establish a complete archiving solution.

1

Understand Archiving Concepts

Learn about archiving architecture, storage backends, and how archiving works in LogScale

Archiving Overview

conceptintermediate

Introduction to archiving concepts, benefits, and supported storage backends
2

Choose Your Storage Backend

Select and configure your cloud storage backend (S3, Azure, or GCS)

Amazon S3 Archiving

guideintermediate

Configure S3 archiving with AWS, including bucket setup, IAM roles, and credentials

Azure Blob Storage Archiving

guideintermediate

Configure Azure archiving with blob storage containers and authentication

Google Cloud Storage Archiving

guideintermediate

Configure GCS archiving with buckets and service account authentication
3

Configure File Format and Layout

Set up archiving file formats (NDJSON or Parquet) and storage path structure

S3 File Format

guideintermediate

Choose between NDJSON and Parquet formats for archived data

S3 Storage Layout

guideintermediate

Understand the directory structure and path organization for archived files

Azure File Format

guideintermediate

Configure file format for Azure blob storage archiving

GCS File Format

guideintermediate

Configure file format for Google Cloud Storage archiving
4

Enable and Configure Archiving

Enable archiving for repositories and configure archiving settings

Enable S3 Archiving

guideintermediate

Configure and enable S3 archiving using GraphQL API or UI

Enable Azure Archiving

guideintermediate

Configure and enable Azure archiving for your repositories

Enable GCS Archiving

guideintermediate

Configure and enable GCS archiving for your repositories
5

Monitor and Troubleshoot Archiving

Monitor archiving operations, track storage usage, and troubleshoot common issues

Monitor S3 Archiving

guideintermediate

Query metrics and logs to monitor S3 archiving operations

Monitor Azure Archiving

guideintermediate

Monitor Azure archiving status and performance

Monitor GCS Archiving

guideintermediate

Track GCS archiving operations and storage usage

Troubleshoot Archiving Issues

troubleshootingintermediate

Diagnose and resolve common archiving problems

Workflow 2: Optimize Storage and Costs

Configure retention policies and monitor storage to optimize costs while meeting compliance requirements. Learn how to balance storage costs with data availability needs.

1

Understand Data Retention

Learn about retention policies, compressed vs. uncompressed storage, and retention strategies

Data Retention Overview

conceptintermediate

Introduction to retention concepts, storage tiers, and retention configuration

Compressed Storage

guideintermediate

Understand compressed storage for cost-effective long-term retention

Uncompressed Storage

guideintermediate

Learn about uncompressed storage for fast query performance
2

Configure Retention Policies

Set time-based and size-based retention policies for repositories

Time-Based Retention

guideintermediate

Configure time-based retention policies using the GraphQL API or UI
3

Monitor Storage Usage and Optimize Storage Performance

Track storage consumption, identify high-volume repositories, and plan capacity; and tune storage performance for repositories with long retention periods

Performance Tuning for Long Retention

guideadvanced

Query storage metrics, monitor retention compliance, and configure segment and ingest settings for optimal performance
4

Implement Cost Optimization

Apply strategies to reduce storage costs while maintaining data availability

Combine Retention with Archiving

guideadvanced

Balance retention, compression, and archiving to reduce on-cluster storage costs

Workflow 3: Backup and Restore Strategies

Implement comprehensive data protection and disaster recovery procedures using archiving and restore capabilities.

1

Configure Archiving for Data Protection

Set up archiving as a backup strategy for long-term data protection

Archiving for Data Protection

guideintermediate

Understand how archiving provides data backup and immutability

Complete Archiving Setup

Follow the complete archiving configuration workflow
2

Understand Data Immutability

Learn how LogScale ensures data integrity and immutability for compliance

Data Immutability and Security

conceptintermediate

Overview of data immutability features for compliance and data protection
3

Restore Archived Data

Data reingestion from archiving proviers

Re-ingest S3 Archived Data

guideintermediate

Restore archived data from S3 storage for disaster recovery

Re-ingest GCS Archived Data

guideintermediate

Restore archived data from Google Cloud Storage for disaster recovery

Re-ingest Azure Archived Data

guideintermediate

Restore archived data from Azure storage for disaster recovery
4

Verify and Audit Data Protection

Regularly audit backup and restore procedures to ensure data protection compliance

Audit Data Access and Changes

guideintermediate

Monitor audit logs for data access and archiving operations

Compliance and Auditing

Follow the compliance workflow to meet regulatory requirements