Purpose: Provides automatic traffic failover between primary and secondary clusters using Azure Traffic Manager.

When users access your LogScale cluster, they use a single global DNS name (like logscale.example.com). This module creates an Azure Traffic Manager that continuously monitors both clusters' health. If the primary cluster becomes unhealthy, Traffic Manager automatically routes all traffic to the secondary cluster - no DNS changes needed, no manual intervention required.

Deployed on: Primary (active) cluster only when manage_traffic_manager = true

Key resources created:

Secondary Endpoint Registration:

The secondary (standby) cluster automatically registers itself with Traffic Manager using azapi_resource.traffic_manager_secondary_endpoint. This resource:

This approach eliminates the need for the primary cluster to know the secondary's IP address in advance, simplifying the deployment sequence.

Traffic Manager Resource Architecture

DNS Configuration Options:

The root DNS zone for your global LogScale hostname can be hosted anywhere - Azure DNS, AWS Route 53, Cloudflare, or any other DNS provider. Traffic Manager uses its own *.trafficmanager.net domain; you only need a CNAME record in your DNS provider pointing to the Traffic Manager FQDN.

When DNS is hosted outside of Azure (e.g., AWS Route 53):

traffic_manager_dns_zone_name           = "example.com"         # Still required - used for TM host header
traffic_manager_dns_zone_resource_group = ""                    # No Azure DNS zone
traffic_manager_create_dns_record       = false                 # Skip Azure DNS CNAME creation

Then create a CNAME record in your DNS provider:

traffic_manager_dns_zone_name is always required even when DNS is external - Traffic Manager uses it as the host header in health probes so the ingress controller can route the request correctly.

Traffic Manager Priority Routing:

Priority Routing Logic:

The following table shows priority routing logic:

Expected Profile Status:

When the secondary cluster is in standby mode (dr="standby"), the Traffic Manager profile will show a status of "Degraded", while it should be healthy.

Verification:

Despite the "Degraded" profile status, traffic routes correctly:

# Global DR FQDN should return HTTP 200
curl -sk https://<global_logscale_hostname>.<zone>/api/v1/status
# DNS resolves to primary IP
dig +short <global_logscale_hostname>.<zone>

For health check settings, expected status table, and failover timing details, see Traffic Manager Configuration.

Purpose: Automatically starts the LogScale application on the secondary cluster when the primary fails.

The secondary cluster runs in a minimal "standby" state to save costs - the Humio operator is scaled to zero, so no LogScale pods are running. When the primary cluster fails, this module's Azure Function automatically scales up the Humio operator, which then starts the LogScale pod to recover from the primary's data. This happens automatically, triggered by the same health check that Traffic Manager uses.

Deployed on: Secondary (standby) cluster only when dr = "standby" and dr_failover_function_enabled = true

Key resources created:

Metric Alert Configuration:

The metric alert monitors the health state of the primary Traffic Manager external endpoint:

Implementation note:

The module scopes the alert to the Traffic Manager profile and uses the EndpointName dimension (extracted from the primary endpoint resource ID) to target only the primary endpoint.

How it works:

The following diagram provides an overview of the process:

Failover chain timing:

Configuration options (in tfvars):

For timing and retry variables (pre_failover_failure_seconds, cooldown_seconds, max_retries), see Failover Timing Reference.

Function App SKU Selection:

Azure Function Apps support multiple pricing tiers. The choice depends on quota availability in your target region:

Checking Azure Quota:

If deployment fails with quota errors, check available quota:

# Check Dynamic VMs quota (for Y1 Consumption plan)
az vm list-usage --location <region> -o table | grep -i dynamic

# Check BS Series quota (for B1/B2/B3 Basic plan)
az vm list-usage --location <region> -o table | grep -i "BS Family"

Example tfvars for Basic SKU (quota workaround):

# Use Basic plan when Consumption (Y1) quota is unavailable
dr_failover_function_sku      = "B1"
dr_failover_function_location = "eastus2"

Cross-Region Deployment:

The Function App can be deployed to a different Azure region than the AKS cluster if quota constraints prevent deployment in the primary region. This is configured using the dr_failover_function_location variable:

# Deploy Function App to westus when eastus2 lacks Y1 quota
dr_failover_function_location = "westus"

Why this works:

The Azure Function communicates with AKS using Azure's control plane API (ARM), not the pod network. The Function App's Managed Identity is granted the "Azure Kubernetes Service Cluster Admin Role" on the AKS cluster, which is a subscription-scoped RBAC assignment that works regardless of the Function App's region.

AKS Authorized IP Ranges (Critical):

When AKS is configured with authorized IP ranges (ip_ranges_allowed_to_kubeapi), the Function App's outbound IPs must be included in the authorized list. Otherwise, the function will fail to connect to the Kubernetes API with connection timeout errors.

This is handled automatically by the azapi_update_resource.aks_authorized_ips_for_function resource in data-sources.tf, which:

Why this requires a separate resource:

A circular dependency exists because:

The azapi_update_resource.aks_authorized_ips_for_function resource breaks this cycle by updating AKS after both resources exist.

Security: The Function App is locked down so only Azure Monitor can invoke it - all inbound traffic is denied by default (ip_restriction_default_action = "Deny") except the ActionGroup service tag. HTTPS and TLS 1.2 are enforced, FTP is disabled, and the HTTP trigger requires a function-level key. SCM (deployment) access is similarly restricted. See modules/azure/dr-failover-function/main.tf for the full configuration.

The ingress_extra_hostnames variable automatically adds the global DR FQDN to the TLS certificate via cert-manager. Both the cluster-specific and global DR hostnames are included as SANs.

Verify with:

kubectl get certificate <fqdn> -n logging -o jsonpath='{.spec.dnsNames}'

For the full certificate flow, SANs table, and verification commands, see TLS Certificate Configuration.

This section covers how LogScale authenticates to Azure Blob Storage and how the storage firewall is configured so both clusters can access the primary's data during DR recovery.

Authentication

LogScale authenticates to Azure Blob Storage using a storage account access key (Azure Workload Identity is not supported by LogScale at the moment). Both the access key and a LogScale-level encryption key are stored in a Kubernetes secret and injected into pods as environment variables.

On the primary cluster, the encryption key is randomly generated at first deploy and the storage account key comes from the storage account itself. On the secondary cluster, Terraform copies both keys from the primary's Terraform state (via terraform_remote_state) so the secondary can authenticate to and decrypt the primary's blob data. If remote state is unavailable, the keys can be supplied manually via existing_storage_encryption_key and azure_recover_from_accountkey variables. Terraform validates at plan time that a standby deployment has both keys.

The pod receives five storage-related environment variables: AZURE_STORAGE_ACCOUNTNAME, AZURE_STORAGE_ACCOUNTKEY, AZURE_STORAGE_BUCKET, AZURE_STORAGE_ENDPOINT_BASE, and AZURE_STORAGE_ENCRYPTION_KEY. The account name, bucket, and endpoint are set directly; the two keys come from the Kubernetes secret. On the secondary, additional AZURE_RECOVER_FROM_* environment variables point LogScale to the primary's storage account for snapshot recovery.

Storage Firewall and Cross-Region Access

The primary storage account has a firewall that only allows traffic from specific IP addresses. Both clusters write firewall rules to this account - and because Azure replaces the entire ipRules array on every update, each side must merge existing rules with its own to avoid dropping the other's entries.

How it works:

The primary uses remote state instead of a live API read because reading its own storage firewall would create a Terraform dependency cycle. AKS pods egress through the load balancer IPs, not the NAT gateway.

Recovery-Time Data Flow

At recovery time, the secondary LogScale pod authenticates to the primary storage account using the storage account key and reads the global snapshot:

Operational notes:

See Terraform Configuration for backend, workspace, remote state configuration, and the full deployment commands.

For the complete data flow summary table and storage firewall verification commands, see Azure DR Technical Reference: Cross-Region Storage Access

Standby clusters exclude UI and Ingest node pools to save costs. These are created automatically during promotion (dr="standby" → dr="active"), taking 5-10 minutes per pool.

For the full node pool matrix, creation logic (Terraform count conditions), and rationale, see Node Pool Creation by DR Mode.

This section covers backend setup, workspace management, remote state data flow, and the deployment sequence for both primary and secondary clusters.

This implementation uses two separate Terraform workspaces (primary and secondary) within the same Azure Blob Storage backend. This is a simplified approach that allows the secondary cluster to read the primary cluster's outputs (such as the storage account encryption key, storage account name, and storage access credentials) directly via data.terraform_remote_state, without requiring manual key exchange or an external secrets manager. Both workspaces share the same backend storage account, so cross-workspace state access works out of the box.

The DR deployment uses separate Terraform state files for primary and secondary clusters. Both state files are stored in the same Azure Blob Storage backend but isolated by workspace name.