oci iam user get --user-id $OCI_USER_OCID
oci os ns get
terraform version

Storage + remote state:

Strimzi generates the Kafka TLS truststore secret (${name_prefix}-strimzi-kafka-cluster-ca-cert) only after Kafka is up.

Humio pods mount this secret and use its ca.password for KAFKA_COMMON_SSL_TRUSTSTORE_PASSWORD and /tmp/kafka/ca.p12 for KAFKA_COMMON_SSL_TRUSTSTORE_LOCATION.

If Humio starts before the secret exists, the pod fails to mount the volume and crashloops.

Deploy Kafka/Strimzi first, then let Humio start.

OCI LogScale deployments use AWS S3-compatible environment variables for DR recovery (since OCI Object Storage provides S3-compatible API). These variables are automatically set by Terraform when dr = "standby".

Environment Variable Reference

The primary cluster is provisioned as usual. The dr="active" variable is required.

Template primary-us-chicago-1.tfvars.example (copy to primary-us-chicago-1.tfvars locally)

workspace_name         = "primary"
dr                     = "active"
region                 = "us-chicago-1"
cluster_name           = "dr-primary"
logscale_public_fqdn   = "logscale-dr.example.com"

# Global DNS is managed from the active workspace only
manage_global_dns      = true
create_global_dns_zone = true
dns_zone_name          = "example.com"

# Security allowlists (examples) - replace with your real office/VPN IPs.
# Do NOT use 0.0.0.0/0.
bastion_client_allow_list = [
  "YOUR.PUBLIC.IP/32",
]

public_lb_cidrs = [
  "YOUR.PUBLIC.IP/32",
]

Commands:

terraform workspace select primary
terraform apply -var-file=primary-us-chicago-1.tfvars

Verify:

oci ce cluster get --cluster-id <cluster-ocid> --region us-chicago-1
terraform output
# Key outputs include: storage_bucket_name and storage_encryption_key_value (sensitive)

The tfvars file for the secondary OKE cluster configures the HumioCluster environment variables S3_RECOVER_FROM_* to point to the primary bucket.

Standby Cluster Initial State:

When dr = "standby", the secondary cluster is deployed with digest/ingress/Kafka capacity, but Humio stays offline until the operator is scaled up:

For the HumioCluster "License Error": See Issue 2: HumioCluster Shows "License Error" on Standby Cluster for details on the expected "license error" status on standby clusters.

Template secondary-us-chicago-1.tfvars.example (copy to secondary-us-chicago-1.tfvars locally)

workspace_name = "secondary"
dr = "standby"
region = "us-chicago-1"
cluster_name = "dr-secondary"
# Standby does not manage global DNS objects
manage_global_dns = false
create_global_dns_zone = false
primary_remote_state_config = {
backend = "oci"
workspace = "primary"
config = {
bucket = "your-terraform-state-bucket"
namespace = "your-namespace"
region = "us-chicago-1"
key = "env:/logscale-oci-oke"
auth = "ApiKey"
config_file_profile = "DEFAULT"
}
}
# DR recovery (S3-compatible env vars for OCI Object Storage)
# IMPORTANT: S3_RECOVER_FROM_* refer to the PRIMARY (source) cluster
s3_recover_from_region = "us-chicago-1"
s3_recover_from_replace_region = "us-chicago-1/us-chicago-1"
# s3_recover_from_bucket / s3_recover_from_endpoint_base / s3_recover_from_replace_bucket
# are inferred from primary remote state; override only for bootstrap/debug.
# Security allowlists (examples) - replace with your real office/VPN IPs.
# Do NOT use 0.0.0.0/0.
bastion_client_allow_list = [
"YOUR.PUBLIC.IP/32",
]
public_lb_cidrs = [
"YOUR.PUBLIC.IP/32",
]

Commands

terraform workspace select secondary
terraform apply -var-file=secondary-us-chicago-1.tfvars

Verify

# Encryption keys match (compare hashes)
kubectl get secret -n logging dr-primary-oci-storage-encryption --context oci-primary -o jsonpath='{.data.oci-storage-encryption-key}' | base64 -d | shasum -a 256
kubectl get secret -n logging dr-secondary-oci-storage-encryption --context oci-secondary -o jsonpath='{.data.oci-storage-encryption-key}' | base64 -d | shasum -a 256
# Pods minimal on secondary
kubectl get pods -n logging --context oci-secondary

Why targetReplicationFactor is 1 on standby:

The targetReplicationFactor represents the desired number of replicas for data segments when the cluster is operational.

Setting it to 1 on standby indicates that when the single node starts during failover, data should be replicated once (within that node).

The Humio operator allows this standby configuration.

The following table illustrates the automated DR failover sequence triggered by standby automation (Monitoring Alarm → ONS → Function) and enforced via OCI DNS steering policy.

Failover Sequence

On standby, the HumioCluster already declares nodeCount=1, but the Humio operator is scaled to 0. When the Humio operator is scaled to 1 (by the OCI Function on health check failure or manually), it reconciles the HumioCluster and starts a single LogScale pod.

kubectl --context oci-secondary -n logging scale deploy humio-operator --replicas=1

.

What Data is Transferred in the Global Snapshot

The global snapshot is a JSON-based export of LogScale's internal cluster state at boot time. Understanding what transfers (and what doesn't) is critical for DR planning:

Key insight: The global snapshot is LogScale's configuration and metadata state (~MBs), not your log data (~TBs). During DR, the secondary cluster reads the actual log events directly from the primary's Object Storage bucket using the segment metadata as a map.

Bucket storage configs are patched to point to primary storage as read-only

Kubernetes Resources Requiring Manual Sync (from humio-operator analysis):

Before executing a DR failover, ensure these resources exist in the secondary cluster:

Spot-check pods on secondary:

kubectl --context oci-secondary -n logging get pods
# Expect humio-operator (1/1), one Humio pod once recovery starts, and Kafka components running

Log in to the secondary LogScale cluster UI, open the Humio repository, and run the following query:

DataSnapshotLoader
| #kind != threaddumps

You should see messages similar to:

Checking bucket storage localAndHttpWereEmpty=true
Trying to fetch a global snapshot from bucket storage s3 if one exists in bucket=dr-secondary-logscale-data
Fetching global snapshot from bucket storage s3 found no snapshot to fetch.
Trying to fetch a global snapshot as recovery source from bucket storage in s3
Trying to fetch a global snapshot from bucket storage s3 if one exists in bucket=dr-primary-logscale-data
Fetched global snapshot from bucket storage s3 found snapshot with epochOffset={epoch=0 offset=699094}
Fetched a global snapshot as recovery source from bucket storage in s3 and got snapshot with epochOffset={epoch=0 offset=699094} now patching...
Snapshots to choose from, last is better...: List(({epoch=0 offset=699094},s3)) using kafkaMinOffsetOpt of Some(KafkaMinOffset(...))
Selecting snapshot from source=s3 with epochOffset={epoch=0 offset=699094}
updateSnapshotForDisasterRecovery: Patching region using from=us-chicago-1 to=us-chicago-1 on bucketId=1
updateSnapshotForDisasterRecovery: Patching bucket using from=dr-primary-logscale-data to=dr-secondary-logscale-data on bucketId=1
updateSnapshotForDisasterRecovery: Patching access configs from RECOVER_FROM on bucketId=1
updateSnapshotForDisasterRecovery: setting readOnly=true on bucketId=1 keyPrefix= new value for bucket=dr-secondary-logscale-data

Once the LogScale pod is running and has successfully read the global snapshot from the primary bucket, the cluster can be promoted to active status.

For zero-downtime DR promotion, use the two-phase terraform apply approach with the dr_use_dedicated_routing variable. This ensures traffic continues to flow to the existing digest pod while UI/Ingest pods scale up.

Understanding dr_use_dedicated_routing

In plain terms, this variable controls how Kubernetes services find LogScale pods

Why this matters during promotion

In standby mode, the DR cluster runs a single "digest" pod that can handle all request types (UI, ingest, queries). When you promote to active, the cluster needs to scale up specialized UI and ingest pods, which takes 1-2 minutes.

If you promote with dr_use_dedicated_routing = true (the default), the services immediately start looking for UI pods that don't exist yet. Result: 503 errors until the new pods are ready.

If you promote with dr_use_dedicated_routing = false first, the services continue routing to the existing digest pod while the new pods scale up.

Result: zero downtime.

Once all pods are ready, you apply again with dr_use_dedicated_routing = true to enable optimal routing.

dr_use_dedicated_routing Behavior Matrix:

Why two phases are needed:

When promoting from dr="standby" to dr="active", the HumioCluster's node pool configuration changes from digest-only (1 pod) to the full production topology (digest + UI + ingest pods). Without the two-phase approach:

With the two phase approach:

Phase 1: Promote with Generic Selectors (Zero-Downtime)

# Edit tfvars for Phase 1
vi secondary-us-chicago-1.tfvars
dr = "active"
dr_use_dedicated_routing = false # Generic selector - matches ALL pods
# Apply Phase 1
terraform workspace select secondary
terraform apply -var-file=secondary-us-chicago-1.tfvars
# Verify UI and Ingest pods are coming up
kubectl --context oci-secondary -n logging get pods -l humio.com/node-pool
# Wait until UI and Ingest pods show Running and Ready

Phase 2: Enable Dedicated Routing (After Pods Ready)

# Edit tfvars for Phase 2 - Choose one of two options:
# Option A: Stay in DR mode with optimal routing
vi secondary-us-chicago-1.tfvars
dr = "active"
dr_use_dedicated_routing = true # Pool-specific selectors - optimal routing
# Option B: Exit DR mode entirely (also enables optimal routing)
vi secondary-us-chicago-1.tfvars
dr = "" # Non-DR mode - also uses pool-specific routing automatically
# dr_use_dedicated_routing is ignored when dr="" (always uses pool-specific)
# Apply Phase 2
terraform workspace select secondary
terraform apply -var-file=secondary-us-chicago-1.tfvars
# Verify services have correct endpoints
kubectl --context oci-secondary -n logging get endpoints

If downtime during promotion is acceptable, you can use a single apply:

Actions

# Edit tfvars, switch to active
vi secondary-us-chicago-1.tfvars
dr = "active" # or dr = "" for non-DR mode (both work for promotion)
# dr_use_dedicated_routing defaults to true (pool-specific routing)
# Apply in secondary workspace
terraform workspace select secondary
terraform apply -var-file=secondary-us-chicago-1.tfvars

Why env vars are preserved

The humio-operator calculates a hash of the pod spec (including environment variables) to determine if pods need to be recreated. If env vars were removed during promotion:

Why this is safe:

The S3_RECOVER_FROM_* env vars are only read at startup by DataSnapshotLoader.scala:

Behavior matrix:

Resources destroyed during promotion:

When promoting from dr="standby" to dr="active", Terraform destroys the entire module.dr-failover-function because automated failover is no longer needed on an active cluster. The key resources removed are:

In total, approximately 20 OCI and Kubernetes resources are removed. The DR-related Terraform outputs (dr_failover_alarm_id, dr_failover_function_*, dr_failover_topic_id) are also no longer available after promotion.

Verify promotion:

kubectl get humiocluster -n logging --context oci-secondary -o jsonpath='{.spec.environmentVariables}' | jq '.[] | select(.name | startswith("S3_RECOVER"))'
# => empty
kubectl get humiocluster -n logging --context oci-secondary -o jsonpath='{.spec.nodeCount}'
# => production value
kubectl get pods -n logging --context oci-secondary
# => all pods running

When you use the DR global DNS pattern (${global_logscale_hostname}.${dns_zone_name}) with OCI DNS Steering Policy failover records, ingestion and UI clients point at a single global FQDN.

In normal operation this record resolves to the primary load balancer IP and the secondary HumioCluster declares nodeCount=1 but runs no Humio pods because the operator is scaled to 0.

If the primary health check fails and OCI DNS Steering Policy updates the global DNS to return the secondary IP, the OCI Function failover scaler scales the Humio operator from 0 → 1 so the secondary can start the single digest pod and serve traffic. There is no automatic scale-down of the operator or Humio; scale back manually or by re-applying Terraform with dr="standby" after failback.

In this mode, failover/failback tests for ingestion use the same global FQDN and do not require manual DNS record changes; the DNS steering policy and OCI Function together handle the traffic switch.

To verify which cluster is currently serving traffic:

GLOBAL_DR_FQDN="logscale-dr.oci-dr.humio.net" # Your global DR FQDN
dig +short "${GLOBAL_DR_FQDN}"
curl -I "https://${GLOBAL_DR_FQDN}"