Each DR cluster (primary and secondary) has its own VCN with the following architecture:

Public Route Table (${cluster_name}-public-rt):

Private Route Table (${cluster_name}-private-rt):

Bastion Enhanced Route Table (${cluster_name}-bastion-enhanced-rt):

Four NSGs control traffic flow between the internet, load balancer, worker nodes, and Kubernetes API.

The most common networking issue is TLS timeouts when accessing the load balancer from the internet. This happens when the Load Balancer NSG has an egress rule to send traffic to worker nodes, but the Worker NSG is missing the corresponding ingress rule to accept that traffic.

Both rules are required because OCI NSG rules are unidirectional.

Quick fix for TLS timeout: Ensure the Worker NSG has an ingress rule from the LB NSG on ports 30000-32767 (see worker_ingress_from_lb_nsg rule in section 2 below).

Ingress Rules:

Egress Rules:

Ingress Rules:

Egress Rules:

OCI NSG rules are unidirectional - having an egress rule on NSG-A to NSG-B does not automatically allow the traffic. NSG-B must have a corresponding ingress rule from NSG-A

Required Rule Pair for LB→Worker Traffic:

Without worker_ingress_from_lb_nsg: TLS handshake times-out because the LB can send traffic, but the worker NSG blocks it at the receiver.

Terraform code (modules/oci/core/main.tf):

resource "oci_core_network_security_group_security_rule" "worker_ingress_from_lb_nsg" {
  network_security_group_id = oci_core_network_security_group.worker.id
  direction                 = "INGRESS"
  protocol                  = "6" # TCP
  source                    = oci_core_network_security_group.public_lb.id
  source_type               = "NETWORK_SECURITY_GROUP"
  description               = "Allow NodePort traffic from public LB NSG"
  tcp_options {
    destination_port_range {
      min = 30000
      max = 32767
    }
  }
}

Ingress Rules:

Egress Rules:

Ingress Rules:

Egress Rules:

The public_lb_cidrs variable controls which IP ranges can access the load balancer over HTTPS (port 443). This is a security measure to restrict access to known IP ranges (e.g., corporate networks, VPNs)

# Example: Restrict LB access to specific IP ranges
public_lb_cidrs = [
  "10.110.192.0/22",    # Internal network
  "130.41.55.146/32",   # Specific external IP
  "208.127.86.195/32",  # Another external IP
]

Impact on certificate validation: - When public_lb_cidrs restricts access, Let's Encrypt HTTP-01 validation fails because Let's Encrypt cannot reach port 80 - Solution: Use DNS-01 validation instead (see DNS-01 Certificate Issuance in the DR Operations Guide).

The bastion_client_allow_list variable controls which IP ranges can connect to the OCI Bastion Service for cluster access. This is separate from the load balancer security list.

Bastion is enabled by default (provision_bastion = true) for maximum security. To disable bastion and use public endpoint access instead, set provision_bastion = false and endpoint_public_access = true.

Required when provision_bastion=true (default). The list cannot include 0.0.0.0/0.

# Example: Restrict bastion access to specific IPs
bastion_client_allow_list = [
  "203.176.185.196/32",  # Your IP
  "147.161.213.22/32",   # VPN endpoint
]

The control_plane_allowed_cidrs variable controls which IP ranges can access the Kubernetes API (port 6443) via the API endpoint NSG.

Required when endpoint_public_access=true. The list cannot include 0.0.0.0/0.

# Example: Restrict K8s API access to specific IPs
control_plane_allowed_cidrs = [
  "10.0.0.0/16",         # VCN CIDR (always included)
  "203.176.185.196/32",  # Your IP for direct API access
]

OKE clusters support two access modes for the Kubernetes API endpoint:

Bastion Tunnel Mode (Recommended for Production):

Public Endpoint Mode (Development/Testing):

Security architecture:

By default, the OCI Cloud Controller Manager (CCM) adds all cluster nodes as backends to a Service of type LoadBalancer. This causes health check and traffic failures on nodes that don't run the target pods (for example, nodes without ingress-nginx).

To restrict backends to only the nodes running ingress-nginx pods, this implementation uses the OCI Classic Flexible Load Balancer with node label selector and externalTrafficPolicy: Local:

# Applied to ingress-nginx Service via nginx_ingress_sets in main.tf
annotations:
  oci.oraclecloud.com/load-balancer-type: lb
  service.beta.kubernetes.io/oci-load-balancer-subnet1: <lb_subnet_id>
  service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: "All"
  oci.oraclecloud.com/oci-network-security-groups: <lb_nsg_id>
  service.beta.kubernetes.io/oci-load-balancer-shape: flexible
  service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "10"
  service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
  service.beta.kubernetes.io/oci-load-balancer-backend-protocol: TCP
  oci.oraclecloud.com/node-label-selector: k8s-app=logscale-ingress
spec:
  externalTrafficPolicy: Local

Key configuration options:

Configuration location: main.tf (root module)

nginx_ingress_sets = [
  {
    name  = "controller.service.annotations.oci\\.oraclecloud\\.com/load-balancer-type"
    value = "lb"
  },
  {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/oci-load-balancer-subnet1"
    value = module.core.lb_subnet_id
  },
  {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/oci-load-balancer-security-list-management-mode"
    value = "All"
  },
  {
    name  = "controller.service.annotations.oci\\.oraclecloud\\.com/oci-network-security-groups"
    value = module.core.lb_nsg_id
  },
  {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/oci-load-balancer-shape"
    value = "flexible"
  },
  {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/oci-load-balancer-backend-protocol"
    value = "TCP"
  },
  {
    name  = "controller.service.annotations.oci\\.oraclecloud\\.com/node-label-selector"
    value = "k8s-app=logscale-ingress"
  },
  {
    name  = "controller.service.externalTrafficPolicy"
    value = "Local"
  }
]

Troubleshooting load balancer health checks:

Verify that the annotation is applied:

kubectl --context oci-primary get svc dr-primary-nginx-ingress-nginx-controller \
  -n logging-ingress -o jsonpath='{.metadata.annotations}' | jq .

The following diagram illustrates the complete network traffic flow from an external client to LogScale pods, showing each network boundary crossing and the security controls at each layer.

Traffic Flow Details:

Key Security Boundaries:

When is-preserve-source: true is set on the OCI Classic Load Balancer, the client's original IP address is preserved through the entire path. This requires special security rules.

Source IP Comparison:

Key Points: is-preserve-source=TRUE

When is-preserve-source=TRUE is enabled on the OCI LB:

The diagram below illustrates the network path changes during a DR failover event. When the primary cluster becomes unavailable, the OCI DNS steering policy redirects traffic to the secondary cluster. For the full failover procedure and automation details, see the OCI Disaster Recovery (DR) Operations Guide.

The diagram below shows how kubectl commands reach the private Kubernetes API endpoint through the OCI Bastion Service SSH tunnel. This is the recommended access pattern for production clusters where provision_bastion=true (default).

For DR failover to work correctly, ensure: