Kubernetes Components

CRDs

The CRDS Terraform module installs the necessary Custom Resource Definitions (CRDs) for both Cert-Manager and LogScale withing the EKS cluster. It retrieves the CRD manifests from specified URLs, decodes them, and applies them to the Kubernetes cluster. Additionally, it includes a provisioner to wait for the CRDs to be applied before proceeding to ensure proper setup. The CDR deployment follows the same steps for both services.

  • Data Retrieval: The CRD YAML file is fetched from a specified URL.

  • YAML Decoding: The fetched YAML content is split into individual documents and decoded.

  • CRD Application: Each decoded CRD manifest is applied to the Kubernetes cluster using the kubernetes_manifest resource.

  • Waiting Period: A 30-second wait period ensures that all Custom Resource Definitions (CRDs) are fully applied and propagated within the Kubernetes cluster before any dependent operations proceed. This waiting period acts as a buffer to allow the cluster to stabilize after the CRD application, reducing the risk of race conditions or incomplete resource recognition. By enforcing this delay, the null_resource helps guarantee that subsequent Terraform operations can reliably interact with the newly installed CRDs.

Configuring LogScale

Cert-Manager

Cert-manager is a pre-requisite for LogScale, deployed as a Helm chart in its own namespace. It includes the Cert-Manager ClusterIssuer, a Kubernetes add-on that automates the management and issuance of TLS certificates from various issuing sources, such as Let's Encrypt.

This section deplys the following Kubernetes resources:

  1. CustomResourceDefinition Check:

    • Verifies if the ClusterIssuer CRD has been applied in the Kubernetes cluster to ensure cert-manager can create and manage issuers.

  2. Namespace Creation:

    • Creates a dedicated namespace for cert-manager to organize and isolate its resources.

  3. Cert-Manager Deployment:

    • Deploys the cert-manager Helm chart in the specified namespace, configuring it with default issuer settings using Helm values.

  4. ClusterIssuer Configuration:

    • Defines a ClusterIssuer resource using cert-manager to automate the issuance of TLS certificates. This resource is configured for ACME protocol with Let's Encrypt as the certificate authority, specifying the email, private key secret reference, and HTTP-01 challenge solver.

Configuring Data Ingress

The Ingress leverages the AWS Load Balancer Controller service, a Kubernetes controller that integrates with Amazon Elastic Kubernetes Service (EKS) to manage AWS Elastic Load Balancers (ELB) for Kubernetes applications. It supports both Application Load Balancers (ALB) and Network Load Balancers (NLB), automating the creation and management of load balancers for services running within a Kubernetes cluster. The controller watches Ingress resources and services within the Kubernetes cluster, provisioning and configuring the necessary ALB based on defined Ingress resources to route external traffic to Kubernetes services.

This section deploys the following resources:

AWS ELB Ingress Controller:

    1. Kubernetes Service Account: Creates a Kubernetes service account for the AWS Load Balancer Controller with the necessary IAM role annotation.

    2. Helm Deployment: Deploys the AWS Load Balancer Controller using Helm, configured with the cluster name and service account.

    3. ExternalDNS Helm Deployment: Deploys ExternalDNS using a Helm chart to manage DNS records for Ingress resources, with configurations for AWS, region, zone type, and service account annotations.

NodePort Services and Ingresses:

  1. LogScale Basic Cluster Type:

    1. NodePort Service: Creates a NodePort service for the LogScale basic cluster type.

    2. Ingress Resource: Configures an Ingress resource to route external traffic to the NodePort service using an internet-facing ALB, with annotations for HTTPS, backend protocol, health checks, and ExternalDNS settings, including:

      shell
      external-dns.alpha.kubernetes.io/hostname = ${var.hostname}.${var.zone_name}

      This annotation allows ExternalDNS to create a DNS record that points to the ALB associated with the Ingress.

  2. LogScale Ingress Cluster Type:

    1. NodePort Service: Creates a NodePort service for the LogScale ingress-only cluster.

    2. Ingress Resource: Configures an Ingress resource to route external traffic to the NodePort service using an internet-facing ALB, with annotations for HTTPS, backend protocol, health checks, and ExternalDNS settings, including:

      shell
      external-dns.alpha.kubernetes.io/hostname = ${var.hostname}.${var.zone_name}

      This annotation allows ExternalDNS to create a DNS record that points to the ALB associated with the Ingress.

  3. LogScale UI:

    1. NodePort Service: Creates a NodePort service for the LogScale UI-only cluster type.

    2. Ingress Resource: Configures an Ingress resource to route external traffic to the NodePort service using an internet-facing ALB, with annotations for HTTPS, backend protocol, health checks, and ExternalDNS settings, including:

      shell
      external-dns.alpha.kubernetes.io/hostname = ${var.hostname}.${var.zone_name}

      This annotation allows ExternalDNS to create a DNS record that points to the ALB associated with the Ingress.

  4. LogScale Ingest Cluster:

    1. NodePort Service: Creates a NodePort service for the LogScale internal-ingest cluster type.

    2. Ingress Resource: Configures an Ingress resource to route internal traffic to the NodePort service using an internal ALB, with annotations for HTTPS, backend protocol, health checks, and TLS settings.

A TLS certificate is configured for the Ingress using cert-manager, ensuring secure communication for the Ingress resources. This setup provides a comprehensive ingress solution for the LogScale application, leveraging AWS Load Balancer Controller and ExternalDNS for dynamic and scalable traffic management, including automatic DNS record creation in Route 53 with the specified ExternalDNS annotations to point to the ALBs associated with the Ingresses.

Deploying LogScale (logscale-1)

This section of the Terraform EKS module is responsible for deploying the LogScale application and the necessary Kubernetes resources. The configuration ensures the correct namespace, operators, secrets, and pods are set up to run the LogScale application efficiently. The following tasks are performed:

  1. Namespace Creation:

    Creates the logscale namespace where all related Kubernetes resources will be deployed.

  2. Humio Operator Installation:

    Installs the Humio operator in the logscale namespace using a Helm chart. The operator manages the deployment and lifecycle of the LogScale application.

  3. The Humio operator is configured with specific settings for liveness and readiness probes, and any additional values provided in var.humio_operator_extra_values.

  4. Secrets Management:

    1. LogScale License: Creates a Kubernetes secret containing the LogScale license key ${var.cluster_name}-license.

    2. S3 Encryption Key: Generates a random encryption key ${var.cluster_name}-s3-storage-encryption for securing data in the S3 bucket and stores it as a Kubernetes secret.

    3. Admin Password: Generates a random password ${var.cluster_name}-single-user-password for the admin user and stores it as a Kubernetes secret.

  5. LogScale Application Deployment:

    1. Deploys the LogScale application pods based on the specified cluster type (basic, ingress, internal-ingest), with configurations for node affinity, pod anti-affinity, persistent volume claims, and environment variables.

    2. Configures environment variables to integrate with external services such as S3, Kafka, and ZooKeeper, ensuring proper security and data encryption settings.

    3. Uses TopoLVM for managing the persistent volumes, enabling dynamic provisioning and volume expansion.

    4. Specifies node roles and additional configurations for each node pool type (e.g., ingest-only, ui-only).

  6. Storage Configuration:

    Creates persistent volume claims (PVCs) with the gp2 storage class, ensuring the dynamic provisioning of storage volumes managed by gp2.

  7. Cluster Types:

    Differentiates between various LogScale cluster types to ensure appropriate resource allocation and configurations for each type, including node pools and specific roles for nodes (e.g., ingestonly, uionly, or basic).

Persistent Volumes

The LogScale application uses persistent volumes managed by gp2 for dynamic volume provisioning.

The humio cluster resources assume that disks are prepped on the underlying Kubernetes worker nodes. We utilize RAID0 on the local SSDs provided by AWS's i4i instances, in combination with bucket storage. As long as Kafka is stable and bucket storage is functioning, using RAID0 on individual Kubernetes workers is sufficient. gp2 simplifies disk space management for Kubernetes pods by providing dynamic volume provisioning using Logical Volume Management (LVM).

  1. gp2 Storage Classes:

    gp2 Controller Installation: Deploys the gp2 controller via a Helm release, configured using a values file for specific settings. This controller manages LVM for Kubernetes, enabling dynamic provisioning of storage volumes.

  2. Persistent Volume Claims:

    Data Storage: The persistent volume claims for LogScale data storage use TopoLVM as the storage class. This ensures that data volumes are dynamically provisioned and managed by TopoLVM, supporting encryption and volume expansion.

This setup provides a robust and flexible persistent storage solution for the LogScale application, leveraging TopoLVM for advanced volume management and dynamic provisioning.

ExternalDNS

In order to synchronize exposed Kubernetes Services and Ingresses with Route53, ExternalDNS is used. ExternalDNS automatically creates CNAME records in Route53, pointing the Ingress Application Load Balancer (ALB) to the LogScale UI Fully Qualified Domain Name (FQDN). This solution also ensures the created Application Load Balancers (ALBs) are properly cleaned up upon calling terraform destroy.

The configuration includes annotations in the Kubernetes Ingress resources, such as:

shell
external-dns.alpha.kubernetes.io/hostname = ${var.hostname}.${var.zone_name}

Allowing ExternalDNS to create DNS records that point to the ALB associated with the Ingress. ExternalDNS is deployed and configured using a Helm chart with settings for the provider, AWS region, zone type, service account, and necessary IAM roles.

This setup provides an automated and efficient DNS management solution for Kubernetes Ingress resources, ensuring seamless integration with Route53 and proper resource cleanup.