Amazon Web Services

Setting up Amazon Web Services

VPC (Virtual Private Network) Module

The VPC module creates a VPC with private, public, and intra subnets across three availability zones. It sets up a NAT gateway for internet access for instances in private subnets and enables DNS hostnames. Additionally, it configures security groups to allow internal traffic within the VPC and creates a specific security group for MSK (Managed Streaming for Apache Kafka) with appropriate ingress rules.

Key Components:

  • Availability Zones: Defines three availability zones for redundancy and high availability.

  • Subnets: Creates private, public, and intra subnets within the specified VPC CIDR block.

  • NAT Gateway: Ensures that instances in private subnets can access the internet.

  • DNS Hostnames: Enables DNS hostnames for instances within the VPC.

  • Security Groups: Configures security rules to allow internal traffic within the VPC. I

    • It also sets the security group msk_sg with specific ingress rules to allow unencrypted TCP and encrypted TCP traffic on the ports used by the Kafka brokers.

MSK (Amazon Managed Streaming for Kafka) Module

The MSK module sets up a Kafka cluster within AWS, providing a managed and scalable streaming platform for real-time data processing for logs ingested by LogScale. It specifies the Kafka version, number of broker nodes, instance types for the brokers, along with other features like using a KMS key for encryption of data at rest, as well as encryption in transit using TLS.

Key Components:

  • KMS Key: Creates a KMS key for encryption of data at rest.

  • MSK Cluster: Configures the Kafka cluster with specified settings, including the number of broker nodes, instance types, and security configurations.

  • S3 bucket: Stores MSK logs

EKS (Elastic Kubernetes Service) Module

The EKS module provisions a Kubernetes cluster within AWS.

Key Components:

  • ACM Certificate

    • Manages SSL/TLS certificates to enable secure and encrypted communication between the LogScale service exposed by ingress and the client sending logs. This includes creating and validating an ACM certificate using DNS validation, with the certificate lifecycle managed to ensure it is always available.

  • IAM Roles

    Configures and manages the following IAM roles and policies for EKS and associated services:

    1. logscale-${cluster_name}:

      • Assumed by the main service account ${var.cluster_name}-humio, allowing it to store and retrieve segment files from the S3 bucket.

      • The IAM role logscale-${var.cluster_name} is created with the necessary policies to interact with the S3 bucket and KMS keys.

      • Policy includes permissions for s3:ListBucket, s3:PutObject, s3:GetObject, s3:DeleteObject, kms:GenerateDataKey, and kms:Decrypt.

    2. AmazonEKS_LoadBalancer_Controller_Role-${var.cluster_name}:

      • Assumed by the Load Balancer Controller service account aws-load-balancer-controller.

      • Created using the terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks module.

      • Includes the necessary policy to manage AWS load balancers.

    3. AmazonEKS_EBS_CSI_DriverRole_${var.cluster_name}:

      • Assumed by the EBS CSI Driver service account ebs-csi-controller-sa in the kube-system namespace.

      • Allows the EBS CSI Driver to interact with EBS volumes.

    4. External_DNS_${var.cluster_name}:

      • Assumed by the external-dns service account in the kube-system namespace.

      • Allows ExternalDNS to manage DNS records in Route53.

      • The policy AllowExternalDNSUpdates grants permissions for route53:ChangeResourceRecordSets, route53:ListHostedZones, route53:ListResourceRecordSets, and route53:ListTagsForResource.

  • EKS Cluster

    • Sets up the EKS (Elastic Kubernetes Service) cluster with necessary configurations, including managed node groups configuration, worker nodes, add-ons, Kubernetes version, etc. Each managed group has a customizable number of worker nodes.

      • logscale: Nodes that host the Humio-core pods (all deployments).

      • logscale-ingress: Nodes that host the ingress pods (ingress deployment only).

      • logscale-ingest: Nodes that host the ingest-only-core pods (internal-ingest deployment only).

      • logscale-ui: Nodes that host the UI-only-core pods (internal-ingest deployment only).

  • S3 Bucket

    The S3 bucket for LogScale logs is configured using the Terraform S3 module that ensures secure storage and proper management of log data. Depending on whether a bucket prefix is specified, the bucket name will either use var.s3_bucket_prefix or default to var.cluster_name. The bucket is set to private, enforces object ownership to the bucket owner, and allows forced deletion even if objects are present. Additionally, lifecycle rules are in place to expire noncurrent object versions after one day, aiding in cost management and data hygiene.