The reference architecture diagram for a GCP deployment looks like the following diagram:

This includes a number of distinct components described below.

Private GKE cluster

Private clusters offer increased isolation by utilizing private IP addresses for nodes and providing private or public endpoints for control plane nodes, which can be further isolated. With private clusters, we can still access Google APIs through Private Google Access. Within a private cluster, Pods are segregated from both inbound and outbound communication, establishing a cluster perimeter. The directional flows of these communications can be managed by exposing services through load balancing and Cloud NAT. To enable a private endpoint when you create a cluster, use the --enable-private-endpoint flag.

Bastion Host

By default the Terraform will create a bastion host. The bastion host creation can be disabled by adding setting the bastion_host_enabled variable to false on the command line or by adding it to the _override.tf file. A jump host plays a crucial role in enhancing the security and manageability of the GKE private cluster. The cluster nodes are not directly accessible from the public internet, which provides an inherent layer of security. The host serves as an intermediary server that allows authorised users to access the private cluster securely. It acts as a single entry point for SSH access.

The Bastion host acts as a barrier between the public internet and the cluster's internal network, mitigating the risk of unauthorised access or attacks. Access to the bastion host can be tightly controlled using standard authentication mechanisms, in this setup we use SSH key pairs to access the Bastion host and the cluster private nodes, user can generate ssh key pair.

The Bastion host can also be utilised for tasks like maintenance, troubleshooting, and updates. Once deployed, the host can be used as a proxy to run changes on the private cluster as described in Google Documentation.

By default, ssh sessions to GCP VMs timeout after 10 minutes, the session can be extended by adding --ssh-flag="-ServerAliveInterval=60" to gcloud compute ssh command.

The host has a dedicated ssh firewall rule to restrict access except from IP addresses that gcloud IAP uses for TCP forwarding as per Google Documentation

Cert Manager and Let's Encrypt issuer

Cert Manager is a popular Kubernetes tool used for managing the TLS certificates. It is combined with Let's Encrypt issuer to automate the process of obtaining and renewing SSL/TLS certificates for LogScale application. Cert Manager is responsible for certificate management within the GKE cluster, it can generate renew and keep track of TLS certificates. Let's encrypt is a certificate authority that provides TLS/SSL certificates.

Cloud NAT:

To enable outbound internet connectivity for the pods within the private GKE cluster, you can set up Cloud NAT in your VPC network. Cloud NAT acts as a gateway that translates the private IP addresses of the pods to public IP addresses, allowing them to access the internet. It's configured for logscale egress traffic.

Specific component version requirements for deploying LogScale, including the Humio Operator chart 0.22.0, Strimzi helm chart 0.37.0, cert-manager helm chart v1.13.1, and kubernetes version 1.28+. The deployment specifications also require LogScale Docker image version 1.142.1 or higher and supports x86_64 and amd64 CPU architectures.

Deployment Versions list

Essential prerequisites and tooling requirements for deploying LogScale on Google Cloud Platform (GCP), including specific version requirements for tools like Terraform, kubectl, gcloud CLI, and Helm. Key infrastructure requirements cover bucket storage using NVMe SSDs, Kubernetes cluster configuration, Strimzi Operator for Kafka management, TLS security implementation, TopoLVM for disk preparation, Workload Identity for GCS authentication, and instance sizing recommendations for different deployment scales.

Before following this guide, there are some basic prerequisites and tooling:

Assumptions:

Kafka clusters are separate from LogScale clusters to avoid resource contention and allow independent management.

Step-by-step instructions for deploying LogScale on Google Cloud Platform (GCP) using Terraform, including the setup of required GCP resources like Compute Engine, GCS, IAM, Kubernetes Engine, and VPC Networks. The deployment process covers cloning necessary repositories, configuring Terraform state storage, deploying GCP infrastructure, setting up cluster credentials, and installing LogScale components with proper licensing and public URL configuration.

In order to deploy the following Terraform you will need sufficient permissions, the Terraform will require access to create, modify, and delete resources in the following GCP services:

Learn how to access a deployed LogScale instance on Google Cloud Platform (GCP) by configuring DNS records and logging into the system. After obtaining the IP address from the ingress resource and setting up appropriate DNS records, users can access the LogScale cluster using admin credentials, where the password can be retrieved through a specific kubectl command.

To access the deployed cluster:

$ kubectl -n logging get ingress
# example output
# name                          class    hosts   address      ports   age
# logscale-xxxxx-basic-ingress   none   *      24.14.19.33  80      20m

$ kubectl get secret logscale-XXXX-static-users  -n logging -o=template --template={{.data.users}} | base64 -D
# Example output: 
# Depending on the shell this might include a % sign, omit it when you copy the password.
# admin:XXXXXXXXXXXX%

Step-by-step instructions for testing a GCP deployment by sending data to a LogScale cluster, including the creation of repositories and ingest tokens. This page details how to configure and install Fluentbit in a Kubernetes environment to gather pod logs, and explains the verification process to ensure proper log ingestion through the LogScale UI.

Once deployed, the deployment should be tested:

$ helm repo add humio https://github.com/humio/humio-helm-charts/
$ helm repo update

humio-fluentbit:
  enabled: true
  humioHostname: $INGRESS_ES_HOSTNAME
  es:
    tls: true
    port: 443
    inputConfig: |-
      [INPUT]
           Name             tail
           Path             /var/log/containers/*.log
           Parser           docker
           # The path to the DB file must be unique and
           # not conflict with another fluentbit running on the same nodes.
           DB               /var/log/flb_kube.db
           Tag              kube.*
           Refresh_Interval 5
           Mem_Buf_Limit    512MB
           Skip_Long_Lines  On
    resources:
      limits:
        cpu: 100m
        memory: 1024Mi
      requests:
        cpu: 100m
        memory: 512Mi

$ helm install test humio/humio-helm-charts \ --namespace logging \ --set humio-fluentbit.token=$INGEST_TOKEN \ --values humio-agent.yaml

The cleanup process for a GCP deployment includes infrastructure removal using Terraform destroy commands and manual deletion steps. The process involves executing specific commands from the logscale-gcp-components directory, removing DNS records, and deleting the GCS bucket named 'logscale-terraform-state' through the console.