Azure Reference Architecture

This section of the documentation contains Terraform configurations to deploy a Microsoft Azure-based architecture for LogScale. It leverages multiple Azure services including Azure Key Vault, Azure Kubernetes Service, and Azure Storage.

The following guidance box provides links to the main section of this documentation:

Architecture Types

This section describes the supported architecture types.

Requirements

This section describes prerequisites for deploying a Microsoft Azure reference architecture for LogScale.

Terraform Modules

This section describes Terraform modules for deploying the Azure reference architecture.

Build Process

This section describes the build process.

Overview

The logscale-azure terraform modules provide the following architecture choices:

Basic use cases:

  • Development, Testing

  • Smaller Search Teams

  • Minimal ingest processing

Ingress use cases:

  • Ingress tier in specified DMZ

  • Ingress resources not shared with Kubernetes system resources

Dedicated UI use cases:

  • Separated ingress tier

  • Separated UI tier

  • Ingest/Digest on same hosts

Advanced use cases:

  • Dedicated processing tier (ingestion) scaling separate from digest/storage

  • Dedicated UI tier for dashboards and search

  • Segmentation of system responsibilities and independent scaling

All architecture choices rely on the same underlying technologies:

  • Azure Infrastructure and Resource Groups - Note that the user will have 2 resource groups, 1 created by this terraform and the other associated to the managed Kubernetes services.

  • Azure Kubernetes Service (AKS)

  • Azure NAT Gateway - Allows egress data to pull images for kubernetes pods

  • Azure KeyVault:

    • Stores kube secrets for the environment

    • Stores LogScale encryption string

    • Stores LogScale single user password

    • Stores encryption key for AKS disk encryption set

  • Azure Load Balancer - Allow data ingest and UI access to the environment

  • Azure Storage Account - Object storage for LogScale data durability

  • Kubernetes Apps:

    • cert-manager: For automated provisioning of certificates in the environment

    • strimzi-operator: For provisioning Kafka broker nodes (kraft mode)

    • humio-operator: For provisioning of LogScale clusters in the environment

    • nginx-ingress: For connecting the Azure Load Balancer

By default:

  • Kubernetes API is public but has IP restrictions applied based on user-controlled variables to limit access.

  • Ingest endpoint is publicly available:

    • Port 80: Used for Let's Encrypt certificate signing - Access is global but only available temporarily during ACME challenge/response while the cert-manager response pod is alive.

    • Port 443: Used for ingest/UI to LogScale - Access is limited to user configured IP list variable

  • Storage account access is restricted to:

    • User provided IP ranges

    • Subnet(s) created for LogScale nodes

    • Azure Trusted Services, Logging, and Metrics

  • KeyVault access is restricted to:

    • User provided IP ranges

    • Azure Trusted Services

  • All AKS nodes are made available on required ports to AzureCloud IP ranges to allow for Kubernetes control plane operations

  • All AKS nodes are made accessible within the VNET

Important

Some settings are configurable and still pending testing. For example, it is a configuration option to make the Load Balancer and Kubernetes API internal-only but this has not been tested.

Networking is created as follows:

Purpose Architecture Created IP Range
Virtual Network all 172.16.0.0/16
Kubernetes System Nodes all 172.16.0.0/24
Kafka Nodes all 172.16.2.0/24
Bastion Nodes all 172.16.1.0/26
LogScale UI Nodes advanced 172.16.6.0/24
Nginx Ingress Nodes ingress, advanced 172.16.4.0/24
LogScale Ingest Nodes advanced 172.16.5.0/24
LogScale Digest Nodes all 172.16.3.0/24

Diagnostic (Audit) Logging

Logging can be enabled for supported resources to supported target destinations that must be created outside of this terraform. Currently supported resources:

  • Azure Key Vault

  • Azure Kubernetes Service

  • Azure Storage Account

Currently supported destinations:

  • Azure Event Hub

  • Azure Storage Account

  • Azure Log Analytics

Update your created .tfvars file to enable this feature. Example:

ini
enable_auditlogging_to_storage      = true
diag_logging_storage_account_id     = "/subscriptions/$mysubscription/resourceGroups/$myresourcegroup/providers/Microsoft.Storage/storageAccounts/$mystorageacct"