Azure AKS

This track maps each requirement to an AKS component. It is built from the Microsoft and Harthelm documentation and has not been run end to end on AKS: prove your cluster with the checks at the end before you give it to users.

The shapeCopy Anchor Link

AKS Standard (not Automatic), Azure CNI Overlay with the Cilium data plane, the OIDC issuer and Workload Identity for cert-manager, an attached ACR, a system pool for the platform and a user pool for workspaces, LocalDNS disabled, Azure Disk or Azure Files NFS for workspaces, cert-manager with the Azure DNS solver, CloudNativePG, and an Ingress controller or the app routing Gateway API implementation.

Pieces by requirementCopy Anchor Link

Create the clusterCopy Anchor Link

az group create --name rg-harthelm --location westeurope
az aks create \
  --resource-group rg-harthelm --name aks-harthelm \
  --network-plugin azure --network-plugin-mode overlay --pod-cidr 192.168.0.0/16 \
  --network-dataplane cilium \
  --enable-oidc-issuer --enable-workload-identity \
  --attach-acr <acr name> \
  --node-count 2 \
  --generate-ssh-keys
az aks nodepool add --resource-group rg-harthelm --cluster-name aks-harthelm --name ws \
  --node-count 2 --labels example.com/pool=workspaces
az aks get-credentials --resource-group rg-harthelm --name aks-harthelm

With the Cilium data plane you need no separate policy engine, and CiliumClusterwideNetworkPolicy is supported (Azure CNI powered by Cilium). Give core the rule for its backstop and the platform pods their API access as in Network policies. On this data plane, ipBlock cannot select node or pod addresses; Harthelm's own rules use selectors.

Switch LocalDNS offCopy Anchor Link

LocalDNS makes pods resolve through 169.254.10.10 and 169.254.10.11. Harthelm allows DNS only to the kube-dns pods and keeps workspaces away from link-local addresses, so with LocalDNS on, pods would lose name resolution (expected from the rules, not verified on AKS). From 1.37, Standard pools default to Preferred. Set "mode": "Disabled" in the pool's localdnsconfig.json and apply it, which reimages the pool (LocalDNS):

az aks nodepool update --resource-group rg-harthelm --cluster-name aks-harthelm --name ws \
  --localdns-config ./localdnsconfig.json

StorageCopy Anchor Link

• Azure Disk is ReadWriteOnce on one node: use it with storage.mode rwo-colocated. Every workspace volume is one disk, and the number of disks per node depends on the VM size.
• Azure Files NFS works with rwx. The built-in Azure Files classes are SMB, whose ownership comes from mount options; Harthelm changes owners and modes on its volumes, so use a class with protocol: nfs on a Premium account:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile-nfs-premiumv2
provisioner: file.csi.azure.com
parameters: {protocol: nfs, skuName: PremiumV2_LRS}
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions: [nconnect=4, noresvport, actimeo=30]

Premium shares have a minimum size of 100 GiB; whether a smaller claim is rounded up is not verified (Azure Files).

EdgeCopy Anchor Link

With app routing Gateway API, set routing.edge gatewayAPI and routing.gatewayAPI.parentRefs to the Gateway. The Gateway needs HTTPS listeners with certificates for the console host and *.<workspace domain>, and one listener per custom domain, which is manual work: prefer an Ingress controller when your teams use custom domains.

Either way, set networkPolicy.edgeNamespaceSelector to the namespace of the gateway or controller pods and trustedProxies to their addresses. Idle IDE WebSockets depend on the load balancer's idle timeout (4 to 30 minutes).

TLS with Azure DNSCopy Anchor Link

The console and workspace domains are two registrable domains, so two Azure DNS zones. Give cert-manager a managed identity with "DNS Zone Contributor" on both zones and a federated credential for system:serviceaccount:cert-manager:cert-manager, label its pods azure.workload.identity/use: "true", and use a ClusterIssuer with dns01.azureDNS (cert-manager Azure DNS).

External databaseCopy Anchor Link

Azure Database for PostgreSQL Flexible Server 16, 17 or 18 works as an external database. Point the DSN at port 5432: the built-in PgBouncer on 6432 runs in transaction mode, which breaks the LISTEN connection core keeps.

Sign-in with Entra IDCopy Anchor Link

Entra ID sends no email_verified claim, and Harthelm attaches invitations only to verified addresses: with Entra directly, invitations never attach. Entra also emits group object IDs by default, so oidc.adminGroup would be an object ID, and reports say it refuses the scope groups that Harthelm always requests (not verified). Put a broker (Keycloak, Dex or another provider that sets email_verified and a groups claim) between Entra ID and Harthelm. See Sign-in.

Metadata and WireServerCopy Anchor Link

IMDS (169.254.169.254) is reachable from every pod by default; Harthelm's workspace policies already exclude link-local addresses. The Azure WireServer 168.63.129.16 is a public address and so not in that exclusion. Whether overlay pods reach it is not verified; on Cilium you can deny it to the team namespaces with a CiliumClusterwideNetworkPolicy of your own.

Do not use AKS AutomaticCopy Anchor Link

Deployment Safeguards are enforced there and mutate requests below 100m CPU and 100Mi memory (the router requests 25m and 48Mi), exclusions are by namespace name while team namespaces are created at runtime, and LocalDNS is on.

CheckCopy Anchor Link

helm test harthelm -n harthelm-system
hart admin doctor --capabilities

Then resolve a name in a workspace terminal to prove DNS. From Harthelm 1.2 on, hart smoke --team <id> checks a workspace end to end (Sync, prepare, Run, the app's answer on its dev URL) and deletes it again; see Check the installation.