Google GKE

This track maps each requirement to a GKE component. It is built from the Google and Harthelm documentation and has not been run end to end on GKE: prove your cluster with the checks at the end before you give it to users.

Pieces by requirementCopy Anchor Link

DNSCopy Anchor Link

This is the GKE pitfall. Harthelm lets pods ask DNS only of pods it selects by label (default k8s-app: kube-dns in kube-system) and keeps workspaces away from link-local addresses.

• Cloud DNS answers on 169.254.169.254, the metadata server, which Harthelm's rules cannot reach: every workspace loses DNS. Create the cluster with --cluster-dns=kube-dns.
• NodeLocal DNSCache is on by default in Standard from 1.34.1-gke.3720000, and its pods do not carry the kube-dns label (kube-dns). Either switch it off (--update-addons=NodeLocalDNS=DISABLED, which recreates the nodes) or widen both DNS selectors to every pod in kube-system:

teams:
  network:
    dns: {namespaceSelector: {kubernetes.io/metadata.name: kube-system}, podSelector: {}}
networkPolicy:
  dns: {namespaceSelector: {kubernetes.io/metadata.name: kube-system}, podSelector: {}}

See teams.network.dns and networkPolicy.dns. Prove it with a name lookup in a workspace terminal.

Network policyCopy Anchor Link

Dataplane V2 enforces NetworkPolicy always (Dataplane V2). Whether it honours endPort is not stated: helm test proves it on your cluster.

--enable-cilium-clusterwide-network-policy makes Harthelm render its Cilium backstop, which uses fields the GKE page does not list. Whether GKE accepts them is not verified: leave the flag off (Harthelm then relies on NetworkPolicy alone, as on any other CNI) or turn it on and check that the harthelm-backstop-rg-* objects are accepted.

EdgeCopy Anchor Link

GKE's container-native load balancing (GKE Ingress, GKE Gateway) sends traffic and health checks from Google's front ends straight to pod IPs, and core and the router accept traffic only from pods. Run an in-cluster controller behind a type: LoadBalancer Service (a passthrough Network Load Balancer) instead.

If you use GKE Gateway anyway, add a NetworkPolicy through extraObjects that admits the load balancer's ranges to core and the router on port 8080, and raise the backend timeout: idle WebSockets close after the default 30 seconds (GCPBackendPolicy.timeoutSec).

TLS with Cloud DNSCopy Anchor Link

Give cert-manager a Google service account with roles/dns.admin (or a custom role with dns.resourceRecordSets.*, dns.changes.* and dns.managedZones.list), bind it with Workload Identity, and use the solver cloudDNS: {project: <project id>} without serviceAccountSecretRef (cert-manager Cloud DNS):

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:<project id>.svc.id.goog[cert-manager/cert-manager]" \
  dns01-solver@<project id>.iam.gserviceaccount.com
kubectl annotate serviceaccount --namespace=cert-manager cert-manager \
  "iam.gke.io/gcp-service-account=dns01-solver@<project id>.iam.gserviceaccount.com"

StorageCopy Anchor Link

Persistent Disk and Hyperdisk are block volumes with a root-owned 0755 root: they cannot hold workspace volumes, because workspace pods run as a non-root UID without fsGroup. Use them for the databases and the backup volume, with backup.fsGroup set.

The Filestore CSI classes do not fit as they are: every claim is an instance of at least 1 TiB, the share's root is 0755, and the classes are WaitForFirstConsumer, which storage.mode: auto reads as ReadWriteOnce. Share one Filestore instance through csi-driver-nfs with mountPermissions: "0775", and set storage.mode rwx explicitly. Volumes are directories of one share, so sizes are not enforced per volume. This setup is not verified on GKE.

External databaseCopy Anchor Link

Cloud SQL for PostgreSQL 16 to 18 works as an external database. Connect over private IP and narrow networkPolicy.egress.database to it. Use verify-full with the shared or customer-managed CA, verify-ca with the per-instance CA. The chart has no Cloud SQL Auth Proxy sidecar, and a transaction-mode pooler breaks core's LISTEN connection.

Sign-in with GoogleCopy Anchor Link

Google's ID token has email_verified but no groups claim, so oidc.adminGroup never matches, and Google may refuse the scope groups that Harthelm always requests (not verified). Sign in through a broker that issues groups (Dex, Keycloak) instead. See Sign-in.

A cluster to start fromCopy Anchor Link

Every flag is from the gcloud container clusters create reference; the combination is not verified:

gcloud container clusters create harthelm \
  --location=europe-west3 --release-channel=regular \
  --enable-ip-alias --enable-dataplane-v2 \
  --cluster-dns=kube-dns \
  --workload-pool=<project id>.svc.id.goog \
  --addons=GcePersistentDiskCsiDriver,GcpFilestoreCsiDriver,HttpLoadBalancing \
  --node-labels=example.com/pool=platform --num-nodes=1 --machine-type=e2-standard-4
gcloud container node-pools create workspaces --cluster=harthelm --location=europe-west3 \
  --node-labels=example.com/pool=workspaces --workload-metadata=GKE_METADATA --machine-type=e2-standard-8

Spot node pools carry cloud.google.com/gke-spot=true, possibly with a NoSchedule taint: a fit for scheduling.ephemeralNodeSelector with a toleration in scheduling.workspaces.

CheckCopy Anchor Link

helm test harthelm -n harthelm-system
hart admin doctor --capabilities

Then resolve a name in a workspace terminal to prove DNS. From Harthelm 1.2 on, hart smoke --team <id> checks a workspace end to end (Sync, prepare, Run, the app's answer on its dev URL) and deletes it again; see Check the installation.