Amazon EKS

This track maps each requirement to an EKS component. It is built from the AWS and Harthelm documentation and has not been run end to end on EKS: prove your cluster with the checks at the end before you give it to users.

Pieces by requirementCopy Anchor Link

Network policyCopy Anchor Link

Switch it on in the VPC CNI add-on:

aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni --resolve-conflicts PRESERVE \
  --configuration-values '{"enableNetworkPolicy": "true"}'

Afterwards the aws-node pods show 2/2 containers (configure network policy).

Things to know about the VPC CNI's enforcement:

• In the default standard mode, a new pod starts allowing everything until its policies are programmed. NETWORK_POLICY_ENFORCING_MODE=strict starts pods denied, but then CoreDNS needs policies of its own.
• AWS requires "the service port must be the same as the container port" (EKS network policy). Harthelm's git Service maps port 8443 to 8445 (and 8080 to 8082). Whether clones through the git proxy are affected is not verified: test a workspace from a Git repository before you rely on it.
• endPort is not mentioned in the EKS guide. helm test proves it on your cluster.
• A NetworkPolicy deleted and recreated under the same name may stay unenforced for minutes (controller #265). helm test recreates its probe policies on every run, so a "not enforced" right after a re-run may be false: run it again later.

EdgeCopy Anchor Link

Core and the router accept traffic only from pods. An ALB in IP mode sends from outside the cluster straight to pod IPs and is dropped. Run an in-cluster controller (Traefik, Envoy Gateway, NGINX Gateway Fabric or another with WebSockets; not ingress-nginx, which is retired) behind an NLB:

• service.beta.kubernetes.io/aws-load-balancer-type: "external"
• service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip" with the VPC CNI, "instance" with Cilium
• service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" (the default is internal)

Keep the client address: either service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true, or proxy protocol v2, which your controller must then speak (annotations). Then set trustedProxies to the controller pods' addresses.

DNS and TLSCopy Anchor Link

Create Route 53 alias records for console.example.com and *.example.net to the NLB, and set customDomains.cnameTarget to a name that resolves to it. cert-manager needs route53:GetChange, route53:ChangeResourceRecordSets, route53:ListResourceRecordSets and route53:ListHostedZonesByName; with Pod Identity the solver is just route53: {}. Use a ClusterIssuer: cert-manager uses ambient credentials only for those (cert-manager Route 53).

StorageCopy Anchor Link

Workspace pods run as a non-root UID in group 0 without fsGroup, so the volume root must be group-writable. A fresh EBS volume is root:root 0755: it cannot hold workspace volumes, and rwo-colocated does not help.

Use one EFS file system through csi-driver-nfs:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: workspaces-efs
provisioner: nfs.csi.k8s.io
parameters:
  server: <file system id>.efs.<region>.amazonaws.com
  share: /
  mountPermissions: "0775"
mountOptions: [nfsvers=4.1]
reclaimPolicy: Delete
volumeBindingMode: Immediate

Then set storage.class workspaces-efs and storage.mode rwx. Volumes are directories of one file system, so sizes are not enforced per volume. EFS access points (provisioningMode: efs-ap) map every file operation to the access point's UID and are not verified with Harthelm.

Use EBS gp3 with volumeBindingMode: WaitForFirstConsumer for the databases and the backup volume, and set backup.fsGroup, because a fresh ext4 root is not group-writable. EBS is zonal: keep a platform node in every zone that holds a database volume.

External databaseCopy Anchor Link

Amazon RDS for PostgreSQL 16 to 18 works as an external database. Use sslmode: verify-full with the RDS CA bundle in database.external.ca, connect core to the instance endpoint directly (not through RDS Proxy: LISTEN pins connections there), and set networkPolicy.egress.database to the database subnets.

Sign-in with CognitoCopy Anchor Link

Cognito puts groups in cognito:groups, so set oidc.groupsClaim to cognito:groups (Cognito ID token). Harthelm always requests the scope groups; whether Cognito accepts it is not verified: test a sign-in first, or put a broker that issues groups in front.

A cluster to start fromCopy Anchor Link

An eksctl config with the pieces above; the combination is not verified:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata: {name: harthelm, region: eu-central-1, version: "1.36"}
addons:
  - name: vpc-cni
    configurationValues: '{"enableNetworkPolicy": "true"}'
    resolveConflicts: overwrite
  - name: coredns
  - name: kube-proxy
  - name: eks-pod-identity-agent
  - name: aws-ebs-csi-driver
  - name: aws-efs-csi-driver
managedNodeGroups:
  - name: platform
    instanceType: m7i.large
    desiredCapacity: 2
    labels: {example.com/pool: platform}
    disablePodIMDS: true
  - name: workspaces
    instanceType: m7i.xlarge
    desiredCapacity: 2
    labels: {example.com/pool: workspaces}
    disablePodIMDS: true

Then: IAM roles for the EBS and EFS drivers through Pod Identity, an EFS file system with mount targets in the node subnets and a security group that allows NFS (TCP 2049) from the nodes, the two StorageClasses, the AWS Load Balancer Controller, an Ingress controller behind an NLB, cert-manager with the Route 53 ClusterIssuer, the CloudNativePG operator in cnpg-system, and Harthelm.

Spot node groups carry eks.amazonaws.com/capacityType: SPOT: a fit for scheduling.ephemeralNodeSelector. Keep core, the router and every database on on-demand nodes.

CheckCopy Anchor Link

helm test harthelm -n harthelm-system
hart admin doctor --capabilities

Then create a workspace from a Git repository (the git proxy's port mapping) and resolve a name in its terminal. From Harthelm 1.2 on, hart smoke --team <id> checks a workspace end to end (Sync, prepare, Run, the app's answer on its dev URL) and deletes it again; see Check the installation.