On-premises clusters

On your own hardware you choose every piece Harthelm needs. This page says which choices work, by requirement, for kubeadm, k3s and RKE2. Then install as in Install with Helm.

Pieces by requirementCopy Anchor Link

k3s: replace the built-in policy controllerCopy Anchor Link

k3s enforces NetworkPolicy with an embedded kube-router controller. Two open issues affect Harthelm: rules that combine a podSelector with a namespaceSelector are intermittently not enforced (k3s #14776), which is exactly the shape of Harthelm's DNS and platform rules, and new pods start unfiltered for a short window (k3s #14711). Run k3s with --flannel-backend=none --disable-network-policy and install Cilium or Calico (k3s network options).

CiliumCopy Anchor Link

Cilium cannot match the API server by address in a NetworkPolicy. Give core, the router, the preflight Job and the bundled database their API access with CiliumNetworkPolicies, and give core the rule for its per-team backstop. The full values, a supported starting point since 1.5.0 that you keep as a second values file beside yours, are in Network policies.

Without a load balancerCopy Anchor Link

On bare nodes there is often no cloud load balancer: the Ingress controller listens with a hostPort (or a NodePort) on some or all nodes, and the names of the console, the workspace wildcard and the OIDC issuer are plain A records of the nodes' public addresses. Three things differ from the vanilla install.

• trustedProxies is the range of the Ingress controller's pods, not of the nodes (kubectl -n <ingress namespace> get pods -o wide shows it). Core and the router believe X-Forwarded-For only from there. With a hostPort the controller sees the real client address when your CNI does not source-NAT it (Cilium does not), and it replaces the header of clients it does not trust. A node range or 0.0.0.0/0 would let any client forge the address that rate limits and audit rows use.

• The OIDC issuer behind the same nodes. When the same Ingress controller serves the issuer (a Dex in the cluster, for instance), core reaches it through the nodes' public addresses. To the address of the node core runs on, the CNI hands the connection straight to the controller's pod; to another node's address it leaves the cluster and comes back as external traffic. networkPolicy.egress.idp has to allow both: the Ingress controller's namespace (the pod, on its own port: for example 8443 behind a hostPort 443) and the nodes' public addresses.

  Copy Text

  networkPolicy:
    egress:
      idp:
        to:
          - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: <ingress namespace>}}
          - ipBlock: {cidr: 203.0.113.10/32}   # one entry per node that carries the issuer's A record
          - ipBlock: {cidr: 203.0.113.11/32}
        ports: [443, 8443]

  When this is missing or wrong, the doctor's oidc check fails with a connection error, which looks the same for "the policy drops it" and "the address is unreachable". Read the CNI's drop log, or run kubectl debug on a copy of core's pod (its image has no shell) and fetch https://<issuer>/.well-known/openid-configuration. Since 1.5.0 the check's detail names the likely cause: a refused, unreachable or timed-out connection points to networkPolicy.egress.idp and this section, a certificate error to the private CA, an unknown host to the resolver.

• Node identities on Cilium. If Cilium learns the public addresses as node addresses (a cloud controller manager that sets ExternalIP on the nodes, for instance), traffic to them is classified host or remote-node, an ipBlock no longer matches, and sign-in fails depending on which A record core resolves. Add a CiliumNetworkPolicy for core with toEntities: [host, remote-node] on port 443 to extraObjects, in the same list as the Cilium values. It is only for this layout, since it opens port 443 of every node to core:

  Copy Text

  - apiVersion: cilium.io/v2
    kind: CiliumNetworkPolicy
    metadata: {name: harthelm-core-idp, namespace: harthelm-system}
    spec:
      endpointSelector:
        matchLabels: {app.kubernetes.io/name: harthelm, app.kubernetes.io/component: core}
      egress:
        - toEntities: [host, remote-node]
          toPorts: [{ports: [{port: "443", protocol: TCP}]}]

The workspace wildcard certificate needs a DNS-01 issuer (HTTP-01 cannot issue a wildcard), and the HTTP-01 challenges of custom domains need port 80 of the nodes to reach the controller.

StorageCopy Anchor Link

• Node-local disks (TopoLVM, local-path) with storage.mode rwo-colocated are the fast choice: every pod of a workspace runs on the node of its volume. local-path ignores the requested size, so the team quota only counts what was requested.
• NFS with rwx: csi-driver-nfs on an export with no_root_squash is the tested setup; root_squash is not tested. Network storage is much slower for many small files (package installs, git status on large trees), so prefer local disks for developer-heavy installs.
• Longhorn RWX serves volumes through NFS share-manager pods; while one restarts, I/O to its volume blocks.

Storage explains the modes and the UID rule.

Local-path storageCopy Anchor Link

Rancher's local-path class (the default of k3s and of many small clusters) and other node-local classes work within these limits:

Pin where the data goes with scheduling.platform and the scheduling block of each managed service, and keep it off nodes that are drained or switched off.

A private CACopy Anchor Link

If your OIDC provider or Git server uses a certificate from a private CA, put the CA into a ConfigMap and name it:

kubectl -n harthelm-system create configmap private-ca --from-file=ca.crt=./ca.crt

oidc:
  caConfigMap: private-ca
git:
  caConfigMap: private-ca

For the edge and custom domains, a cert-manager CA Issuer or an internal ACME server works as routing.tls.certManager.issuerRef and customDomains.issuer. With a public ACME server, the wildcard for the workspace domain needs DNS-01.

CheckCopy Anchor Link

helm test harthelm -n harthelm-system
hart admin doctor --capabilities

Both must say the NetworkPolicies are enforced with endPort. Then open a terminal in a workspace and resolve a name (getent hosts example.com) to prove DNS. From Harthelm 1.2 on, hart smoke --team <id> checks a workspace end to end (Sync, prepare, Run, the app's answer on its dev URL) and deletes it again; see Check the installation.