Harthelm 1.6.0

Released 2026-10-20. API version 0.12.0. A minor release with no database migration and two new values, both off by default. Nothing needs to be done before the upgrade, but some things restart once after it: every MariaDB instance, every RabbitMQ broker, and the services that read a managed PostgreSQL. Read Before you upgrade.

helm upgrade harthelm oci://<your registry>/charts/harthelm --version 1.6.0 -n harthelm-system -f my-values.yaml

From 1.6.0 on, <your registry> can be Harthelm's own registry, registry.harthelm.dev/harthelm, with the login that comes with your licence: see Get Harthelm.

Before you upgradeCopy Anchor Link

• Core and the router roll, as with every release: their image changes. The backup jobs take the new image at their next run.
• Every MariaDB instance and every RabbitMQ broker restarts once. Docker Hub rebuilt the images of MariaDB 11.4.13, 11.8.9 and 12.3.3 and RabbitMQ 4.3.6 after Harthelm pinned them, and 1.6.0 pins the rebuilds: the same versions, new digests. After the upgrade each instance is rendered with its version's new build and restarts once onto it, with its volume and its settings. Clients lose their connection for the restart and reconnect; nothing else about the instances changes.
• Services that read a managed PostgreSQL restart once. The databases' details gain serverCA and tlsHost (New). When core first reports them, shortly after the upgrade, every service whose env reads a managed PostgreSQL (${{ services.<name>.dsn }} and the like) restarts once; one that sleeps takes the change when it wakes. The databases themselves do not restart, and a later renewal of their CA restarts only the services that read serverCA itself.
• Workspaces move to the agent's image volume at their next start, where the cluster supports it (New). Nothing restarts for it: running workspaces keep their pods until they sleep or stop and show "Upgrade pending" meanwhile. To keep 1.5's init container, set agent.delivery to initContainer in the same upgrade.
• Core's RBAC changes. Core's ClusterRole no longer reads Secrets; the chart adds a ClusterRole that core binds to itself in each team namespace, and core may write RoleBindings for it (Changed). Whoever upgrades needs to be able to grant these, as before: in practice cluster admin. After the upgrade the new core writes one RoleBinding per team namespace; until a namespace has it, core's reads there are refused and its reconciles retry, for seconds after the start. Tooling of your own that relied on core's ServiceAccount reading Secrets in other namespaces loses that.
• Two new values, both off. networkPolicy.cilium.enabled and networkPolicy.egress.idp.viaIngress. The 1.5 chart's schema rejects them: if you keep your values in Git for Argo CD, move the pin to the 1.6 chart before or together with a change of these values.
• On Cilium with 1.5's example values, switch to the flag in the same upgrade: drop the four Cilium entries from extraObjects and the ciliumclusterwidenetworkpolicies rule from rbac.extraClusterRules, and set networkPolicy.cilium.enabled: true. Under the release name harthelm the objects keep their names and are updated in place. See Network policies › Cilium.
• A log filter may need a change. A PostgreSQL dump now says tls_verify=server, the platform dump's backup written carries sslmode and tls_verify, the router logs core's clean end of the route stream at INFO, and core's in-cluster API port logs server=api-tls once instead of also server=core (Changed).
• hart 1.5 keeps working against 1.6 (the API is the same), but warns that its minor version differs. Update it from the release's files (Install hart); 1.6 adds hart ws purge.

NewCopy Anchor Link

The public release channel. Every release from 1.6.0 on is published on two hosts. registry.harthelm.dev holds the images and the chart: every image one level below registry.harthelm.dev/harthelm/, multi-arch, and the chart at oci://registry.harthelm.dev/harthelm/charts/harthelm, pinned to those images by digest. Nothing there is anonymous: the user name is your licence id and the password the registry token that comes with your licence, one per installation, which stops working when the licence ends. get.harthelm.dev/releases/<version>/ holds the release files: the same chart as harthelm-<version>-public.tgz, images-public.txt and mirror.sh to mirror the images into your own registry, the hart archives, the SBOMs, the restore runbooks and the air-gapped bundles, each with its checksum. The source code of the open-source components is a plain download under get.harthelm.dev/sources/. See Get Harthelm.

Signed releases. Every image of a release (its index and each platform's manifest) and its chart are signed with Harthelm's release key, with cosign and without a transparency log: you verify with the public key alone, cosign verify --key cosign.pub --insecure-ignore-tlog. The key is on Get Harthelm. Mirrors and bundles carry no signatures, so verify on the connected side before you copy.

SBOMs. harthelm-<version>-sboms.tar.gz holds one SPDX 2.3 document per Harthelm image and platform, made from the released images. See Read the SBOMs.

Air-gapped bundles. harthelm-<version>-amd64.tar and harthelm-<version>-arm64.tar each hold every image of the release for that platform as an OCI image layout, a chart pinned to those images, mirror.sh and a README. On a host that reaches only your registry, mirror.sh --from-bundle loads the images with their digests and mirror.sh --verify compares every digest afterwards. See Air-gapped bundles.

The agent as an image volume. Every workspace pod runs the Harthelm agent. Until 1.5 an init container copied it into each pod. Harthelm 1.6 checks whether your cluster can mount the agent's image as a Kubernetes image volume instead (Kubernetes 1.33 or newer with the ImageVolume feature, and a container runtime that supports it, such as containerd 2.1 or newer). Core groups the nodes workspace pods may run on by kubelet version, container runtime and architecture, and runs a short probe Job per group in its own namespace, at start and hourly: a pod mounts the agent image exactly as workspace pods do and runs the agent from it. Only when that worked on every group does Harthelm switch: with the default agent.delivery: auto each workspace then gets the agent as an image volume at its next start, without the copy, so it starts a little faster and without the copy's memory. Where the probe does not pass, workspaces keep the init container. Nothing is restarted for it. Platform → Capabilities shows Agent delivered as an image volume and what the probe found; to stay on the copy, set agent.delivery: initContainer. See Workspaces › Agent delivery.

The capability report on the Doctor page. Platform → Doctor lists, under the checks, what hart admin doctor --capabilities adds: the Kubernetes version, when the cluster was probed, the volumes new workspaces get, and every optional feature with its state and, when it is not available, the reason, in the same rows as the Capabilities page. Run again reads that report again too; it runs nothing on core. The NetworkPolicy self-check stays on the Capabilities page, which the section links to. See Monitoring › The Doctor page.

hart ws purge. A team admin purges a deleted workspace at once instead of after its 7 days, by its id or by its name among the team's deleted workspaces: its volume goes and the storage it held no longer counts against the team's quota. It asks first unless --yes is given. See hart ws purge.

PostgreSQL's server certificate in the details. A managed PostgreSQL's details carry serverCA, the certificate of the CA that signs the server's certificate, and tlsHost, the name that certificate carries. An app that wants to check the server connects to ${{ services.<name>.details.tlsHost }} with sslmode=verify-full and the CA from ${{ services.<name>.details.serverCA }} written to a file as sslrootcert. See Managed services › Connecting.

Chart flags for Cilium. networkPolicy.cilium.enabled renders the CiliumNetworkPolicies that let core, the router, the preflight Job and the bundled database reach the API server, and gives core the rule for its per-team Cilium backstop. They replace the values example of 1.5. networkPolicy.egress.idp.viaIngress lets core reach an identity provider that your own Ingress controller serves behind the nodes' addresses. See Network policies › Cilium and On-premises clusters › Without a load balancer.

ChangedCopy Anchor Link

• Core reads Secrets only in its own namespace and in the team namespaces. Until 1.5 Kubernetes RBAC let core's ServiceAccount read Secrets in every namespace. Its ClusterRole now has no read on Secrets. It reads them in the release namespace through its Role, and in each team namespace through a RoleBinding that core writes right after the namespace: it binds the chart's ClusterRole <release>-core-team (read on Secrets, nothing else, never bound cluster-wide) to core's own ServiceAccount. Core may bind that ClusterRole and no other, and the admission guard admits no other RoleBinding of core's. Core still writes Secrets across the cluster, confined like every write of core's by the admission guard to its team namespaces and the release namespace. See Permissions.
• PostgreSQL dumps verify the server. The nightly dumps of managed PostgreSQL connect to tlsHost with sslmode=verify-full and the operator's CA, which core reads from the database's namespace; a server that does not prove itself with that CA and name is not dumped. Until core has reported an instance's CA, its dump falls back to sslmode=require with a warning. The platform database's dump does the same with the bundled database's CA, which the backup pods mount. The dump lines say tls_verify=server, and backup written carries sslmode and tls_verify. An external platform database keeps its database.external.sslmode. See Backups and restore.
• The quota refusal names the deleted workspaces. When recently deleted workspaces hold storage, the refusal says how much, how many workspaces, and how to free it: "20 GiB are held by 2 recently deleted workspaces; a team admin can purge them now to free the space (console: Recently deleted; CLI: hart ws purge)." The team's Settings → Quota says the same next to the held space.
• The router logs core's clean end of the route stream at INFO. A rolling update of core no longer costs one WARN per router. A restart that leaves no core ready (the only core pod deleted, evicted or crashed) still logs a WARN per refused connect until a new core is ready.
• The workspace agent logs what a stop dropped. When a service stops with stage events core has not taken yet, the agent logs one WARN, stopping with stage events core has not taken, with the number of events and the steps. The line is for diagnosis, not a stable interface.
• The Cilium values example of 1.5 is gone. The chart flags above render the same objects under the same names.

FixedCopy Anchor Link

• A restricted installation refused a first sign-in without a reason. The login page showed nothing and core logged an INFO line about a disabled user. Now the page says that the installation has no valid licence, that an administrator installs it, and that the break-glass account is the way in until then (the code licence_restricted); core logs the refusal at WARN with the installation id. account_disabled is left for a user who is disabled in Harthelm. See Sign-in › Sign-in errors.
• The nightly MariaDB dump failed for an instance that was still starting. Since 1.5 the backup run skips a managed service that has not reported where it listens yet, but the MariaDB dump, a Job per instance in the server's own image, still failed on one. It now skips it the same way, with the same WARN line, and the next night dumps it. See Backups and restore.
• hart ws delete --wait printed the deletion as pending after the wait had seen it succeed. It prints the operation as the wait ended, in the table and with -o json or -o yaml.
• Core's in-cluster API port named two servers in its log lines, "server":"api-tls","server":"core", which a JSON reader takes as core. They carry server=api-tls once; a filter on server=core no longer matches them.

For operatorsCopy Anchor Link

The release channel. An installation that mirrors a release from images.txt with mirror.sh keeps working as before. To move to the public channel, log in with your licence id and registry token, give the nodes a pull Secret in global.imagePullSecrets, and install the chart from registry.harthelm.dev; or mirror from images-public.txt into your registry, where the images land at <your registry>/harthelm/<image> and global.imageRegistry is your registry's host. A registry filled in that one-level layout keeps the chart under charts/harthelm, so an installation that follows one moves its chart source to <host>/<project>/charts and global.imageRegistry to <host> in the same change as the version; a rollback to 1.5 takes all three back. Before the release, the distribution was rehearsed against the live registry: requests without a login, forged tokens and an expired licence's token are refused, and a licence's token may pull and nothing else.

The image-volume probe. With agent.delivery auto or imageVolume, core runs Jobs harthelm-agent-probe-… in the release namespace, labelled app.kubernetes.io/component: agent-probe: one per node runtime that is due (new, an hour after a failure, a day after a pass, and again when a node's kubelet or runtime changes), pinned to a ready node of the group, with the workspaces' tolerations and pull Secrets, no API token, under Pod Security restricted. Each is deleted after its probe. Core logs every probe as image-volume probe (INFO when it passed, WARN otherwise, with the node, its runtime and what the agent printed). The doctor's capabilities check ends with the agent delivery in one sentence, and warns when imageVolume is forced and the probe has not passed. With initContainer nothing is probed.

RBAC. Core's ClusterRole: Secrets create, update, patch and delete, no read; RoleBindings in full, and bind of the team ClusterRole only. Core's Role in the release namespace: Secrets read. The team ClusterRole <release>-core-team: Secrets read. The admission guard has one rule more, for core's RoleBindings. With rbac.create: false you bring core's RBAC yourself, core writes no RoleBindings and reads Secrets with whatever you grant it.

Backups. The backup CronJob, the dump Jobs made from it and the pre-upgrade dump mount the bundled database's CA (<release>-db-ca, optional) at /etc/harthelm/database-ca/ca.crt. Until the operator has created it, the platform dump warns and connects with sslmode=require.

API 0.12.0. Unchanged; only the description of the quota problem's held_by_deleted names the new wording.

Changed valuesCopy Anchor Link

agent.delivery keeps its default, auto, which now follows the probe.

Known limitsCopy Anchor Link

• Mirrors and bundles carry no signatures. Verify the images and the chart while you still reach registry.harthelm.dev. A bundle's own chart is not signed; its checksum covers it.
• A bundle is for one architecture. A cluster with nodes of both needs the multi-arch images from images-public.txt.
• agent.delivery: imageVolume does not wait for the probe. It renders the image volume whatever the probe found, and workspace pods on nodes that cannot mount it do not start. Keep auto unless you know every node can.
• Core's Secret writes are confined by the admission guard. With security.admissionGuard.enabled off, nothing limits where core writes Secrets or RoleBindings.
• Restores are runbooks. A button that restores a backup into a new instance is not in this release.