Troubleshooting
The doctor checks one by one: what a failure means and what to do.
Start with the doctor; most problems show up there:
hart admin doctor --capabilities
kubectl -n harthelm-system get pods,jobsIn the console, a cluster admin finds the same checks under Platform → Doctor (since 1.5.0; see The Doctor page). The sections below go by the check's name, which is the same in both.
The install or upgrade stops
The preflight Job failed
The preflight Job checks the cluster against your values before anything changes, and fails with a list of what is missing. Read it:
kubectl -n harthelm-system logs job/harthelm-preflightIt checks the Kubernetes version, the CRDs your values need (CloudNativePG, cert-manager and the named issuers, the
Gateway API), every Secret and ConfigMap your values name with their keys, the StorageClasses and IngressClass, and that
the database answers. It also parses the settings core will read, so a size that the schema accepts but core refuses
fails here instead of crash-looping core. With license.existingSecret
set, it checks the licence in that Secret with the keys of the release it installs: a missing Secret or key, or a
licence the release does not accept, fails; one that has ended only gives a note. Fix what it lists and run the
install again.
The pre-upgrade dump failed
A failed dump stops the upgrade on purpose: no migration runs without a copy. Its log is the Job
harthelm-backup-pre-upgrade. Usual causes: the backup volume is full or not writable (set backup.fsGroup on a
class whose root is not group-writable), or the database does not answer.
The render fails
The schema is strict. The message names the key. Common ones: an unknown key (a typo, or a key a newer chart renamed),
an OIDC issuer without oidc.existingSecret, routing.edge: gatewayAPI without parentRefs, and a different
security.defaultUid while volumes exist.
The doctor's checks
database
fail: core cannot reach the database. Check the database pod (kubectl -n harthelm-system get cluster), its
NetworkPolicy, and for an external database networkPolicy.egress.database. warn: migrations are not applied;
core migrates at start, so restart core.
capabilities
warn: the probe has not run yet (it runs at start and hourly), it is stale (the job leader is not running), or ValidatingAdmissionPolicy is not served, so the admission guard is not in effect. Components the probe did not find are listed; what needs them is greyed out in the console.
volume-uid
warn: some workspace volumes belong to another UID than security.defaultUid, after a deliberate change. Re-own
them per team, see Storage.
custom-domains
fail: routing.edge is gatewayAPI but the HTTPRoute CRD is missing, or an issuer is configured without
cert-manager. warn: routing.edge is none (domains are not routed), or there is no issuer and no wildcard, so
domains serve only with an uploaded certificate. See Custom domains.
network-policy
fail: the CNI does not enforce NetworkPolicies, so teams are not isolated. Use a CNI that does. warn with
"endPort": policies are enforced but port ranges are not; workspaces then lose most of their outbound ports. warn
"has not run": run helm test harthelm -n harthelm-system. The check also warns about every
networkPolicy.egress rule without to, which opens its ports to every address, and names a rule without ports
either with "all ports": that one does not restrict egress at all. See
Network policies.
internal-tls
fail: the certificate is missing, expired, or not valid for core's Service names. Check
kubectl -n harthelm-system get certificate and cert-manager's logs. warn: it expires within 14 days; cert-manager
should have renewed it. skip: internal.tls is off.
fail: the relay does not answer. Check mail.smtp.host, the port, and networkPolicy.egress.smtp. Mails
wait in the queue meanwhile. See Mail.
oidc
fail: the issuer's discovery document cannot be read. Check oidc.issuer (with its trailing slash, as the
provider writes it), networkPolicy.egress.idp, and for a private CA oidc.caConfigMap. Until it is fixed, only
the break-glass admin can sign in.
Since 1.5.0 the detail names the hint that fits the cause, where 1.4 and before pointed at the CA for every failure:
a certificate error points to HART_OIDC_CA_FILE and oidc.caConfigMap; a DNS error points to the resolver
and networkPolicy.dns; a refused, unreachable or timed-out connection points to networkPolicy.egress.idp and,
for nodes without a load balancer, to the node-address trap described in
Without a load balancer. Any other failure gets no hint.
backups
fail: the last platform dump or the last restore check failed. warn: no dump in the last 26 hours, or workspace
archives failed in the last 24 hours. Look at the failed Job in the release namespace
(kubectl -n harthelm-system get jobs) and its log.
jobs
warn: background jobs used all their attempts and gave up in the last 24 hours. The message names them; the failed operation in the console or core's log says why.
routers
warn: no router is connected to this core replica. Check that the router pods run, their logs, and the NetworkPolicy to core's internal port. While no router is connected, workspace URLs and custom domains do not serve.
license
ok: the licence is valid. warn: it ends within 60 days, it is in its grace period (an installation upgraded from 1.0 without a licence included), a limit is exceeded, the licence names other console hosts, the cluster's clock is more than a day behind, or the licence is signed with a key only development builds accept. fail: the installation is restricted (no valid licence, or the grace period is over). The message says what to do; install a licence as in Licence. A restricted installation keeps running everything that exists. Without a licence in force the detail also names the installation id (since 1.5.0; see The installation id before the first sign-in).
pull-secrets
skip: global.imagePullSecrets names no
Secret. fail: a named Secret is missing in the release namespace or has the wrong type; the team namespaces get no
copy, so their pods pull without these credentials. warn: copies in team namespaces are missing or out of date;
the team reconcile writes them within 30 seconds of a change, at the latest at its next resync.
managed-images
fail: a pinned image of a managed service does not resolve in its registry: mirror it, or fix its digest in the values. warn: an image is not pinned by a digest, or could not be checked. skip: none of Valkey, S3, MariaDB and RabbitMQ is enabled.
managed-storage
warn: a storage class of the managed services does not exist, or does not allow volume expansion, so their storage cannot grow and the API refuses a plan that needs more. See Storage.
managed-backups
fail: the backup pods' NetworkPolicy does not allow a managed service's port (or its TLS port), or the dumps
cannot be read. warn: instances with backups on have no dump of the last 26 hours; the message names them, with
(never) for one that never had a dump. Look at the failed Job in the release namespace and its log. An instance that
never reports where it listens is skipped by the nightly run without failing it (since 1.5.0, a WARN line
managed service not ready yet; skipped), so this warning is where it shows up; see
A service that is still starting. skip: backup.enabled is off.
managed-tls
warn: cert-manager is not installed, or a managed service's certificate is not ready, was not renewed when due,
or expired; an instance without its certificate listens on its plain ports only. Check
kubectl -n harthelm-system get certificate and cert-manager's logs. skip: in-cluster TLS is off for managed
services.
managed-storage-write
Since 1.4.0. fail: a fresh volume of a class does not work for the managed services' pods, so new instances there
cannot start. The message names the class, the providers on it, the step that failed (create, write, rename and
so on), the pod's user and groups and the volume root's owner and mode. Typical cause: an NFS server with
manage-gids that does not let the volume's group write. Fix the class, or name another one in the provider's
storageClass. warn: a class was not probed yet (core probes within a minute of its start and then hourly), or
the probe could not judge it (its image did not pull, its pod was not scheduled); core tries again within the hour.
skip: the probe is off (marketplace.storageWriteProbe.enabled),
or none of Valkey, S3, MariaDB and RabbitMQ is enabled. See
Managed services › The write probe.
managed-s3-backup-space
Since 1.4.0. fail: less than a tenth of what the S3 snapshots reference is free on the backup volume: a few nights of changes fill it, and then every dump of the night fails. Grow the volume. warn: less than 1.2 times that is free, or the backup role's record of the volume is older than 48 hours, or no managed run recorded it since the upgrade to 1.4. skip: no S3 snapshot yet, or the first managed run since the upgrade has not ended. See Backups and restore.
managed-s3-blob-gc
Since 1.4.0. warn: the cleanup of unreferenced S3 backup data has not gone through every snapshot directory within
48 hours, never did, or failed in its last run; the message names the directory or the manifest. A manifest it cannot
read must be removed or restored from your own copy of the volume; a directory it cannot list needs group 0 and mode
2775 back. Until then unreferenced data stays on the backup volume. skip: as for managed-s3-backup-space.
Common problems
| Symptom | Likely cause |
|---|---|
Workspaces never start; the IDE's seed fails creating /data/app | The StorageClass's volume root is not writable by group 0. See Storage. |
prepare cannot download anything | DNS from team namespaces does not reach the DNS pods (teams.network.dns), or the CNI ignores endPort, or a mirror on a private address is not in teams.network.mirrors. |
| The console or workspace URLs time out | A load balancer sends straight to pod IPs, which the policies drop; or networkPolicy.edgeNamespaceSelector does not select the controller's namespace. |
| The IDE keeps reconnecting | The edge or a load balancer cuts idle WebSockets; raise its timeout. |
| Invitations are never accepted | The identity provider does not send email_verified: true. See Sign-in. |
| Nobody is cluster admin | The groups claim does not reach Harthelm, or oidc.adminGroup differs from the group's name in the token. Use the break-glass admin meanwhile. |
| Every user shares one sign-in rate limit; audit rows show one address | trustedProxies is empty or wrong. |
| A wake answers "no capacity" | The pods would not fit on any node. Add nodes, or see the team quota and wake.capacityGuard. |
| Every wake answers "paused" | wake.maxPerMinute is 0. |
| Core does not start after a restore | The master key Secret is not the one the database was written under. |
| Open management UI ends at RabbitMQ's login with "Not_Authorized" | The broker has not restarted yet since the upgrade to 1.4 or since browser hosts were switched on, so it does not know the single sign-on yet; core restarts each broker once for it. Meanwhile its user and password work. |
Requests to an S3 store's host fail with SignatureDoesNotMatch | The client signed Harthelm-Authorization along. Add that header after signing; see Managed services › S3 from browsers. |
fetch from a dev host to an S3 store's host gets 401 | The host is behind the sign-in, and cross-origin requests carry no cookie. Switch the store public, or use a forward route on the dev domain. |