Monitoring

Harthelm tells you how it is doing three ways: the doctor (a list of checks you run, in a terminal or on the console's Doctor page), the Capabilities page (what the cluster offers), and Prometheus metrics.

The doctorCopy Anchor Link

As a cluster admin:

hart admin doctor

It runs every check core can make from its own state and prints one line per check: ok, warn, fail or skip (a feature that is switched off). The exit status is 1 when a check failed, so it fits a cron job or a CI step. -o json gives the report as one document.

What each result means and what to do: Troubleshooting. The same report is GET /admin/doctor; see hart admin doctor.

The Doctor pageCopy Anchor Link

Since 1.5.0 the console runs the same checks under Platform → Doctor, between Capabilities and Licence. Only cluster admins see the page; anyone else who opens its address is told that only cluster admins can run the checks. It shows a table of the checks in the order of hart admin doctor, with the check's name, its status (ok, warn, fail, or skip, greyed out) and its detail; above it, the count per status, when the checks ran and how long that took, and the command that runs the same checks from a terminal.

Core runs every check each time the report is asked for, as with the command, so opening the page runs them, and Run again runs them once more (the last results stay visible, dimmed, until the new ones are in). The page never runs them on a timer or when you come back to the browser tab. Each check gets at most 10 seconds. There is no switch for the capability probe: what the cluster offers is on the Capabilities page, which the Doctor page links to.

CapabilitiesCopy Anchor Link

hart admin doctor --capabilities

adds what the cluster offers: the Kubernetes version and every optional feature with its state and, when it is not available, the reason. The console shows the same under Platform → Capabilities.

Features that need a missing component are greyed out in the console instead of failing. The probe runs at core's start and hourly.

MetricsCopy Anchor Link

Core and the routers serve Prometheus metrics on port 9090 at /metrics. Let your scraper in with networkPolicy.metricsNamespaceSelector:

networkPolicy:
  metricsNamespaceSelector: {kubernetes.io/metadata.name: monitoring}

The families:

The bundled database's exporter (CloudNativePG) listens on 9187; extra queries go in database.bundled.monitoring.customQueries.

Alerts to start withCopy Anchor Link

groups:
  - name: harthelm
    rules:
      - alert: HarthelmBackupFailed
        expr: harthelm_backup_last_run_failed == 1
      - alert: HarthelmBackupMissed
        expr: time() - harthelm_backup_last_success_timestamp_seconds{kind="platform"} > 26 * 3600

Run hart admin doctor on a schedule as well, and alert on its exit status: it covers what metrics do not (certificate expiry, the identity provider, the mail relay).