Metrics
The Prometheus metrics of core, the router, the jobs, scheduling and backups.
Core and the router serve Prometheus metrics at /metrics on port 9090 (the container port metrics). Scrapers
need a NetworkPolicy rule to reach it: set
networkPolicy.metricsNamespaceSelector to the namespace of your
Prometheus. How to scrape and which alerts to set is on Monitoring.
Besides the metrics below, both serve the Go runtime and process metrics of the Prometheus client (go_*,
process_*). Every core replica serves its own counters; a few gauges are published by the replica that holds the
job leader lease, as noted.
Backups
Computed at scrape time from the platform database, at most every 30 seconds, because the backup Jobs do not live long enough to be scraped.
| Metric | Type | Labels | What it says |
|---|---|---|---|
harthelm_backup_last_success_timestamp_seconds | gauge | kind, target | Unix time of the last successful backup, 0 when there never was one. |
harthelm_backup_last_run_failed | gauge | kind, target | 1 when the latest finished backup run failed. |
harthelm_backup_last_size_bytes | gauge | kind, target | Size of the latest finished backup. |
target is platform for the platform database, managed for all managed-database dumps together, workspaces for
all workspace archives together, and restore-check for the monthly restore check of the archives.
Alert on a failed run, and on a missed one:
- alert: HarthelmBackupFailed
expr: harthelm_backup_last_run_failed == 1
- alert: HarthelmBackupMissed
expr: harthelm_backup_last_success_timestamp_seconds > 0 and time() - harthelm_backup_last_success_timestamp_seconds > 26 * 3600Sleep, wake and scheduling
| Metric | Type | Labels | What it says |
|---|---|---|---|
harthelm_core_sleeps_total | counter | kind, reason | Targets put to sleep, by kind (ide, service) and reason (idle, manual, cordon, eviction). |
harthelm_core_idle_sleeps_total | counter | kind | Targets the idle sweep put to sleep, by kind. |
harthelm_scheduling_sleeps_total | counter | reason | Targets put to sleep by the night drain, by reason (cordon, eviction). |
harthelm_scheduling_always_on_wakes_total | counter | Always-running targets the morning's wake woke after the night drain. | |
harthelm_core_agent_heartbeats_total | counter | Heartbeats received from workspace agents. |
Router
| Metric | Type | Labels | What it says |
|---|---|---|---|
harthelm_router_requests_total | counter | kind, code | Requests by host kind (ide, dev, domain, managed since 1.4.0, other) and status code. |
harthelm_router_gate_total | counter | endpoint, result | Sign-in gate decisions by endpoint and result. |
harthelm_router_upgraded_connections | gauge | Open WebSocket connections. | |
harthelm_router_wake_calls_total | counter | result | Wake calls to core by answer (the wake state, or error). |
harthelm_router_held_requests_total | counter | outcome | Requests to a sleeping or unready target by outcome (ready, rerouted, waiting-page, timeout, refused, stopped, canceled, asleep-again). |
harthelm_router_wake_seconds | histogram | target | Time from the first wake call for a sleeping target until the router sees it ready, by kind of target (ide, service). Refused wakes are not counted. |
harthelm_router_dial_retries_total | counter | Connections to a workspace tried again after a refused or unreachable dial (a just-woken target, a rolling service). | |
harthelm_router_forward_failures_total | counter | result | forward requests that did not reach their target. |
harthelm_router_activity_flushes_total | counter | result | Activity reports sent to core by result (ok, retry, dropped). |
harthelm_router_route_version | gauge | Version of the route table the router serves. | |
harthelm_router_workspaces | gauge | Workspaces in the route table. | |
harthelm_router_domains | gauge | Custom domains in the route table. | |
harthelm_router_managed_hosts | gauge | Browser hosts of managed services in the route table. Since 1.4.0. | |
harthelm_router_stream_reconnects_total | counter | Route stream sessions to core that ended and were reopened. A few during a rollout of core are normal. | |
harthelm_router_replay_cache_evictions_total | counter | Unexpired sign-in tokens dropped because the replay cache was full. Should stay 0. |
harthelm_router_route_version should follow harthelm_core_route_version within seconds; a router that stays
behind serves stale routes.
Core
| Metric | Type | Labels | What it says |
|---|---|---|---|
harthelm_core_route_version | gauge | Version of the route table core streams to routers. | |
harthelm_core_route_streams | gauge | Routers connected to this replica. | |
harthelm_core_agent_channels | gauge | Open editor and terminal channels this replica relays to workspace agents. | |
harthelm_core_token_redeems_total | counter | result | Sign-in tokens of workspace hosts redeemed, by result. |
harthelm_core_internal_plain_requests_total | counter | Requests on core's plain internal port while in-cluster TLS is on: pods started before it was turned on. Falls to 0 once they restarted. | |
harthelm_mail_messages_total | counter | template, result | Mail delivery attempts by template and result (sent, retry, failed). The licence reminders are the templates license_expiring, license_grace and license_restricted. |
Licence
Computed at scrape time from the licence module's state Since 1.1.0; see Licence.
| Metric | Type | Labels | What it says |
|---|---|---|---|
harthelm_license_state | gauge | state | 1 for the state the installation is in (valid, expiring, grace, restricted), 0 for the others. |
harthelm_license_expiry_timestamp_seconds | gauge | Unix time at which the term of the licence in force ends. | |
harthelm_license_usage | gauge | limit | What the licence's limits count now: vcpus, users, workspaces, nodes. |
harthelm_license_limit | gauge | limit | The licence's limit; absent for a limit the licence does not set. |
Alert before anyone sees a banner, for example 45 days before the end:
- alert: HarthelmLicenseEndsSoon
expr: harthelm_license_expiry_timestamp_seconds - time() < 45 * 86400Jobs
| Metric | Type | Labels | What it says |
|---|---|---|---|
harthelm_jobs_finished_total | counter | kind, outcome | Job runs on this replica by outcome: done, retry, dead, or lost (the lock expired first). |
harthelm_jobs_duration_seconds | histogram | kind | Run time of jobs on this replica. |
harthelm_jobs_running | gauge | kind | Jobs running on this replica. |
harthelm_jobs_queue | gauge | kind, state | Unfinished and dead jobs in the queue by state. Published by the leader. |
harthelm_jobs_rescued_total | counter | Running jobs whose worker stopped renewing the lock and that the leader released. | |
harthelm_jobs_leader | gauge | 1 while this replica holds the job leader lease. Exactly one replica should. |
Dead jobs (harthelm_jobs_queue{state="dead"}) are also listed by the jobs check of
hart admin doctor.
Reconciler
| Metric | Type | Labels | What it says |
|---|---|---|---|
harthelm_reconcile_writes_total | counter | op, kind | Kubernetes API writes by operation (apply, delete) and kind. A steady cluster makes none. |
harthelm_reconcile_unchanged_total | counter | kind | Rendered objects skipped because the cluster already matched. |
harthelm_reconcile_orphans | gauge | Objects Harthelm manages that belong to no workspace, team or service, as of the last cleanup run. | |
harthelm_reconcile_fenced_total | counter | Reconciles skipped because a newer core runs (during an upgrade). |
Git proxy
| Metric | Type | Labels | What it says |
|---|---|---|---|
harthelm_gitproxy_requests_total | counter | service, code | Git requests through core's proxy by Git service and status. |
harthelm_gitproxy_body_bytes_total | counter | direction | Bytes through the proxy; up is towards the Git provider. |
harthelm_gitproxy_upstream_failures_total | counter | reason | Provider answers the proxy refused to pass on, and transport errors. |
harthelm_gitproxy_inflight_requests | gauge | Requests in flight. |
Databases
With the bundled database, CloudNativePG's exporter serves its PostgreSQL metrics on port 9187; add your own queries
with the chart's database values.