Managed services

Managed services are databases and similar services that users create from a catalog, standalone or as part of a workspace's landscape. Managed services and the add a database guide are the user's side.

Harthelm has five built-in providers, all on by default: PostgreSQL, run by the CloudNativePG operator, and MariaDB, Valkey, RabbitMQ and S3 object storage Since 1.3.0, which core runs itself without an operator. Their images are pinned by tag and digest in the values: a Harthelm upgrade that pins a new patch restarts every instance of that provider once. Every instance is one pod without replicas; its pods may reach their team's namespace and DNS, not the internet.

The built-in PostgreSQLCopy Anchor Link

Each database is a CloudNativePG cluster of one instance in its team's namespace, PostgreSQL 17 or 18. It needs the CloudNativePG operator and rbac.cnpg; without the operator the console lists the provider as unavailable.

marketplace:
  builtin:
    postgres:
      storageClass: fast-rwo
      scheduling:
        nodeSelector: {example.com/pool: platform}

Databases count against their team's quota.

NetworkCopy Anchor Link

Core gives each database a NetworkPolicy: in from its team's pods and the backup pods on 5432, from the operator on 5432 and 8000, out to the API server. networkPolicy.cnpgNamespaceSelector must select the operator's namespace and networkPolicy.egress.apiServer must reach the API server; both apply even with networkPolicy.enabled off. On Cilium, core also renders a policy that lets the database pods reach the API server.

BackupsCopy Anchor Link

With backup.enabled, the nightly backup run also dumps every database whose backups setting is on (the default). A dump on demand starts with POST /managed-services/{ms}/dump. See Backups and restore.

Valkey, MariaDB, RabbitMQ and S3Copy Anchor Link

The four share one shape of values; replace valkey with mariadb, rabbitmq or s3:

marketplace:
  builtin:
    mariadb:
      storageClass: fast-rwo
      scheduling:
        nodeSelector: {example.com/pool: platform}
    rabbitmq:
      enabled: false

The pods run under the restricted Pod Security standard as security.defaultUid, with that number as their primary group and fsGroup. The Valkey, MariaDB and RabbitMQ images come from Docker Hub, which rate-limits anonymous pulls: give the nodes a login with global.imagePullSecrets or mirror the images (air-gapped installs).

hart admin doctor checks them with managed-images (each pinned digest resolves), managed-storage (the classes allow volume expansion), managed-backups (the backup pods reach every enabled provider, and every instance with backups on has a dump of the last 26 hours), managed-tls, and since 1.4.0 managed-storage-write (below) and the two S3 backup checks managed-s3-backup-space and managed-s3-blob-gc (Backups and restore).

The write probeCopy Anchor Link

A storage class can exist and grow volumes and still refuse writes from the managed services' pods, for example an NFS server with manage-gids that does not let the volume's group write. With marketplace.storageWriteProbe.enabled (the default) Since 1.4.0, core asks each class that Valkey, S3, MariaDB and RabbitMQ put new volumes on: a Job harthelm-write-probe-… in the release namespace starts a pod with exactly their security context and a fresh 64 MiB volume of the class, creates, writes, renames, reads and removes files on it, and is deleted with its volume within two minutes. Core probes when it becomes the leader (so new values or a new release are probed within a minute) and then what is due each hour: a class without a result, an hour after a failure, a day after a pass. On a class with reclaimPolicy: Retain, which keeps every probe's volume, it probes a passed class again only when the class or the security context changes. The Job runs core's own image and needs no rights core does not have.

The doctor's managed-storage-write shows the newest result per class: fail names the class, the providers on it, the step that failed, the pod's user and groups and the volume root's owner and mode; warn means a class was not probed yet or could not be judged (the image did not pull, the pod was not scheduled). Fix the class, or name another one in the provider's storageClass.

In-cluster TLSCopy Anchor Link

With internal.tls on, cert-manager installed and rbac.certManager, core asks the internal issuer for a certificate per Valkey, RabbitMQ, MariaDB and S3 Since 1.4.0 instance (a Certificate mstls-<id> in the release namespace) and mounts it into the instance. Valkey listens with TLS on 6380 as well, RabbitMQ on 5671 and 15671, S3 on 8443, MariaDB offers TLS on 3306; the plain ports stay. The nightly backups then always go over TLS, verified with the internal CA. A new instance waits up to two minutes for its certificate before it starts, then starts without TLS and says why. A renewal, by default 15 days before the end, restarts the server. Core removes the Certificate and its Secret with the instance. An S3 store of 1.3 gets its certificate after the upgrade to 1.4 and restarts once for it.

VersionsCopy Anchor Link

• MariaDB: users move an instance to a newer series in place (11.4, 11.8, 12.3), never back.
• Valkey: a later minor version is offered when a release adds one; a new major is a new instance.
• RabbitMQ: a release ships the current series and, while it is patched, the previous one. When a release drops a series, core moves each broker on it to the next series after a backup; the release notes say so. An instance more than one series behind stays paused and comes back by restoring its dump into a new instance.
• S3: each release pins one SeaweedFS release; every store restarts once at the upgrade, for about 20 to 35 seconds.

Browser hostsCopy Anchor Link

Every RabbitMQ broker and every S3 store gets its own host Since 1.4.0, ms-<n>.<workspace domain> (<n> the instance's number, never reused). It is one label under the workspace domain, so its wildcard DNS name, its wildcard certificate and the edge's route to the router cover it: nothing new is needed in DNS, certificates or the edge. The users' side: Managed services › From a browser.

marketplace:
  browserHosts:
    enabled: true       # false: no hosts at all (1.3's behaviour)
    allowPublic: true   # false: S3 hosts stay behind the sign-in whatever a team asks

How the router serves a host:

• RabbitMQ (the management UI and its HTTP API): always behind the team's sign-in, never public. The router dials the broker's management port, 15671 verified with the internal CA when the broker has in-cluster TLS (never the plain port then), adds frame-ancestors 'none' and accepts changing requests only from the host itself and the console.
• S3 (the S3 API, path-style): behind the sign-in by default, or public per store. On a public host the router also answers CORS for the team's verified custom domains (since 1.5.0, Managed services); the dev hosts' CORS stays a flag of the store. The router passes path and query unchanged with the host as Host, so signatures verify; under in-cluster TLS it dials 8443 with the internal CA. A public host has no sign-in at all: the store's own keys decide, and anonymous clients only read the objects of its public buckets. SeaweedFS is then reachable from the internet; forbid public hosts with allowPublic: false if you do not want that.
• A paused instance answers 503 and is not woken; one without a ready pod answers 503 "This service is starting".
• The sign-in is the one of private dev URLs: a cookie of the host, set through the console, or an API key of a team member in Harthelm-Authorization, which the router removes before the instance sees the request.

Single sign-on to RabbitMQ. With the hosts on, each broker trusts tokens that core signs for it (RabbitMQ's OAuth 2.0 plugin): Open management UI sends the browser through the console, which checks the session and the permission, mints a token for one hour and has the router post it to the broker's login. Each broker has its own ES256 key, derived from the master key and the revision of the broker's password; nothing new is stored, and a Rotate or a master key rotation changes the key. The token never appears in a URL, a log line or an audit row. Its rights (tags management and policymaker on the broker's virtual host) are less than the instance user's. The internal users come first, so AMQP clients, the backups and the login form with the broker's user and password work as before.

Every opening of a host through the console writes an audit row managed_service.open (the service, its host, whether single sign-on was used). The router reports the number of hosts as harthelm_router_managed_hosts and counts their requests with the kind managed.

REST provider backendsCopy Anchor Link

Cluster admins can publish further providers whose instances a backend of your own creates, through PUT /providers/{name}. Harthelm calls the backend over REST and polls it for status:

Backends are URLs only an admin can set. A REST backend backs up its own instances; Harthelm's backups never reach them.

Provider definitionsCopy Anchor Link

A provider is a YAML or JSON document in the format harthelm/provider-v2. The built-in PostgreSQL is one too, and marketplace.builtin.postgres.overrides patches it. A definition for a REST backend:

schema: harthelm/provider-v2
name: kv
version: "1"
displayName: Key-value store
description: A small key-value store per instance.
onePerTeam: true                  # at most one instance per team
fields:
  config:                         # what users set when they create an instance
    type: object
    properties:
      engine:
        type: string
        enum: ["7", "8"]
        default: "8"
        x-hart-change: minor-upgrade-only
    additionalProperties: false
  secrets:                        # values Harthelm generates or users enter, kept in the vault
    type: object
    properties:
      password: {type: string, format: password, minLength: 16}
    required: [password]
  details:                        # what the backend reports back: host, port, connection string
    type: object
    properties:
      host: {type: string}
      port: {type: integer}
resources:
  memory: {kind: memory-mib, title: Memory (MiB), minimum: 128, x-hart-change: grow-only}
plans:
  - {id: small, name: Small, default: true, resources: {memory: 256}}
backend:
  rest: {url: "http://kv-backend.platform.svc:8080/v1"}

• fields.config, fields.secrets and fields.details are JSON Schemas. The console builds the create dialog from config and secrets, and details are what services read as ${{ services.<name>.details.<field> }}.
• x-hart-change says how a value may change after the instance exists: fixed (never; on a secret it also means no rotation), grow-only (a number that can only get larger, such as storage) or minor-upgrade-only (a version string that keeps its major version). A change the rule forbids fails the Sync with update-constraint.
• onePerTeam: true allows one instance per team; a second one is refused with one-per-team.