Backups and restore

All platform state is in the database. Everything core created in Kubernetes is derived from it and comes back by itself after a restore. Harthelm writes its backups to one volume in the release namespace; copy that volume to your own backup system.

What is backed upCopy Anchor Link

backup:
  enabled: true
  schedule: "30 2 * * *"
  timeZone: Europe/Berlin
  persistence:
    storageClass: ""
    size: 20Gi
  workspaces:
    enabled: true
    at: "01:00"

The backup volume (backup.persistence, or your own claim in backup.persistence.existingClaim) is kept when the release is deleted. Its layout:

platform/platform-<stamp>.dump
platform/pre-migration-<stamp>.dump
ms/<team id>/<object name>-<id>/<object name>-<id>-<stamp>.dump
workspaces/<team id>/<workspace id>/<time>.tar.gz

A failed run leaves no file and keeps every older one.

Workspace archivesCopy Anchor Link

An archive is taken only while a workspace's IDE and every service sleep, and only if it woke since its last archive. A workspace that never sleeps is never archived; the console shows the age of its last archive. If a workspace wakes during its archive, the run is abandoned and the next sleep brings the next try. At most backup.workspaces.parallel (2) archives run at once.

Pick backup.workspaces.at for a time when the nodes holding the volumes are on and the workspaces sleep. On a cluster that powers its workspace nodes off at night, that is the morning after they are back.

On the first of each month the receiver reads up to 5 of the newest archives end to end (checksum, gzip, tar) to catch damage on the volume.

The master keyCopy Anchor Link

The master key encrypts every stored Git token, secret and sealed value. A restored database is useless without the key it was written under, and core refuses to start with a database set up under another key.

Export it once and keep it with your secrets:

kubectl -n harthelm-system get secret harthelm-master-key -o yaml > harthelm-master-key.yaml

The Secret is <release>-master-key, or the one masterKey.existingSecret names. The signing keys (<release>-signing-key) are regenerated if lost; users then sign in to their workspaces again.

Restore the platform databaseCopy Anchor Link

You need the dump, the master key Secret and your values. Run these steps in the release namespace.

kubectl -n harthelm-system scale deployment harthelm-core --replicas=0

kubectl -n harthelm-system get cronjob harthelm-backup -o json | jq '{apiVersion: "v1", kind: "Pod",
    metadata: {name: "restore-files", namespace: .metadata.namespace, labels: .spec.jobTemplate.spec.template.metadata.labels},
    spec: (.spec.jobTemplate.spec.template.spec | .restartPolicy = "Never"
      | .containers = [.containers[0] | del(.args) | .command = ["sleep", "3600"]])}' | kubectl apply -f -
kubectl -n harthelm-system wait --for=condition=Ready pod/restore-files
kubectl -n harthelm-system exec restore-files -- ls -lt /backups/platform/

kubectl -n harthelm-system exec -i restore-files -- sh -c 'cat > /backups/platform/platform-<stamp>.dump' < platform-<stamp>.dump

kubectl -n harthelm-system exec restore-files -- sh -c 'f=/backups/platform/platform-<stamp>.dump
  pg_restore --list "$f" | grep -v -E "^[0-9]+; [0-9]+ [0-9]+ (EXTENSION|COMMENT - EXTENSION) " > /tmp/toc &&
  pg_restore --use-list=/tmp/toc --no-owner --role=harthelm --exit-on-error --dbname="$HART_DATABASE_URL" "$f"'
kubectl -n harthelm-system delete pod restore-files

kubectl -n harthelm-system scale deployment harthelm-core --replicas=1

Not in the dump, and lost with the cluster: the workspace volumes (a workspace whose volume is gone gets a new, empty one and clones its repository again; unpushed work comes back only from an archive) and the data of managed services.

Managed servicesCopy Anchor Link

Each instance's backups are in its own directory, ms/<team id>/<object name>-<id>/: the instance id is in GET /managed-services/{ms}; its first 12 hex digits without dashes end the directory's name. What a file holds depends on the provider:

An S3 instance larger than backup.managed.s3.maxBytes (100 GiB by default) is not copied: its dump is recorded as failed. With in-cluster TLS the backups reach Valkey, RabbitMQ, MariaDB and S3 Since 1.4.0 over TLS and verify them with the internal CA.

What the dump's log line saysCopy Anchor Link

The success line of a dump, managed dump written (and managed dump written with a warning), says how the dump connected. Since 1.5.0 it carries three fields:

• tls: the transport the dumper actually used, true for an encrypted connection. pg_dump always connects with sslmode=require and mariadb-dump always with --ssl, so both say true whether or not the instance serves in-cluster TLS; Valkey, RabbitMQ and S3 say true exactly while it does.
• tls_verify: server when the dumper checks the server's certificate, none when it does not or the connection is plain. PostgreSQL says none (the database's own CA is not in the backup pods); MariaDB says server; Valkey checks the chain, not the name.
• port: where the dumper connects (the TLS port while the instance serves TLS).

A PostgreSQL dump logs tls=true tls_verify=none port=5432, a MariaDB dump tls=true tls_verify=server port=3306, a Valkey dump with in-cluster TLS tls=true tls_verify=server port=6380 and without tls=false tls_verify=none port=6379. The S3 line, S3 snapshot written, carries tls and port. Where a line names the target itself, the flag is internal_tls (whether the instance serves in-cluster TLS, the instance's detail tls): in 1.4 it was called tls and read false for an encrypted PostgreSQL dump.

A service that is still startingCopy Anchor Link

A new instance reports its host and port once its server runs, seconds to minutes after it was created. Since 1.5.0 the nightly run skips an instance that has not reported them yet: it logs one warning, writes no backup record for it, and does not fail. In 1.4 a service created shortly before the run failed the whole backup Job, and its retry, although every other dump was written. The line names the instance, its driver, its status and the time since it was created:

level=WARN msg="managed service not ready yet; skipped" service=<id> driver=seaweedfs status=creating age=1m2s

The next night dumps it. The skip looks only at the missing host and port, not at the status, so an instance that never reports them is never dumped and never fails a run: the doctor's managed-backups check warns about it once it has had no dump for 26 hours ((never) when it never had one), so a stuck instance shows up the day after. What still fails:

• an instance that reports where it listens and whose dump fails, as before;
• a dump on demand (Dump now) of an instance that has not reported where it listens, with the instance has not reported where it listens yet, because you asked for that dump;
• the nightly MariaDB dump, a Job per instance in the server's own image after the backup run, fails the same way for an instance without a host and port; it is never started for an instance created after that night's run began.

S3 snapshots and the backup volumeCopy Anchor Link

An S3 snapshot is a manifest, one line per object, plus the objects' contents in blobs/, which all snapshots of the instance share: a night copies only what changed. The retention prunes manifests like every other dump; after it, a cleanup removes the blobs that no manifest references any more. A manifest that a restore is reading is kept until the restore ends Since 1.4.0.

Core does not mount the backup volume, so the backup role records it at the end of every managed run (the nightly one, a dump on demand): its size and free space, the bytes of the S3 blobs, and when the blob cleanup last went through every S3 directory or why it could not. Two doctor checks read that record Since 1.4.0:

Both say skip while there is no S3 snapshot, and after an upgrade from 1.3 until the first managed run of 1.4 has recorded the volume; they warn when no run did within 48 hours of the upgrade. A manifest the cleanup cannot read stops the cleanup of its directory: remove it, or restore it from your own copy of the volume. A directory it cannot list needs its owner back: group 0, mode 2775.

Size the backup volume for about the S3 data of all instances times one plus five weeks of changes, plus 1.2 times the data free. The first snapshot of a 50 GiB store takes 10 to 20 minutes, later nights only the changes. Give backup.resources more than its 256 MiB for an instance above about two million objects.

A restore loads a backup into a running instance of the same provider, usually a new one, as that instance's user. If you do not have its password, rotate it first (hart ms rotate).

PostgreSQL, into an empty database, from a pod in the team namespace:

pg_restore --no-owner --role=<user> \
  --dbname="postgresql://<user>@<object name>.rg-<team id>.svc:5432/<database>" \
  /backups/ms/<team id>/<object name>-<id>/<object name>-<id>-<stamp>.dump

MariaDB, from a pod in the team namespace; the DEFINER clauses are dropped because the target's user may differ from the dump's:

gunzip -c /backups/ms/<team id>/<object name>-<id>/<object name>-<id>-<stamp>.sql.gz |
  sed -E 's/DEFINER=`[^`]+`@`[^`]+`//g' |
  MYSQL_PWD='<password>' mariadb --host=<object name>.rg-<team id>.svc --user=<user> <database>

Valkey: the snapshot is loaded into a scratch server, and its keys are moved into the target instance; keys of the same name are overwritten. RabbitMQ: the definitions are posted to the target's management API; queues come back empty, and the producers send their messages again. S3 object storage: the backup role itself puts the objects of one manifest into buckets of the target that exist and are empty. Each of these runs in a pod or Job made from the backup CronJob's template, which mounts the backup volume read-only, with the runbooks of your release (below).

A dump on demand: POST /managed-services/{ms}/dump.

The restore runbooksCopy Anchor Link

Every release attaches the restore runbooks Since 1.4.0 as harthelm-restore-<version>.tar.gz, with its checksum harthelm-restore-<version>.tar.gz.sha256, next to the chart, images.txt and mirror.sh. Take the archive of the release your installation runs (hart version names it): its scripts make their pods and Jobs from that release's backup CronJob. You need kubectl, jq and access to the release namespace.

v=1.4.0
sha256sum -c "harthelm-restore-$v.tar.gz.sha256"      # macOS: shasum -a 256 -c
tar -xzf "harthelm-restore-$v.tar.gz" && cd "harthelm-restore-$v"

The directory holds a README.md and, per service, a script and the jq program that makes its pod or Job. Run the steps below in it. First find the backup: the instance's id is in GET /managed-services/{ms} (a deleted one's with include_deleted=true on the team's list), and its first 12 hex digits without dashes end the directory's name. Then create the target instance and know its user and password (rotate first if you do not have it; a bound instance's password is in its workspace's vault). The examples use the release harthelm in harthelm-system; <object> is the target instance's object in the team namespace rg-<team id>.

Valkey. valkey-restore.sh runs in a pod of the target's own image: it loads the .rdb into a scratch server and moves every key into the target. A target with in-cluster TLS gets its keys over TLS on 6380, checked against the internal CA; without TLS the script uses 6379 and says so.

ns=harthelm-system release=harthelm team=<team id> object=<object>
dump=ms/<team id>/<object name>-<id>/<object name>-<id>-<stamp>.rdb       # below /backups
image=$(kubectl -n "rg-$team" get statefulset "$object" -o jsonpath='{.spec.template.spec.containers[0].image}')
tls=$(kubectl -n "rg-$team" get statefulset "$object" -o jsonpath='{.spec.template.metadata.annotations.harthelm\.dev/tls}')
kubectl -n "$ns" create configmap valkey-restore --from-file=valkey-restore.sh
kubectl -n "$ns" create secret generic valkey-restore --from-literal=VALKEY_USER=app --from-literal=VALKEY_PASSWORD='<password>'
kubectl -n "$ns" get cronjob "$release-backup" -o json |
  jq -f valkey-restore-pod.jq --arg image "$image" --arg dump "$dump" --arg host "$object.rg-$team.svc" --arg tls "$tls" |
  kubectl -n "$ns" apply -f -
kubectl -n "$ns" logs -f pod/valkey-restore
kubectl -n "$ns" delete pod/valkey-restore configmap/valkey-restore secret/valkey-restore

The pod needs memory for the dump's data; set resources on its container for a large one.

MariaDB. mariadb-restore.sh runs in a pod of the target's own image, checks the file, drops the DEFINER clauses and loads it as the target's user, over TLS where the target has it:

ns=harthelm-system release=harthelm team=<team id> object=<object> database=<the target's database>
dump=ms/<team id>/<object name>-<id>/<object name>-<id>-<stamp>.sql.gz    # below /backups
image=$(kubectl -n "rg-$team" get statefulset "$object" -o jsonpath='{.spec.template.spec.containers[0].image}')
kubectl -n "$ns" create configmap mariadb-restore --from-file=mariadb-restore.sh
kubectl -n "$ns" create secret generic mariadb-restore --from-literal=MARIADB_USER=<user> --from-literal=MARIADB_PASSWORD='<password>' \
  --from-literal=MARIADB_SSL_CA=/etc/harthelm/internal-ca/ca.crt   # only when the target's details say tls: true
kubectl -n "$ns" get cronjob "$release-backup" -o json |
  jq -f mariadb-restore-pod.jq --arg image "$image" --arg dump "$dump" --arg host "$object.rg-$team.svc" --arg database "$database" |
  kubectl -n "$ns" apply -f -
kubectl -n "$ns" logs -f pod/mariadb-restore
kubectl -n "$ns" delete pod/mariadb-restore configmap/mariadb-restore secret/mariadb-restore

RabbitMQ. Copy the .json off the backup volume, then post it through a port-forward with rabbitmq-restore.sh. By default it adds only what the target lacks and leaves its users, queues and policies as they are (RABBITMQ_RESTORE_EXISTING=overwrite resets them to the dump). It posts everything but the bindings first, posts queues the broker's import left out again (up to five times), and then the bindings; a queue that is still missing ends the script with an error that names it.

ns=harthelm-system release=harthelm team=<team id> object=<object>
dump=ms/<team id>/<object name>-<id>/<object name>-<id>-<stamp>.json      # below /backups
kubectl -n "$ns" get cronjob "$release-backup" -o json | jq -f rabbitmq-files-pod.jq | kubectl -n "$ns" apply -f -
kubectl -n "$ns" wait --for=condition=Ready pod/rabbitmq-restore-files
kubectl -n "$ns" exec rabbitmq-restore-files -- cat "/backups/$dump" > restore.json
kubectl -n "$ns" delete pod rabbitmq-restore-files
kubectl -n "rg-$team" port-forward "svc/$object" 15672:15672 &
RABBITMQ_USER=app RABBITMQ_PASSWORD='<password>' ./rabbitmq-restore.sh restore.json http://localhost:15672

S3 object storage. Create the target with every bucket of the snapshot in its buckets (the manifest's header lists them) and wait for Synchronized; the restore refuses a bucket that is missing or not empty. Raise bucketQuotaMiB first for a large snapshot. s3-restore.sh starts a Job of the backup role, waits for it and prints its log; the Job holds its snapshot while it reads it, so the nightly retention keeps it, and you need not suspend the backup CronJob. An optional third argument restores only the buckets it lists.

NAMESPACE=harthelm-system CRONJOB=harthelm-backup \
  ./s3-restore.sh <target instance id> ms/<team id>/<object name>-<id>/<object name>-<id>-<stamp>.manifest.zst [data,assets]

Exit status 0 means every object of the restored buckets is in the target. If the script stops waiting, the Job goes on and keeps its hold; kubectl -n harthelm-system delete job <job> stops it. After a failed restore, empty the target's buckets or delete the target, and start again.

Workspace archivesCopy Anchor Link

An archive is restored into a new workspace, never over an existing one. In the console or with POST /archives/{archive}/restore: the new workspace starts asleep, the archive is unpacked into its volume, and it wakes when that succeeded. Variables, secrets and the landscape are not in an archive: Sync again. The restore a workspace guide is the user's side.

Monitor the backupsCopy Anchor Link

Core exports per kind and target:

Alert on a failed run, and on a platform dump older than 26 hours:

- alert: HarthelmBackupFailed
  expr: harthelm_backup_last_run_failed == 1
- alert: HarthelmBackupMissed
  expr: time() - harthelm_backup_last_success_timestamp_seconds{kind="platform"} > 26 * 3600

Practise the restoreCopy Anchor Link

Restore the newest dump into a scratch database once in a while, and restore one workspace archive into a new workspace, open it and compare git status with the original.