Status texts
Every state and status text of workspaces, services, stages, managed services and domains, what it means and what to do.
Harthelm shows a short state ("Running", "Sleeping") and often a status text next to it. This page lists them in the
words the console, hart and the API use, what each one means and what you can do about it. Search the page for the
text you see.
The API returns the same states in machine form: a workspace's state and status_detail
(GET /api/v1/workspaces/{ws}), a service's status
(Get the landscape). Errors that the API answers with are on
Problem types; errors in harthelm.yml are on
harthelm.yml errors.
Workspace states
The state is in the workspace list and in hart ws list; the status text is in the workspace's status bar and in
status_detail.
| State | Status text | What it means | What to do |
|---|---|---|---|
| Creating | Configuring storage | The volume is being created and bound, or the pod has not started yet. | Wait. |
| Creating | Waiting for a node: … | No node can take the workspace pod yet. The text after the colon is the scheduler's reason, for example too little CPU or memory. | Wait a little; if it stays, ask your administrator. The cluster is short of room or the plan is larger than any node. |
| Creating | Starting | The agent is being installed into the pod. | Wait. |
| Creating | Preparing the volume | An empty workspace's volume is being set up. | Wait. |
| Cloning | Cloning repository | The repository is being cloned into /data/app. | Wait; a large repository takes longer. |
| Creating, Cloning | Starting IDE | The editor is starting. | Wait. |
| Running | Workspace is running | The editor is ready. | Nothing. |
| Sleeping | Asleep | The workspace sleeps: the editor and services have no pods, the files are kept. | Open it or click Wake workspace. |
| Sleeping | Asleep after being unused | Nobody used it for its idle timeout. | Open it to wake it. |
| Sleeping | Asleep while its node is drained | Its node was taken out of service for the night. | Open it to wake it on another node, or wait for the morning if the console says it sleeps until a time. |
| Sleeping | Asleep since its pod was evicted | Its node was drained. | Open it to wake it. |
| Sleeping | Restoring the archive | A workspace restored from an archive sleeps until its volume is filled. | Wait; it can be woken once the restore is done. |
| Waking up | Waking up | A wake is under way. | Wait; it usually takes a few seconds. |
| Stopped | Changing the owner of the volume | The files are being handed to another user ID after an administrator changed it. | Wait; the workspace starts again afterwards. |
| Error | Cloning the repository failed: … | The clone failed or keeps failing. The text names the reason, for example an image that cannot be pulled or an exit code. | Check that the repository and branch exist and that you still have access to them; then delete the workspace and create it again. |
| Error | Preparing the volume failed: … | The volume of an empty workspace could not be set up. | Ask your administrator. |
| Error | Installing the agent failed: … | The workspace pod could not start its agent. | Ask your administrator; this is an installation problem. |
| Error | The IDE does not start: … | The editor container cannot start or keeps crashing. "the image … cannot be pulled" names the image; "exit code N, retrying" means it crashed. | If you chose a custom base image, check it; otherwise ask your administrator. The pod keeps retrying and the state follows. |
| Deleting | The workspace was deleted. It stays in Recently deleted for 7 days and can be restored there. | Restore it, or Purge now to free its storage at once. |
An error state is not final: Kubernetes keeps retrying, and the state changes as soon as the pod starts.
"Upgrade pending" next to a workspace or a service means it runs on the pod template from before the last platform upgrade. It switches over the next time it sleeps, stops or restarts. You need to do nothing.
Sleep and wake
When you open a sleeping workspace or one of its addresses, Harthelm wakes it. The console and the waiting page on the workspace's address say why it slept and, if the wake cannot start, why not.
| Text | What it means | What to do |
|---|---|---|
| It went to sleep because nobody used it. | Nobody used it for its idle timeout. | Nothing; it is waking. |
| It was put to sleep by hand. | Someone put it to sleep through the API (POST /api/v1/workspaces/{ws}/sleep). | Nothing; it is waking. |
| It went to sleep when its node was taken out of service for the night. | The installation turns some nodes off at night. | Nothing, or wait for the time the page names. |
| It went to sleep when its node was drained. | The node was drained for maintenance. | Nothing; it wakes on another node. |
| It is being restored from an archive. | A restore fills its volume. | Wait. |
| There is no room to wake it right now / No capacity right now | No node has room for the pods now. | Try again in a few minutes. The waiting page retries by itself. |
| This workspace sleeps until … / Asleep while its nodes are off | The nodes it must run on are off on a schedule until that time. | Wait until then, or ask your administrator. |
| Waking is paused | The installation has paused all wakes, usually to protect the cluster. | Try again later. |
| This workspace is being worked on / Busy with this workspace | Harthelm is working on its volume, for example a restore. | Try again in a moment. |
| This workspace is stopped | It does not run until it is started. | Start it from the console. |
| This is taking longer than usual. The platform may be short of room; the workspace keeps trying. | The wake takes longer than expected. | Wait; the page keeps asking. |
| Still waking up | The address held your request, but the workspace did not come up in time. | Reload in a few seconds. |
A request that is not a browser page (an API call, curl) gets status 503 with a Retry-After header and the reason
as text. See Sleep and wake.
Service statuses
The Run panel and hart run show one status per service of the landscape.
| Status | API value | What it means | What to do |
|---|---|---|---|
| Pending | pending | No replica is scheduled yet. | Wait. If it stays, the team quota or the cluster may be full. |
| Starting | starting | Replicas start or run their first steps. | Wait. |
| Running | running | The last step runs and the health check passes. | Nothing. |
| Internal | internal | Running, for a service without routes or public ports: it is reachable inside the workspace only. | Nothing. |
| Finished | finished | The last step exited with code 0, or a container's process did. | Nothing, if the service is a job. A server that finishes has stopped serving: check its command. |
| Failed | failure | A step failed, or the last step failed 5 times within 2 minutes. | Open the service's logs (Logs, or hart logs), fix the cause and press Run again. See Debug a failing service. |
| Stopped | stopped | Its run switch is off. A request never starts its steps; if the night drain put it to sleep, Run or Start wakes and starts it. | Start it with Run or Start. |
| Sleeping | It sleeps when idle and nobody used it; the next request wakes it. | Nothing, or Wake. |
More texts next to a service:
| Text | What it means |
|---|---|
| needs api, which is failing | A service it needs (needs in harthelm.yml) is failing. Requests still reach this service; it answers with its own error until the service it needs runs again. |
| no route, reachable inside the workspace only | The service has no route and no public port. |
| no pods, the router forwards | A forward service: requests go to its target, nothing runs. |
| runs as UID … | The user ID the container runs as. |
| asleep, N replicas come back when woken | The service sleeps; the replicas start on the next request. |
| Managed service: … | A managed service; the status after the colon is listed under Managed services. |
Lines in a service's log
The log of a service carries lines from Harthelm itself (stream system) between the output of your steps:
| Line | What it means | What to do |
|---|---|---|
| Waiting for api (the services this one needs). | Before its first step, a replica waits up to 5 minutes for the services it needs (their health check, else their first port). | Nothing. |
| api answered; starting. | The services it needs are up. | Nothing. |
| api did not answer within 5m0s; starting anyway. | A service it needs did not come up in time. | Check that service. |
| Not waiting for api any more: it is failing. | The needed service failed while this one waited. | Fix the needed service. |
| Step exited with code N. | A step before the last one failed; the run fails. | Read the output above the line. |
| Step exited with N, restarting in 4s. | The last step (the server) exited; Harthelm restarts it with a delay that doubles from 1 second up to 1 minute. | Read the output above the line. |
| Giving up: exited with N; 5 failures in 2m0s. | The last step failed 5 times within 2 minutes; the service is now Failed. | Fix the cause and press Run. |
| The service finished (exit code 0). | The last step ended without error. | Nothing, if that is intended. |
| Cannot start the step: … | The shell could not start. | Read the reason. |
| Cannot start node: no such file, or its interpreter or a library it needs is missing. | A container's command is not in the image (exit code 127). | Check command and the image. |
| Cannot start node: permission denied. UID … (group 0) may not execute it. | The file is not executable for the workspace's user (exit code 126). | Make it readable and executable for group 0 in the image. |
| The image runs as …, but workspace pods run every process as one UID of group 0 … | The image expects a fixed user that the pod does not run as. | Make the image work for any user ID in group 0. See Run a container image. |
A failure message of a container service can end in a note in parentheses about the image's user; it has the same cause as the last row.
Dev URL states
The workspace list shows what the dev URL (https://<id>-3000.<workspace domain>) answers with right now:
| Text | What it means |
|---|---|
| (no text) | The service on / runs; the URL answers at once. |
| asleep, wakes on open (~21 s) | It sleeps; opening the URL wakes it (and the services it needs) in about that time. |
| waking up … | A wake is under way. |
| starting … | Its run switch is on and no replica is ready yet. |
| stopped | Its run switch is off, or the landscape is torn down. The URL answers "This service is stopped". |
| failed to start | Its last run failed. |
When no service serves /, the URL answers "Nothing is served here yet".
Landscape and Sync
The banner at the top of the Run panel:
| Text | What it means | What to do |
|---|---|---|
| This workspace has no landscape yet. Review harthelm.yml and Sync to create the services. | No Sync has run. | Click Sync, or hart sync. |
| In sync with harthelm.yml | The services match the file. | Nothing. |
| harthelm.yml changed since the last Sync. The running services do not match it. | The file was edited. | Review the changes and Sync. |
| harthelm.yml was edited, but it describes the same landscape as the last Sync (a comment or formatting). | Only formatting changed. | Nothing. |
| Environment variables or platform settings changed since the last Sync. The running services do not use them yet. | A workspace variable, secret or an installation setting changed. | Sync to apply it. |
| The landscape is deployed from harthelm.yml, but harthelm.staging.yml is selected. … | You selected another profile. | Sync to switch. |
| harthelm.yml has errors | The file does not validate. | Fix the listed errors; see harthelm.yml errors. |
| harthelm.yml could not be compared with the last Sync. … | The workspace could not read the file. | Click Check again. |
| The workspace is not running, so harthelm.yml cannot be compared with the last Sync. | The editor sleeps; it reads the file. | Wake the workspace. |
| The landscape is torn down. Sync builds it again from harthelm.yml. | Someone used Teardown. Code, variables and secrets are kept. | Sync to build it again. |
| db is paused. A Sync resumes it with its data. | A managed service of a torn-down landscape keeps its data without a server. | Sync. |
| Applying changes to the cluster… | A Sync runs. | Wait. |
Plan warnings and blockers shown before a Sync are listed on harthelm.yml errors.
Stage runs and steps
Prepare, test and the start of run show one state per run and per step:
| State | API value | What it means |
|---|---|---|
| Waiting | waiting | Queued; an earlier step or run comes first. |
| Running | running | Runs now. |
| Succeeded | success | Every step exited with 0. |
| Failed | failure | A step exited with another code (shown as "exit N"); the steps after it are skipped. |
| Stopped | aborted | Someone stopped it, or the workspace went to sleep. |
| Skipped | skipped | A step after a failed one. |
"Prepare has not run yet." means prepare has never run in this workspace. Click Run all or use
hart prepare.
Managed service statuses
| Status | API value | What it means | What to do |
|---|---|---|---|
| Creating, Updating, Pausing, Resuming, Deleting | creating, updating, pausing, unpausing, deleting | The provider works on a change. | Wait. |
| Synchronized | synchronized | It runs as asked. | Nothing. |
| Paused | paused | The data is kept, no server runs (the database of a torn-down landscape). | Sync the landscape to resume it. |
| Deleted | deleted | Deleted; it is listed under recently deleted for 7 days. | Nothing. |
| Unknown | unknown | The provider does not answer or does not report it; the connection details are the last known ones. | Wait; if it stays, ask your administrator. |
| Invalid provider | invalid-provider | Its provider version is gone or no longer accepts its settings. | Ask your administrator. |
Custom domain statuses
Verification (Verification status):
| Status | What it means | What to do |
|---|---|---|
| Not checked yet | Nobody has checked the domain. | Add the TXT record _harthelm.<your domain> at your DNS provider, then click Check status. |
| Checking | A check is queued or running. | Wait. |
| Verified | The TXT record was found; the domain serves. | Nothing. |
| Failed | The last check found no TXT record. The reason says what was looked up. | Check the record at your DNS provider; DNS changes can take a while to spread. |
| Lapsed | A verified domain whose TXT record was missing for 7 days in a row (by default). It serves nothing until a check finds the record again. | Put the TXT record back and check again. |
An unverified domain keeps its name for 7 days by default and is then removed.
Certificate (State):
| Status | What it means |
|---|---|
| Being issued | The domain is verified and the certificate is being issued, or the domain is not verified yet. |
| Issued | The certificate is ready. |
| Failed | Issuing failed; the reason says why. |
| Covered | A wildcard certificate of the installation covers the name. |
| Uploaded | You manage the certificate yourself and uploaded it. |
| Not managed | The installation does not create edge objects; the administrator routes the domain. |
| Waits for verification | Nothing is issued before the domain is verified. |
See Custom domains.
Pages on a workspace address
The router answers on the workspace addresses (the IDE, the dev URL, public ports and custom domains). A browser gets a page; other clients get the status and the title as one line of text.
| Status | Title | What it means | What to do |
|---|---|---|---|
| 404 | Nothing here | The address belongs to no workspace, or the workspace no longer exists. | Check the address. |
| 404 | Nothing is served here yet | No service listens on this port, or no service serves / on the dev URL. | Open the workspace, press Run, then reload. |
| 404 | Nothing is served at this path | No route of the landscape (or of the domain) matches the path. | Check routes in harthelm.yml, or the domain's routes. |
| 503 | This service is stopped | The service's run switch is off. If the night drain put it to sleep, opening the address brings its pods back, but it stays stopped. | Run the landscape, then reload. |
| 503 | Workspace is not running | The workspace is stopped. | Start it from the console. |
| 503 | Router is starting | The router has no routes yet. | Try again in a few seconds. |
| 502 | Workspace did not answer | The service runs but did not answer. It may still be starting. | Reload in a moment; if it stays, read the service's logs. |
| 504 | The site did not answer in time | The service took longer than the domain's timeout. | Make the request faster, or raise the domain's timeout. |
| 502 | This forward target is not allowed | The installation's network rules forbid the address a forward target leads to. | Use another target. |
| 502 | The forward target did not answer | The forward target could not be reached. | Check target in harthelm.yml. |
| 401 | Sign-in required | A private address needs a sign-in. | Open it from the console, or in a browser tab. |
| 401 | Your browser blocked the workspace cookie | The browser does not keep the sign-in inside the console's frame. | Open the workspace in a new tab from the console. |
| 401 | Your browser did not keep the workspace cookie | Cookies are blocked for the address. | Allow cookies for the site, then open it again from the console. |
| 403 | This link has expired | Sign-in links work once and for a short time. | Open the workspace again from the console. |
| 403 | Request from another site refused | A change came from a page outside the workspace and the console. | Send the request from the workspace's own pages. |
| 503 | Sign-in is unavailable | The platform could not confirm the sign-in. | Try again in a moment. |
| 413 | Request too large | The body is larger than the domain allows (100 MiB by default). | Send less, or raise the domain's limit. |
| 414 | Address too long | The URL is too long for the router. | Shorten it. |
| 400 | Bad request | The router cannot handle the request. | Check the request. |
The waiting pages of a sleeping workspace are under Sleep and wake.
Pages on a managed service's host
Since 1.4.0 RabbitMQ brokers and S3 stores have a host ms-<n>.<workspace domain>
(Managed services › From a browser). Its pages lead
Back to the console to the service's details. Besides the sign-in pages above:
| Status | Title | What it means | What to do |
|---|---|---|---|
| 200 | S3 endpoint | The root of an S3 store's host, opened in a browser without a signature. Objects are at /<bucket>/<key>. | Open an object's presigned URL, or a public bucket's object. |
| 401 | Sign-in required | The host is behind the team's sign-in, and the request carries neither its cookie nor an API key of the team. A fetch from another site gets this too. | Open the service from the console, or send an API key in Harthelm-Authorization. |
| 503 | This managed service is paused | The service is paused; opening its host does not resume it. | Resume it in the console, or sync its landscape. |
| 503 | This service is starting | No server of the service is ready right now (a restart, a Rotate). | Reload in a few seconds. |
| 502 | The managed service did not answer | It runs but did not answer. | Reload in a moment. |
| 404 | Nothing here | No managed service has this number, or it is deleted. | Check the address. |
Errors in the console
| Text | What it means | What to do |
|---|---|---|
| Harthelm could not be reached. Try again. | The console lost the connection to the API. | Check your network and try again. |
| Your role does not allow this. | Your role in the workspace or team does not include the action. | Ask a team admin for another role. See Invite members. |
| This changed in the meantime. Reload and try again. | Someone else changed the same thing first. | Reload, then repeat your change. |
| The team's quota does not fit this. Team quota exceeded: needs …, … left. | The team's quota for CPU, memory or storage is used up. | Stop or delete another workspace, or ask an administrator for a larger quota. See Plans and quotas. |
| The workspace is over its disk quota. Free some space first. | The files use more than the workspace's volume size; Sync and prepare are refused. The console warns from 90 % on. | Delete files, for example build output and caches. |
| The file or the settings changed while you were reviewing. Review the changes again. | The plan you reviewed is out of date. | Review again and Sync. |
| Some secrets still have no value. | The file references secrets that are not set. | Click Enter secrets. See Secrets. |
| The landscape cannot be synced yet: | The plan has blockers, listed below the text. | See Sync blockers. |
| The landscape is not synced. Sync it first. | Run or a service start before the first Sync, or after a teardown. | Sync. |
| This stage is already running. | A run of prepare or test is active. | Wait for it, or stop it. |
| A managed service has no steps to start or stop. … | Managed services are run from the Managed services page. | Use that page. |
| Wake the workspace first: its IDE reads harthelm.yml. | The editor sleeps. | Wake the workspace. |
| A workspace named … already exists. Rename or delete it, then restore this one. | A restore would reuse a taken name. | Rename or delete the other workspace. |
Each message of the API is a problem type; hart prints the same text and exits with the
code listed in hart › Exit codes.