Debug a failing service
Status texts, logs, health checks and the common causes of a service that does not start.
A service that does not answer has one of a few causes, and the Run panel and the logs usually name it. This guide goes from the status to the cause.
Read the status
Each service's row in the Run panel shows its status:
| Status | Meaning | Look at |
|---|---|---|
| Pending | No replica is scheduled yet | the team's quota, the cluster's capacity |
| Starting | Replicas start or run their first steps | wait; then the logs |
| Running | The last step runs and the health check passes | |
| Internal | Running, without routes or public ports | |
| Finished | The last step exited with 0, or a container's process did | the server should not exit; see below |
| Failed | A step failed, or the server exited 5 times within 2 minutes | the logs |
| Stopped | Its steps are not started | click Run or Start |
| Sleeping | Sleeps when idle, and nobody used it | a request wakes it |
The workspace list shows the same for the dev URL: "failed to start", "stopped", "asleep, wakes on open". The status texts page lists every text.
Read the logs
Click Logs on the service's row, or:
hart logs --service web -fhart logs follows the output live with -f. --replica picks one pod, --since 10m
narrows the time, --stage prepare shows the prepare stage. The last lines before a restart usually say why.
Common causes
Prepare did not run
The service's steps call a tool that prepare installs, and prepare never ran, or failed. The log says command not found. Run prepare (Run all, or hart prepare -f) until it succeeds, then Run again.
The server exits
The last step is the server: it must keep running. When it exits, Harthelm restarts it after a growing pause (1 s
up to 60 s) and marks it Failed after 5 exits within 2 minutes. If it exits with 0, the service shows Finished and is
not restarted. Typical mistakes: a last step that starts the server in the background (npm start &), or a build
script as the last step. Put builds before the server step, or in prepare.
It listens on the wrong port or address
The port in ports must be the one the program listens on, and the program must listen on all interfaces
(0.0.0.0), not only 127.0.0.1. Pass the port in env and read it in your app:
env:
PORT: 3000
ports:
- port: 3000
routes: [{path: /}]The health check never passes
With a health check, a replica gets traffic only once the check passes. Check
that health.port is one of the service's ports, that path exists, and that mode: http-2xx is not stricter than
your app's answer (a redirect is no 2xx). any-http, the default, takes any HTTP answer.
A changed file, variable or secret was not synced
Changes to harthelm.yml, env vars and secrets reach the services only at the next Sync. The Run panel says
"harthelm.yml changed since the last Sync" or "Environment variables or platform settings changed since the last Sync".
Sync, then Run.
A service it needs is failing
The row says "needs api, which is failing". The service runs, but its calls fail until api runs again. Debug api
first; see Services that need each other.
It does not fit
"The team's quota does not fit this" when you Sync, scale or wake: the team's awake workspaces and services use the quota. Put something to sleep, use a smaller plan, or ask a team admin. "There is no room to wake it right now" means the cluster is full; Harthelm asks again by itself.
The Sync is blocked
The Sync review lists blockers with the line and column in harthelm.yml: an image that cannot be pulled
(image-unavailable), a forward target the installation refuses, a provider that is not available, a full workspace
disk. Fix the cause and click Check again. Validation errors are explained under
harthelm.yml errors.
Try it by hand
Open a terminal in the workspace and start the server yourself, with the same command as the last step:
cd /data/app
PORT=3000 npm startIf it fails there too, the problem is in the app. If it works there but not as a service, compare the environment:
services get the workspace's env vars, their own env and their secrets, and run with bash -lc in /data/app.