Stuck-job detection has no reaping path — alerts re-raise forever and cancel is a no-op on orphaned jobs #56

New Issue

2026-08-22T13:28:52+01:00

bobby commented

2026-08-22 13:28:52 +01:00

Summary

#41's detector correctly identifies jobs stuck in running, but there is no way for an operator to clear them. Resolving the alert does not change the job row, so the detector re-raises on its next sweep; and POST /api/jobs/{id}/cancel cannot help because it dispatches to an agent that no longer has the job.

The result is an alert count permanently pinned at a non-zero number, with the only remedy being direct SQL against the server database.

Observed

A fleet accumulated 144 jobs in running dating back three months, from mixed causes (an agent panic since fixed, and older network failures reaching the restic REST store). After #41 shipped, all of them surfaced as job_stuck alerts at once.

Resolving them via the UI route succeeds, and the count drops to zero:

resolved: 144
OpenAlerts: 0

About one minute later, 143 alerts reappear — same underlying job IDs, new CreatedAt, and LastSeenAt advancing on each sweep. That is correct behaviour for the detector: those jobs are still running, so they are still stuck. But it means alert resolution is a no-op against a persistent condition.

Attempting the obvious remedy:

curl -X POST https://<server>/api/jobs/<job-id>/cancel
# -> 202

The job remains status: running indefinitely. Cancellation is delivered to the owning agent, and for a job orphaned months ago there is no agent-side process to cancel — the agent has restarted many times since. The 202 is misleading: it reports acceptance of a request that can never complete.

Why this matters

The alert list becomes unusable. A permanent floor of 143 warnings means a genuinely new stuck job is invisible. This is the same signal-to-noise failure #41 was built to solve.
The only fix is DB surgery. Marking the rows cancelled directly resolved it here, but that is not something most operators should be doing on a backup orchestrator's database, and it is not documented anywhere.
It is self-inflicted on upgrade. Any deployment with historical orphaned jobs gets a large alert spike the moment #41 lands, with no supported way to clear it.

Suggested fix

Give stuck jobs a terminal state. Options, roughly in order of preference:

Reap automatically. When the detector finds a running job whose started_at predates the owning agent's current connection (or whose host has since reconnected/restarted), transition it to cancelled/abandoned with a reason. An agent that has reconnected demonstrably is not still running that job, which makes this safe to decide server-side without agent cooperation.
Add an explicit operator action — POST /api/jobs/{id}/abandon, plus a bulk variant or a UI control on the alerts page, that sets the terminal state server-side without involving the agent.
Make cancel fall back. If the target agent does not acknowledge within a timeout, or is not currently connected, mark the job terminal server-side rather than leaving it running after a 202.

Option 1 alone would have prevented the spike entirely, since every one of the 144 jobs belonged to a host that had reconnected many times since.

A migration that reaps pre-existing orphans on upgrade would also be worth considering, so existing deployments do not inherit the backlog.

Note on the alert semantics

Worth deciding explicitly whether job_stuck alerts should be auto-resolved when the underlying job reaches a terminal state. Currently they must be resolved manually even after the cause is gone — after the DB fix above, the 143 alerts still had to be resolved by hand before the count returned to zero and stayed there.

Environment

Server v1.2.1, agent v1.2.1
144 affected jobs spanning 2026-05-09 to 2026-08-22
Related: #41 (the detector), #37 (one cause of the orphans)

## Summary #41's detector correctly identifies jobs stuck in `running`, but there is **no way for an operator to clear them**. Resolving the alert does not change the job row, so the detector re-raises on its next sweep; and `POST /api/jobs/{id}/cancel` cannot help because it dispatches to an agent that no longer has the job. The result is an alert count permanently pinned at a non-zero number, with the only remedy being direct SQL against the server database. ## Observed A fleet accumulated **144 jobs in `running`** dating back three months, from mixed causes (an agent panic since fixed, and older network failures reaching the restic REST store). After #41 shipped, all of them surfaced as `job_stuck` alerts at once. Resolving them via the UI route succeeds, and the count drops to zero: ``` resolved: 144 OpenAlerts: 0 ``` **About one minute later, 143 alerts reappear** — same underlying job IDs, new `CreatedAt`, and `LastSeenAt` advancing on each sweep. That is correct behaviour for the detector: those jobs are still `running`, so they are still stuck. But it means alert resolution is a no-op against a persistent condition. Attempting the obvious remedy: ```bash curl -X POST https://<server>/api/jobs/<job-id>/cancel # -> 202 ``` The job remains `status: running` indefinitely. Cancellation is delivered to the owning agent, and for a job orphaned months ago there is no agent-side process to cancel — the agent has restarted many times since. The 202 is misleading: it reports acceptance of a request that can never complete. ## Why this matters - **The alert list becomes unusable.** A permanent floor of 143 warnings means a genuinely new stuck job is invisible. This is the same signal-to-noise failure #41 was built to solve. - **The only fix is DB surgery.** Marking the rows `cancelled` directly resolved it here, but that is not something most operators should be doing on a backup orchestrator's database, and it is not documented anywhere. - **It is self-inflicted on upgrade.** Any deployment with historical orphaned jobs gets a large alert spike the moment #41 lands, with no supported way to clear it. ## Suggested fix Give stuck jobs a terminal state. Options, roughly in order of preference: 1. **Reap automatically.** When the detector finds a `running` job whose `started_at` predates the owning agent's current connection (or whose host has since reconnected/restarted), transition it to `cancelled`/`abandoned` with a reason. An agent that has reconnected demonstrably is not still running that job, which makes this safe to decide server-side without agent cooperation. 2. **Add an explicit operator action** — `POST /api/jobs/{id}/abandon`, plus a bulk variant or a UI control on the alerts page, that sets the terminal state server-side without involving the agent. 3. **Make `cancel` fall back.** If the target agent does not acknowledge within a timeout, or is not currently connected, mark the job terminal server-side rather than leaving it `running` after a 202. Option 1 alone would have prevented the spike entirely, since every one of the 144 jobs belonged to a host that had reconnected many times since. A migration that reaps pre-existing orphans on upgrade would also be worth considering, so existing deployments do not inherit the backlog. ## Note on the alert semantics Worth deciding explicitly whether `job_stuck` alerts should be auto-resolved when the underlying job reaches a terminal state. Currently they must be resolved manually even after the cause is gone — after the DB fix above, the 143 alerts still had to be resolved by hand before the count returned to zero and stayed there. ## Environment - Server `v1.2.1`, agent `v1.2.1` - 144 affected jobs spanning 2026-05-09 to 2026-08-22 - Related: #41 (the detector), #37 (one cause of the orphans)

steve referenced a pull request that will close this issue

2026-08-22 13:42:50 +01:00

Fix source-group API, backup validation, and orphaned jobs #57

steve closed this issue

2026-08-22 13:45:04 +01:00

steve referenced this issue

2026-08-22 13:45:39 +01:00

Release v1.2.2 — API and orphaned-job recovery #58

Sign in to join this conversation.