steve/restic-manager
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases 5 Wiki Activity

Detect and alert on jobs stuck in running state #41

New Issue
Closed
opened 2026-08-22 10:03:10 +01:00 by steve · 0 comments
steve commented 2026-08-22 10:03:10 +01:00
Owner
Copy Link
Copy Source

Summary

Jobs can remain in running indefinitely when an agent exits, disconnects, or fails to deliver job.finished. Current alerting reacts to explicit failed results but does not detect jobs that never reach any terminal state.

This was exposed while investigating the stale snapshot projection in #40: the dashboard showed no open alert even though forget jobs were permanently stuck and their follow-up projection refresh never arrived. Projection freshness remains tracked in #40; this issue covers job-lifecycle monitoring only.

Desired behaviour

  • Detect running jobs that exceed a configurable or kind-appropriate threshold.
  • Mark or clearly surface them as stale/stuck without confusing them with actively progressing jobs.
  • Raise a deduplicated alert that identifies the host, job, kind, start time, and last observed activity.
  • Resolve the alert when the job reaches a terminal state or an operator explicitly acknowledges/cancels it.
  • Avoid false positives for legitimately long-running backups by considering recent progress/log activity and job kind.

Design considerations

  • Agent disconnect alone should not immediately fail a job because transient reconnects are expected.
  • Detection should run server-side so it still works when the agent process has died.
  • Define recovery semantics for jobs orphaned across agent/server restarts.
  • Add metrics for stuck-job count and age alongside notification coverage.

Acceptance criteria

  • A job with no terminal result and no recent activity beyond the chosen threshold is surfaced as stuck.
  • A deduplicated alert is emitted and visible through the existing alert paths.
  • Normal long-running jobs with recent progress are not flagged.
  • Tests cover detection, deduplication, recovery, and alert resolution.

Related: #40.

## Summary Jobs can remain in `running` indefinitely when an agent exits, disconnects, or fails to deliver `job.finished`. Current alerting reacts to explicit failed results but does not detect jobs that never reach any terminal state. This was exposed while investigating the stale snapshot projection in #40: the dashboard showed no open alert even though forget jobs were permanently stuck and their follow-up projection refresh never arrived. Projection freshness remains tracked in #40; this issue covers job-lifecycle monitoring only. ## Desired behaviour - Detect `running` jobs that exceed a configurable or kind-appropriate threshold. - Mark or clearly surface them as stale/stuck without confusing them with actively progressing jobs. - Raise a deduplicated alert that identifies the host, job, kind, start time, and last observed activity. - Resolve the alert when the job reaches a terminal state or an operator explicitly acknowledges/cancels it. - Avoid false positives for legitimately long-running backups by considering recent progress/log activity and job kind. ## Design considerations - Agent disconnect alone should not immediately fail a job because transient reconnects are expected. - Detection should run server-side so it still works when the agent process has died. - Define recovery semantics for jobs orphaned across agent/server restarts. - Add metrics for stuck-job count and age alongside notification coverage. ## Acceptance criteria - A job with no terminal result and no recent activity beyond the chosen threshold is surfaced as stuck. - A deduplicated alert is emitted and visible through the existing alert paths. - Normal long-running jobs with recent progress are not flagged. - Tests cover detection, deduplication, recovery, and alert resolution. Related: #40.
steve closed this issue 2026-08-22 10:56:32 +01:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
enhancement
New feature or improvement to existing functionality
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
steve
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: steve/restic-manager#41
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?