steve/restic-manager
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases 5 Wiki Activity

Detect and alert on stuck jobs #46

Merged
steve merged 2 commits from fix-issue-41-stuck-job-alerts into main 2026-08-22 10:56:32 +01:00
Conversation 0 Commits 2 Files Changed 9
+253 -3
steve commented 2026-08-22 10:52:20 +01:00
Owner
Copy Link
Copy Source

Fixes #41.

  • detects running jobs with no persisted activity beyond kind-appropriate thresholds
  • raises one deduplicated job_stuck alert per job and resolves it on terminal state or reconciliation
  • includes start and last-activity timestamps in the alert
  • exposes stuck-job count and oldest inactivity age as Prometheus gauges
  • covers detection, recent activity, deduplication, recovery, resolution, and rendering
Fixes #41. - detects running jobs with no persisted activity beyond kind-appropriate thresholds - raises one deduplicated job_stuck alert per job and resolves it on terminal state or reconciliation - includes start and last-activity timestamps in the alert - exposes stuck-job count and oldest inactivity age as Prometheus gauges - covers detection, recent activity, deduplication, recovery, resolution, and rendering
steve added 1 commit 2026-08-22 10:52:20 +01:00
Detect and alert on stuck jobs
CI / Test (rest) (pull_request) Successful in 39s
Details
CI / Test (store) (pull_request) Successful in 40s
Details
CI / Lint (pull_request) Failing after 11s
Details
CI / Build (linux/amd64) (pull_request) Successful in 8s
Details
CI / Build (linux/arm64) (pull_request) Successful in 7s
Details
CI / Build (windows/amd64) (pull_request) Successful in 25s
Details
CI / Test (server-http) (pull_request) Successful in 1m39s
Details
e2e / Playwright vs docker-compose (pull_request) Successful in 1m16s
Details
2110c7dab4
steve added 1 commit 2026-08-22 10:53:55 +01:00
Close websocket handshake response in test
CI / Test (server-http) (pull_request) Successful in 5s
Details
CI / Test (rest) (pull_request) Successful in 23s
Details
CI / Lint (pull_request) Successful in 10s
Details
CI / Build (windows/amd64) (pull_request) Successful in 7s
Details
CI / Test (store) (pull_request) Successful in 41s
Details
CI / Build (linux/amd64) (pull_request) Successful in 8s
Details
CI / Build (linux/arm64) (pull_request) Successful in 6s
Details
e2e / Playwright vs docker-compose (pull_request) Successful in 1m24s
Details
750a06384b
steve merged commit 8fdf4a1bdf into main 2026-08-22 10:56:32 +01:00
steve deleted branch fix-issue-41-stuck-job-alerts 2026-08-22 10:56:32 +01:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
enhancement
New feature or improvement to existing functionality
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
steve
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: steve/restic-manager#46
Delete Branch "fix-issue-41-stuck-job-alerts"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?