Snapshot projection never refreshes after forget — counts are stale until the next backup #40

New Issue

2026-08-22T09:54:47+01:00

bobby commented

2026-08-22 09:54:47 +01:00

Summary

GET /api/hosts/{id}/snapshots (and hosts.snapshot_count) is a projection refreshed only when the agent sends snapshots.report. The intended design already refreshes after forget — RunForget calls reportSnapshots at internal/agent/runner/runner.go:290 — but in practice that call never runs, so retention changes are invisible until the next backup happens to refresh the projection.

The practical effect: a host whose retention is working perfectly can display a months-old snapshot count, and there is no way to tell the difference between "forget is broken" and "the projection is stale". This cost a full diagnostic session.

Evidence — the refresh never fires after forget

Every host's refreshed_at equals its backup time. Not one matches its forget time:

host	backup	forget	`refreshed_at`	matches
host-C	01:30	02:15	`2026-08-22T01:30:02`	backup
host-E	02:20	03:00	`2026-08-22T02:20:09`	backup
host-B	03:10	03:00	`2026-08-22T03:10:03`	backup
host-D	03:31	03:00	`2026-08-22T03:31:08`	backup
host-A	03:00	04:00	`2026-08-22T08:39:10`	ad-hoc backup

Root cause — interacts with #37

In RunForget:

err := env.RunForget(ctx, groups, r.streamHandler(jobID, &seq))
finishedAt := time.Now().UTC()
r.sendFinished(ctx, jobID, finishedAt, err, nil)   // agent panics here (#37)

if err == nil {
    if rerr := r.reportSnapshots(ctx, env); rerr != nil {   // line 290 — never reached
        ...
    }
}

The nil-pointer panic in #37 kills the process during/just after sendFinished, so execution never reaches reportSnapshots. Fixing #37 will likely fix this too — but the projection is still fragile by design, because a single missed message leaves it stale indefinitely with no indication.

Observed impact

On host-A the API reported 102 snapshots while the repository actually held 12. The stale count persisted for hours and only corrected when an unrelated ad-hoc backup forced a refresh. During that window the host looked like it had never applied retention since May.

Because #37 also leaves forget jobs stuck at status: running (never failed), no alert fires either — so the dashboard shows OpenAlerts: 0, a stale count, and a job that never ends.

Suggested changes

Roughly in order of value for effort:

Make the refresh crash-proof. Move reportSnapshots before sendFinished, or run it in a defer, so a failure on the reporting path cannot skip the projection update. (Fixing #37 addresses the immediate cause; this makes it robust to the next one.)
Surface staleness in the API, don't just expose it. refreshed_at is already returned — add a derived stale: true (or refreshed_age_seconds) when it is older than the host's last completed job, so clients can distinguish "12 snapshots" from "12 snapshots as of three days ago". The UI can then badge it.
Refresh after any repo-mutating job, not just backup/forget — prune also rewrites the index.
Add an explicit refresh trigger, e.g. POST /api/hosts/{id}/snapshots/refresh, so an operator can reconcile without running a backup. Today the only way to force a refresh is to mutate the repo, which is a poor diagnostic tool.
Alert on jobs stuck in running beyond a threshold. Currently only failed alerts, so a hung job is silent — this is what let the stale projection persist unnoticed.

Items 2 and 5 are the ones that would have made this self-diagnosing.

Environment

Agent v1.1.0 (commit 0f5110f3d9b91b269684453e6b8d14dbcffb93c6)
restic 0.18.1, protocol_version 1
Related: #37 (agent panic on forget), #36 (manual forget dispatch)

## Summary `GET /api/hosts/{id}/snapshots` (and `hosts.snapshot_count`) is a **projection refreshed only when the agent sends `snapshots.report`**. The intended design already refreshes after forget — `RunForget` calls `reportSnapshots` at `internal/agent/runner/runner.go:290` — but **in practice that call never runs**, so retention changes are invisible until the *next backup* happens to refresh the projection. The practical effect: a host whose retention is working perfectly can display a months-old snapshot count, and there is no way to tell the difference between "forget is broken" and "the projection is stale". This cost a full diagnostic session. ## Evidence — the refresh never fires after forget Every host's `refreshed_at` equals its **backup** time. Not one matches its **forget** time: | host | backup | forget | `refreshed_at` | matches | |---|---|---|---|---| | host-C | 01:30 | 02:15 | `2026-08-22T01:30:02` | backup | | host-E | 02:20 | 03:00 | `2026-08-22T02:20:09` | backup | | host-B | 03:10 | 03:00 | `2026-08-22T03:10:03` | backup | | host-D | 03:31 | 03:00 | `2026-08-22T03:31:08` | backup | | host-A | 03:00 | 04:00 | `2026-08-22T08:39:10` | ad-hoc backup | ## Root cause — interacts with #37 In `RunForget`: ```go err := env.RunForget(ctx, groups, r.streamHandler(jobID, &seq)) finishedAt := time.Now().UTC() r.sendFinished(ctx, jobID, finishedAt, err, nil) // agent panics here (#37) if err == nil { if rerr := r.reportSnapshots(ctx, env); rerr != nil { // line 290 — never reached ... } } ``` The nil-pointer panic in #37 kills the process during/just after `sendFinished`, so execution never reaches `reportSnapshots`. Fixing #37 will likely fix this too — but the projection is still fragile by design, because a single missed message leaves it stale indefinitely with no indication. ## Observed impact On `host-A` the API reported **102 snapshots** while the repository actually held **12**. The stale count persisted for hours and only corrected when an unrelated ad-hoc backup forced a refresh. During that window the host looked like it had never applied retention since May. Because #37 also leaves forget jobs stuck at `status: running` (never `failed`), **no alert fires either** — so the dashboard shows `OpenAlerts: 0`, a stale count, and a job that never ends. ## Suggested changes Roughly in order of value for effort: 1. **Make the refresh crash-proof.** Move `reportSnapshots` before `sendFinished`, or run it in a `defer`, so a failure on the reporting path cannot skip the projection update. (Fixing #37 addresses the immediate cause; this makes it robust to the next one.) 2. **Surface staleness in the API, don't just expose it.** `refreshed_at` is already returned — add a derived `stale: true` (or `refreshed_age_seconds`) when it is older than the host's last completed job, so clients can distinguish "12 snapshots" from "12 snapshots as of three days ago". The UI can then badge it. 3. **Refresh after any repo-mutating job**, not just backup/forget — `prune` also rewrites the index. 4. **Add an explicit refresh trigger**, e.g. `POST /api/hosts/{id}/snapshots/refresh`, so an operator can reconcile without running a backup. Today the only way to force a refresh is to mutate the repo, which is a poor diagnostic tool. 5. **Alert on jobs stuck in `running`** beyond a threshold. Currently only `failed` alerts, so a hung job is silent — this is what let the stale projection persist unnoticed. Items 2 and 5 are the ones that would have made this self-diagnosing. ## Environment - Agent `v1.1.0` (commit `0f5110f3d9b91b269684453e6b8d14dbcffb93c6`) - restic `0.18.1`, protocol_version 1 - Related: #37 (agent panic on forget), #36 (manual forget dispatch)

steve referenced this issue

2026-08-22 10:03:10 +01:00

Detect and alert on jobs stuck in running state #41

steve commented

2026-08-22 10:03:19 +01:00

Scope note after reviewing the current implementation: I suggest keeping this issue focused on snapshot-projection freshness and reconciliation. The stuck-job detection/alerting concern has been split into #41 so it can be designed and verified independently.

Why narrow it this way:

The immediate missed refresh is strongly related to the #37 failure on deployed v1.1.0 agents, but projection delivery remains best-effort after that fix.
Moving reportSnapshots before sendFinished, or into a defer, would not make delivery crash-proof: both use the connection-scoped context and sender, so neither can report over a lost connection.
Prune removes unreferenced data but does not change the snapshot list; its existing repository-statistics refresh is the relevant operation.
Reliable staleness reporting needs projection metadata stored independently of snapshot rows, because an empty projection currently has nowhere to retain refreshed_at.

A focused resolution here would therefore track host-level snapshot refresh time (including zero-snapshot reports), expose freshness/staleness, and provide a reconciliation path such as explicit or periodic refresh. Server-side detection and alerting for orphaned running jobs is now tracked in #41.

I have left the original title and description unchanged.

Scope note after reviewing the current implementation: I suggest keeping this issue focused on **snapshot-projection freshness and reconciliation**. The stuck-job detection/alerting concern has been split into #41 so it can be designed and verified independently. Why narrow it this way: - The immediate missed refresh is strongly related to the #37 failure on deployed v1.1.0 agents, but projection delivery remains best-effort after that fix. - Moving `reportSnapshots` before `sendFinished`, or into a `defer`, would not make delivery crash-proof: both use the connection-scoped context and sender, so neither can report over a lost connection. - Prune removes unreferenced data but does not change the snapshot list; its existing repository-statistics refresh is the relevant operation. - Reliable staleness reporting needs projection metadata stored independently of snapshot rows, because an empty projection currently has nowhere to retain `refreshed_at`. A focused resolution here would therefore track host-level snapshot refresh time (including zero-snapshot reports), expose freshness/staleness, and provide a reconciliation path such as explicit or periodic refresh. Server-side detection and alerting for orphaned `running` jobs is now tracked in #41. I have left the original title and description unchanged.

bobby referenced this issue

2026-08-22 10:36:58 +01:00

Server WebSocket read limit (32 KiB default) drops snapshots.report on hosts with many snapshots #44

steve referenced a pull request that will close this issue

2026-08-22 11:00:52 +01:00

Make snapshot projections fresh and self-diagnosing #47

steve closed this issue

2026-08-22 11:03:47 +01:00

Sign in to join this conversation.