Always-On vs intermittent host mode (laptops): suppress offline noise, catch up missed backups #31

2026-06-15T22:59:17+01:00

steve commented

2026-06-15 22:59:17 +01:00

Lets an operator mark a host as not always-on (laptop/workstation) so it stops generating offline-alert noise when it legitimately sleeps, shows a calm state, and auto-catches-up a backup it missed while away.

What it does

hosts.always_on column (migration 0024, default 1 → no change for the existing fleet; intermittent is opt-in).
Alerts: intermittent hosts never raise agent_offline; instead the previously-dead stale_schedule alert is wired up for them at a 7-day threshold (only when they have an enabled schedule), resolved on the next successful backup or on mode toggle.
Catch-up: ~60s after an intermittent host reconnects, a server-side scheduler dispatches a backup for any enabled schedule whose window elapsed while it slept (overdue = cron.Next(lastBackup) <= now), guarded against double-dispatch via a real queued/running-backup check.
Toggle: POST /hosts/{id}/mode (operator-band, audited host.mode_updated); clears open offline/staleness alerts so the next sweep re-settles.
UI: intermittent offline hosts render a grey asleep · … · will catch up on return state instead of red "offline"; the host-detail header groups tags + presence into boxed pills with click-to-edit, a 24x7/Free presence chip, and a simplified out of date agent chip.

Design + plan: docs/specs/2026-06-15-always-on-host-mode-design.md, docs/plans/2026-06-15-always-on-host-mode.md. Tracked as NS-08 in tasks.md.

Test plan

go test ./internal/store/ ./internal/alert/ green (always_on round-trip, offline suppression, staleness raise+resolve, mode-change resolve)
go test ./internal/server/http/ -run 'TestScheduleOverdue|TestRunCatchup|TestHostModeSaveToggle' green
go vet ./... clean
Manual: toggle a host to intermittent → 24x7 chip becomes Free, offline host shows 'asleep', reconnect triggers catch-up

Note: this branch does not include the stale-dated sparkline-test fix (that's in PR #30); CI here may show that pre-existing failure until #30 lands in main and this branch is updated.

Lets an operator mark a host as **not** always-on (laptop/workstation) so it stops generating offline-alert noise when it legitimately sleeps, shows a calm state, and auto-catches-up a backup it missed while away. ## What it does - **`hosts.always_on`** column (migration 0024, default 1 → no change for the existing fleet; intermittent is opt-in). - **Alerts:** intermittent hosts never raise `agent_offline`; instead the previously-dead `stale_schedule` alert is wired up for them at a 7-day threshold (only when they have an enabled schedule), resolved on the next successful backup or on mode toggle. - **Catch-up:** ~60s after an intermittent host reconnects, a server-side scheduler dispatches a backup for any enabled schedule whose window elapsed while it slept (overdue = `cron.Next(lastBackup) <= now`), guarded against double-dispatch via a real queued/running-backup check. - **Toggle:** `POST /hosts/{id}/mode` (operator-band, audited `host.mode_updated`); clears open offline/staleness alerts so the next sweep re-settles. - **UI:** intermittent offline hosts render a grey `asleep · … · will catch up on return` state instead of red "offline"; the host-detail header groups tags + presence into boxed pills with click-to-edit, a `24x7`/`Free` presence chip, and a simplified `out of date` agent chip. Design + plan: `docs/specs/2026-06-15-always-on-host-mode-design.md`, `docs/plans/2026-06-15-always-on-host-mode.md`. Tracked as NS-08 in tasks.md. ## Test plan - [ ] `go test ./internal/store/ ./internal/alert/` green (always_on round-trip, offline suppression, staleness raise+resolve, mode-change resolve) - [ ] `go test ./internal/server/http/ -run 'TestScheduleOverdue|TestRunCatchup|TestHostModeSaveToggle'` green - [ ] `go vet ./...` clean - [ ] Manual: toggle a host to intermittent → 24x7 chip becomes Free, offline host shows 'asleep', reconnect triggers catch-up > Note: this branch does not include the stale-dated sparkline-test fix (that's in PR #30); CI here may show that pre-existing failure until #30 lands in main and this branch is updated.

steve added 16 commits 2026-06-15 22:59:18 +01:00

docs(spec): always-on vs intermittent host mode design 0c3a0844e4

docs(spec): clarify staleness vs job-failure alerting for asleep hosts 261b83ec26

docs(plan): always-on vs intermittent host mode implementation plan 9d16e3f7e3

feat(store): add hosts.always_on flag (default on) ff65d39f25

fix(store): SetHostAlwaysOn returns ErrNotFound; test agent-token lookup path 4c9641b6ed

feat(catchup): scheduleOverdue helper for missed-window detection 7aaafceab5

feat(catchup): arm on hello, fire missed-window backups on tick 5c4e0275d9

refactor(catchup): drop dead nil-guard; document per-host baseline limitation e408de9610

feat(alert): suppress offline + add staleness alert for intermittent hosts 25c55e5e4d

refactor(alert): refresh stale_schedule docs; log tick schedule errors; add mode-change + never-backed-up tests 9e6524788f

feat(http): host mode toggle handler + route (host.mode_updated) 1a07fbb217