Expose fleet-wide agent updates with host subset selection #43

New Issue

2026-08-22T10:32:20+01:00

steve commented

2026-08-22 10:32:20 +01:00

Problem

Admins can trigger an agent update from an individual host page, but there is no discoverable UI workflow for updating agents across the fleet or selecting a controlled subset. This makes routine releases unnecessarily repetitive and makes staged rollouts difficult.

There is already substantial backend support that should be reused:

POST /api/fleet/update starts the rolling fleet-update worker.
An empty host_ids list derives all online, out-of-date hosts.
An explicit host_ids list already represents a subset.
/settings/fleet-update already shows rollout progress, cancellation, and the previous result.
The worker updates sequentially, verifies the reported target version, and halts on first failure.

The primary gaps are discoverability and host selection: the fleet-update page is not linked from the Settings navigation, and its start form always submits the complete eligible set.

Proposed solution

Turn the existing page into a first-class admin Agent updates workflow under Settings.

Entry points

Add an Agent updates tab or clearly labelled card in Settings linking to /settings/fleet-update.
Link the dashboard’s “hosts behind” summary to this workflow, preserving any active dashboard filters where practical.
Keep the per-host update action for one-off recovery.

Host selection

Present all update candidates in a selectable table with:

checkbox selection per eligible host;
Select all eligible and Clear selection;
filters for host name, tag, current agent version, and online/update eligibility;
visible current version, target version, connectivity, and exclusion reason;
offline, already-current, unknown-version, and already-updating hosts shown but disabled rather than silently omitted.

Default to all eligible hosts so the common fleet-wide action remains quick. Filtering followed by “select all visible” provides the subset workflow for tags or staged cohorts without inventing a second rollout mechanism.

Review and start

Before dispatch, show a compact rollout summary:

target version;
selected and excluded counts;
sequential execution and halt-on-first-failure behavior;
estimated worst-case timeout based on selected host count;
explicit confirmation using the selected count.

Submit the selected IDs through the existing host_ids API field. Server-side validation must deduplicate IDs, reject unknown or ineligible hosts with structured reasons, and ensure the recorded rollout membership exactly matches the reviewed selection.

Progress and recovery

Reuse the current progress panel, per-host states, cancellation, audits, and alerts. Add actions after a halted rollout to:

retry the failed host;
resume remaining eligible hosts as a new rollout;
start a new rollout from the remaining selection.

Do not automatically queue offline hosts in the first iteration; display them as excluded with a clear reason. Deferred/offline rollout scheduling can be considered separately.

Acceptance criteria

An admin can reach fleet updates from normal Settings/dashboard navigation without knowing the URL.
An admin can update all eligible agents in one rollout.
An admin can select an arbitrary subset, including selecting all hosts matching a tag/filter.
The confirmation view accurately lists the target version, selected count, and exclusions.
The server validates and deduplicates submitted host IDs before creating rollout rows.
Rollouts continue to execute sequentially, halt on failure, remain cancellable, and expose per-host progress.
Non-admin users cannot view or start fleet updates.
Tests cover all-eligible selection, explicit subsets, filtered selection, invalid/ineligible IDs, duplicate IDs, authorization, and halted-rollout recovery actions.

Non-goals

Parallel or percentage-based deployment waves.
Automatically waiting for offline hosts to reconnect.
Updating agents to arbitrary versions other than the running server’s bundled version.

## Problem Admins can trigger an agent update from an individual host page, but there is no discoverable UI workflow for updating agents across the fleet or selecting a controlled subset. This makes routine releases unnecessarily repetitive and makes staged rollouts difficult. There is already substantial backend support that should be reused: - `POST /api/fleet/update` starts the rolling fleet-update worker. - An empty `host_ids` list derives all online, out-of-date hosts. - An explicit `host_ids` list already represents a subset. - `/settings/fleet-update` already shows rollout progress, cancellation, and the previous result. - The worker updates sequentially, verifies the reported target version, and halts on first failure. The primary gaps are discoverability and host selection: the fleet-update page is not linked from the Settings navigation, and its start form always submits the complete eligible set. ## Proposed solution Turn the existing page into a first-class admin **Agent updates** workflow under Settings. ### Entry points - Add an `Agent updates` tab or clearly labelled card in Settings linking to `/settings/fleet-update`. - Link the dashboard’s “hosts behind” summary to this workflow, preserving any active dashboard filters where practical. - Keep the per-host update action for one-off recovery. ### Host selection Present all update candidates in a selectable table with: - checkbox selection per eligible host; - `Select all eligible` and `Clear selection`; - filters for host name, tag, current agent version, and online/update eligibility; - visible current version, target version, connectivity, and exclusion reason; - offline, already-current, unknown-version, and already-updating hosts shown but disabled rather than silently omitted. Default to all eligible hosts so the common fleet-wide action remains quick. Filtering followed by “select all visible” provides the subset workflow for tags or staged cohorts without inventing a second rollout mechanism. ### Review and start Before dispatch, show a compact rollout summary: - target version; - selected and excluded counts; - sequential execution and halt-on-first-failure behavior; - estimated worst-case timeout based on selected host count; - explicit confirmation using the selected count. Submit the selected IDs through the existing `host_ids` API field. Server-side validation must deduplicate IDs, reject unknown or ineligible hosts with structured reasons, and ensure the recorded rollout membership exactly matches the reviewed selection. ### Progress and recovery Reuse the current progress panel, per-host states, cancellation, audits, and alerts. Add actions after a halted rollout to: - retry the failed host; - resume remaining eligible hosts as a new rollout; - start a new rollout from the remaining selection. Do not automatically queue offline hosts in the first iteration; display them as excluded with a clear reason. Deferred/offline rollout scheduling can be considered separately. ## Acceptance criteria - An admin can reach fleet updates from normal Settings/dashboard navigation without knowing the URL. - An admin can update all eligible agents in one rollout. - An admin can select an arbitrary subset, including selecting all hosts matching a tag/filter. - The confirmation view accurately lists the target version, selected count, and exclusions. - The server validates and deduplicates submitted host IDs before creating rollout rows. - Rollouts continue to execute sequentially, halt on failure, remain cancellable, and expose per-host progress. - Non-admin users cannot view or start fleet updates. - Tests cover all-eligible selection, explicit subsets, filtered selection, invalid/ineligible IDs, duplicate IDs, authorization, and halted-rollout recovery actions. ## Non-goals - Parallel or percentage-based deployment waves. - Automatically waiting for offline hosts to reconnect. - Updating agents to arbitrary versions other than the running server’s bundled version.

steve added the enhancement label 2026-08-22 10:32:20 +01:00

bobby commented

2026-08-22 10:38:59 +01:00

Agree with this as specified — reusing POST /api/fleet/update and the existing progress page rather than adding a second rollout mechanism is the right call, and showing ineligible hosts disabled-with-reason instead of omitting them avoids the usual "why isn't my host listed" confusion.

The discoverability premise matches experience: the endpoint was only findable by reading the route table in server.go, and the operator role then gets a 403 insufficient_role with no indication that an admin UI exists for it.

Three additions from a v1.1.0 → v1.1.1 rollout, all observed rather than hypothetical.

1. Verification by `agent_version` is unsound while report delivery can fail

This is the one worth acting on before building the UI.

internal/server/fleetupdate/worker.go confirms a host succeeded by polling the host row's recorded agent_version until it matches the target:

// Poll until the host's recorded agent_version matches target, or timeout.
deadline := time.Now().Add(w.hostTimeout)

That row is refreshed over the agent→server reporting channel — which is exactly the channel that can be severed by the WebSocket read limit in #44. The failure mode is:

Agent updates correctly and restarts on the new version.
Its report is dropped (oversized message closes the connection, or any other delivery failure).
The host row still shows the old version.
The worker times out and calls halt.

Because the worker halts the entire rollout on first failure, one host that updated fine but failed to report blocks every host queued behind it. The operator sees a halted rollout and a host that is, in reality, already on the target version.

This gets more likely as #43 makes fleet updates routine and larger batches run at once.

Two options that avoid depending on projection/report delivery:

Treat the hello handshake on reconnect as authoritative for version. The agent already sends version in agent starting / hello, and a self-updating agent must reconnect to be considered healthy — so the reconnect itself carries a trustworthy version without a separate report.
Failing that, re-read the version once more after the timeout before halting, and distinguish "did not update" from "updated but did not report" in failed_reason. The current message (timeout waiting for <host> to reach <version>) reads as the former when it may be the latter.

2. Real self-update timing, for the worst-case timeout estimate

The issue proposes showing an estimated worst-case timeout based on host count. A measured data point for that:

10:25:37  agent self-update: binary swapped, exiting for systemd restart
10:25:38  systemd: Deactivated successfully
10:25:43  systemd: Scheduled restart job
10:25:43  agent starting  version=v1.1.1
10:25:43  ws agent connected

~6 seconds end-to-end, including the 5s systemd restart delay, on a Linux amd64 host. Against hostTimeout = 95 * time.Second that is roughly 15x headroom, so the default looks well chosen and the estimate shown to the operator can be honest about it being dominated by the timeout ceiling, not by expected duration.

Worth noting the restart delay is a systemd unit property, so the floor is environment-specific — a host with a longer RestartSec will be slower without anything being wrong.

3. Make canary-first a named workflow, not just an achievable one

Subset selection technically permits "select one host, then select the rest", but the pattern worth designing for is update one → verify → continue. In this rollout that sequence caught a real bug (#44) on the first host before it reached the remaining four.

The machinery already exists — the worker is sequential, halts, and the issue proposes resume actions. What is missing is making the intent expressible in one action rather than two rollouts, e.g. a "pause after first host for confirmation" checkbox on the start form, reusing the existing halted-state resume path.

This fits the stated non-goals: it is not parallel waves or percentage-based deployment, just an explicit stop after host 1. It is also the safest default for an operator updating a fleet whose agents all run the same backup-critical code path.

Related: #44 (read limit — the concrete delivery failure behind point 1), #41 (stuck running jobs — why a failed rollout step can be silent).

Agree with this as specified — reusing `POST /api/fleet/update` and the existing progress page rather than adding a second rollout mechanism is the right call, and showing ineligible hosts disabled-with-reason instead of omitting them avoids the usual "why isn't my host listed" confusion. The discoverability premise matches experience: the endpoint was only findable by reading the route table in `server.go`, and the operator role then gets a `403 insufficient_role` with no indication that an admin UI exists for it. Three additions from a v1.1.0 → v1.1.1 rollout, all observed rather than hypothetical. --- ### 1. Verification by `agent_version` is unsound while report delivery can fail This is the one worth acting on before building the UI. `internal/server/fleetupdate/worker.go` confirms a host succeeded by polling **the host row's recorded `agent_version`** until it matches the target: ```go // Poll until the host's recorded agent_version matches target, or timeout. deadline := time.Now().Add(w.hostTimeout) ``` That row is refreshed over the agent→server reporting channel — which is exactly the channel that can be severed by the WebSocket read limit in #44. The failure mode is: 1. Agent updates correctly and restarts on the new version. 2. Its report is dropped (oversized message closes the connection, or any other delivery failure). 3. The host row still shows the old version. 4. The worker times out and calls `halt`. Because the worker **halts the entire rollout on first failure**, one host that updated fine but failed to report blocks every host queued behind it. The operator sees a halted rollout and a host that is, in reality, already on the target version. This gets more likely as #43 makes fleet updates routine and larger batches run at once. Two options that avoid depending on projection/report delivery: - Treat the **hello handshake on reconnect** as authoritative for version. The agent already sends `version` in `agent starting` / hello, and a self-updating agent *must* reconnect to be considered healthy — so the reconnect itself carries a trustworthy version without a separate report. - Failing that, re-read the version once more after the timeout before halting, and distinguish "did not update" from "updated but did not report" in `failed_reason`. The current message (`timeout waiting for <host> to reach <version>`) reads as the former when it may be the latter. ### 2. Real self-update timing, for the worst-case timeout estimate The issue proposes showing an estimated worst-case timeout based on host count. A measured data point for that: ``` 10:25:37 agent self-update: binary swapped, exiting for systemd restart 10:25:38 systemd: Deactivated successfully 10:25:43 systemd: Scheduled restart job 10:25:43 agent starting version=v1.1.1 10:25:43 ws agent connected ``` **~6 seconds** end-to-end, including the 5s systemd restart delay, on a Linux amd64 host. Against `hostTimeout = 95 * time.Second` that is roughly 15x headroom, so the default looks well chosen and the estimate shown to the operator can be honest about it being dominated by the timeout ceiling, not by expected duration. Worth noting the restart delay is a systemd unit property, so the floor is environment-specific — a host with a longer `RestartSec` will be slower without anything being wrong. ### 3. Make canary-first a named workflow, not just an achievable one Subset selection technically permits "select one host, then select the rest", but the pattern worth designing for is **update one → verify → continue**. In this rollout that sequence caught a real bug (#44) on the first host before it reached the remaining four. The machinery already exists — the worker is sequential, halts, and the issue proposes resume actions. What is missing is making the intent expressible in one action rather than two rollouts, e.g. a *"pause after first host for confirmation"* checkbox on the start form, reusing the existing halted-state resume path. This fits the stated non-goals: it is not parallel waves or percentage-based deployment, just an explicit stop after host 1. It is also the safest default for an operator updating a fleet whose agents all run the same backup-critical code path. --- Related: #44 (read limit — the concrete delivery failure behind point 1), #41 (stuck `running` jobs — why a failed rollout step can be silent).

steve referenced a pull request that will close this issue

2026-08-22 11:09:00 +01:00

Add selectable canary-first agent updates #48

steve commented

2026-08-22 11:09:13 +01:00

Implementation is in #48. Bobby’s rollout feedback is addressed without changing this issue:

version success remains based on the reconnect hello persisted by MarkHostHello; the worker now performs one final authoritative read at the deadline and reports the last observed version
the review displays the 95s-per-host worst-case ceiling (with the observed ~6s healthy path understood as typical, not guaranteed)
canary-first is a named/default workflow: the rollout pauses after the first host reconnects on the target version, then offers resume/retry actions

The PR also adds discoverability, filtered subset selection, disabled exclusion reasons, and server-side membership validation/deduplication.

Implementation is in #48. Bobby’s rollout feedback is addressed without changing this issue: - version success remains based on the reconnect `hello` persisted by `MarkHostHello`; the worker now performs one final authoritative read at the deadline and reports the last observed version - the review displays the 95s-per-host worst-case ceiling (with the observed ~6s healthy path understood as typical, not guaranteed) - canary-first is a named/default workflow: the rollout pauses after the first host reconnects on the target version, then offers resume/retry actions The PR also adds discoverability, filtered subset selection, disabled exclusion reasons, and server-side membership validation/deduplication.

steve closed this issue

2026-08-22 11:11:42 +01:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: steve/restic-manager#43

Expose fleet-wide agent updates with host subset selection #43