Server WebSocket read limit (32 KiB default) drops snapshots.report on hosts with many snapshots #44

New Issue

2026-08-22T10:36:58+01:00

bobby commented

2026-08-22 10:36:58 +01:00

Summary

The agent WebSocket handler accepts connections without calling SetReadLimit, so it inherits coder/websocket's 32768-byte default. A snapshots.report for a host with enough snapshots exceeds that, and the server closes the connection mid-message with StatusMessageTooBig. The projection is never updated and the job never reaches a terminal state.

This is a concrete instance of the best-effort delivery problem described in #40 — here the connection is lost because of the report itself, so it is reproducible rather than incidental.

Only observable after #37 is fixed. On v1.1.0 the agent panicked before it ever attempted the report, so the limit was never reached. Upgrading one agent to v1.1.1 surfaced this immediately.

Observed (agent v1.1.1, server v1.1.1)

Manual forget --dry-run on a host with 23 snapshots in the projection:

INFO  agent: accepting forget job   job_id=<job-id> groups=1 dry_run=true
WARN  ws agent disconnect  err="read: failed to get reader: received close frame:
        status = StatusMessageTooBig and reason = \"read limited at 32769 bytes\""
WARN  runner: snapshots.report after forget failed  job_id=<job-id>
        err="failed to write msg: use of closed network connection"
INFO  agent: forget job complete    job_id=<job-id>

The agent then reconnects normally. From the host's perspective everything succeeded — forget job complete, no panic, service healthy. Only the server-side view is wrong.

Resulting state

Job remains status: running with 0 log lines (still running several minutes later).
refreshed_at unchanged — still the previous backup's timestamp.
No alert raised, because only failed alerts today (see #41).

So the operator sees a healthy agent, a job that never ends, and a stale snapshot count, with nothing indicating a problem.

Cause

internal/server/ws/handler.go:

conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{
    InsecureSkipVerify: true,
})
// no conn.SetReadLimit(...) anywhere on this path

coder/websocket defaults to a 32768-byte read limit, which matches the reported read limited at 32769 bytes exactly.

Sizing

Measured from one host's own projection: 23 snapshots ≈ 6999 bytes of JSON, or roughly 304 bytes per snapshot. That puts the ceiling near ~107 snapshots for a single-path host.

Treat that as indicative rather than exact — it is derived from the server's projection shape, and the wire payload may carry fields the projection does not. Hosts with multiple source groups, long path lists, or many tags will hit it sooner.

This is well within normal operating range: a host on keep_daily 30 plus weeklies and monthlies would exceed it in routine use, and any repository that has not been thinned for a while will be over it.

Suggested fix

Set an explicit read limit on the accepted connection, sized for the largest realistic snapshots.report (a few MiB) rather than relying on the library default. Worth setting it explicitly on the agent side too, so the limit is a deliberate protocol decision rather than a library default on both ends.
Consider bounding the payload independently of the limit — chunk or paginate snapshots.report, or send a digest plus deltas. A single host with a large repository should not be able to outgrow the transport.
Surface the failure. The agent already logs snapshots.report ... failed at WARN, but the server has no idea the report was attempted. Reporting the failure back would let #40 flag the projection as stale instead of silently serving an old count.

(1) is the immediate fix; (2) and (3) are what stop it recurring at a larger size.

Environment

Server v1.1.1, agent v1.1.1 (commit 1131f433)
restic 0.18.1, protocol_version 1
github.com/coder/websocket v1.8.14
Related: #40 (stale projection — this is one concrete cause), #41 (stuck running jobs — why it is silent), #37 (fixing the panic is what exposed this)

## Summary The agent WebSocket handler accepts connections without calling `SetReadLimit`, so it inherits `coder/websocket`'s **32768-byte default**. A `snapshots.report` for a host with enough snapshots exceeds that, and **the server closes the connection mid-message** with `StatusMessageTooBig`. The projection is never updated and the job never reaches a terminal state. This is a concrete instance of the best-effort delivery problem described in #40 — here the connection is lost *because of the report itself*, so it is reproducible rather than incidental. **Only observable after #37 is fixed.** On v1.1.0 the agent panicked before it ever attempted the report, so the limit was never reached. Upgrading one agent to v1.1.1 surfaced this immediately. ## Observed (agent v1.1.1, server v1.1.1) Manual `forget --dry-run` on a host with 23 snapshots in the projection: ``` INFO agent: accepting forget job job_id=<job-id> groups=1 dry_run=true WARN ws agent disconnect err="read: failed to get reader: received close frame: status = StatusMessageTooBig and reason = \"read limited at 32769 bytes\"" WARN runner: snapshots.report after forget failed job_id=<job-id> err="failed to write msg: use of closed network connection" INFO agent: forget job complete job_id=<job-id> ``` The agent then reconnects normally. From the host's perspective everything succeeded — `forget job complete`, no panic, service healthy. Only the server-side view is wrong. ## Resulting state - Job remains `status: running` with 0 log lines (still `running` several minutes later). - `refreshed_at` unchanged — still the previous backup's timestamp. - No alert raised, because only `failed` alerts today (see #41). So the operator sees a healthy agent, a job that never ends, and a stale snapshot count, with nothing indicating a problem. ## Cause `internal/server/ws/handler.go`: ```go conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{ InsecureSkipVerify: true, }) // no conn.SetReadLimit(...) anywhere on this path ``` `coder/websocket` defaults to a 32768-byte read limit, which matches the reported `read limited at 32769 bytes` exactly. ## Sizing Measured from one host's own projection: 23 snapshots ≈ 6999 bytes of JSON, or roughly **304 bytes per snapshot**. That puts the ceiling near **~107 snapshots** for a single-path host. Treat that as indicative rather than exact — it is derived from the server's projection shape, and the wire payload may carry fields the projection does not. Hosts with multiple source groups, long path lists, or many tags will hit it sooner. This is well within normal operating range: a host on `keep_daily 30` plus weeklies and monthlies would exceed it in routine use, and any repository that has not been thinned for a while will be over it. ## Suggested fix 1. **Set an explicit read limit** on the accepted connection, sized for the largest realistic `snapshots.report` (a few MiB) rather than relying on the library default. Worth setting it explicitly on the agent side too, so the limit is a deliberate protocol decision rather than a library default on both ends. 2. **Consider bounding the payload independently of the limit** — chunk or paginate `snapshots.report`, or send a digest plus deltas. A single host with a large repository should not be able to outgrow the transport. 3. **Surface the failure.** The agent already logs `snapshots.report ... failed` at WARN, but the server has no idea the report was attempted. Reporting the failure back would let #40 flag the projection as stale instead of silently serving an old count. (1) is the immediate fix; (2) and (3) are what stop it recurring at a larger size. ## Environment - Server `v1.1.1`, agent `v1.1.1` (commit `1131f433`) - restic `0.18.1`, protocol_version 1 - `github.com/coder/websocket v1.8.14` - Related: #40 (stale projection — this is one concrete cause), #41 (stuck `running` jobs — why it is silent), #37 (fixing the panic is what exposed this)

bobby referenced this issue

2026-08-22 10:38:59 +01:00

Expose fleet-wide agent updates with host subset selection #43

steve referenced a pull request that will close this issue

2026-08-22 10:46:02 +01:00

Fix WebSocket limit for large agent messages #45

steve closed this issue

2026-08-22 10:48:32 +01:00

Sign in to join this conversation.