Agent panics (nil pointer) on every forget job — jobs hang at status=running and never alert #37

New Issue

2026-08-22T09:31:57+01:00

bobby commented

2026-08-22 09:31:57 +01:00

Summary

The agent panics with a nil pointer dereference on every forget job, fleet-wide, ~1s after accepting it. systemd restarts the agent, but the job is never reported as finished — so it sits at status: running forever with 0 log lines, and no alert is raised (only failed alerts; a hung job does not).

restic's forget itself appears to complete — hosts with correct config still thin properly — so the crash is on the result-reporting path, after the useful work. That is what makes it so quiet: retention silently works on some hosts and silently doesn't on others, and the dashboard shows neither.

Stack trace

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8e319a]

goroutine 1 [running]:
example.internal/restic-manager/internal/agent/wsclient.connectOnce.func1()
        internal/agent/wsclient/client.go:114 +0x1a
example.internal/restic-manager/internal/agent/wsclient.connectOnce(...)
        internal/agent/wsclient/client.go:165 +0xdda
example.internal/restic-manager/internal/agent/wsclient.Run(...)
        internal/agent/wsclient/client.go:64 +0xd8
main.run()
        cmd/agent/main.go:150 +0x1090
main.main()
        cmd/agent/main.go:29 +0x17

restic-manager-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

Likely cause

client.go:114 is the deferred close of the dial response:

conn, res, err := websocket.Dial(dialCtx, wsURL, dialOpts)
cancel()
if err != nil {
    return fmt.Errorf("dial: %w", err)
}
defer func() { _ = res.Body.Close() }()   // <-- line 114, panics

With github.com/coder/websocket v1.8.14, the *http.Response returned by Dial is not guaranteed non-nil even when err == nil, and res.Body may itself be nil. The deferred closure dereferences it unconditionally, and addr=0x18 is consistent with a nil-struct field offset rather than a nil interface.

Note the panic is in the deferred call, so it fires when connectOnce returns — i.e. when the connection is torn down after handling the forget — which matches the observed ~1s delay after accepting forget job.

Suggested fix:

if res != nil && res.Body != nil {
    defer func() { _ = res.Body.Close() }()
}

Impact

High, mostly because it is silent:

Forget jobs never reach a terminal state — they stay running indefinitely. On host-A the jobs from 2026-08-20, 08-21 and 08-22 are all still running with 0 log lines.
Because only failed raises an alert, a hung forget produces no alert at all. OpenAlerts: 0 while retention has not run for months.
The agent process is killed and restarted each time (restart counter is at 33 on host-B), so any other in-flight work on that agent dies too.

Scope — fleet-wide, all on v1.1.0

host	panics (7d)	last panic	forget job that triggered it
host-B	18	2026-08-22 03:00:57	accepted 03:00:56
host-C	24	2026-08-22 03:15:57	accepted 03:15:56
host-A	(every run)	2026-08-22 04:00:58	accepted 04:00:56
host-E	0	—	host was powered off; no forget ran

In every case the panic timestamp is exactly 1–2s after the accepting forget job log line.

Not related to source-group config

Worth stating explicitly, since the two were initially conflated: re-saving a host's source group changes whether retention is applied, but has no effect on the panic. host-B had its source group re-saved at 2026-08-21T18:21 and still panicked at 2026-08-21 18:21:57 and again at 2026-08-22 03:00:57.

Environment

Agent v1.1.0 (commit 0f5110f3d9b91b269684453e6b8d14dbcffb93c6, built 2026-06-16T06:32:52Z)
restic 0.18.1, protocol_version 1
github.com/coder/websocket v1.8.14
Observed on Ubuntu 24.04 (host-A, host-B, host-C)

## Summary The agent **panics with a nil pointer dereference on every forget job**, fleet-wide, ~1s after accepting it. systemd restarts the agent, but the job is never reported as finished — so it sits at `status: running` forever with 0 log lines, and **no alert is raised** (only `failed` alerts; a hung job does not). restic's forget itself appears to complete — hosts with correct config still thin properly — so the crash is on the result-reporting path, *after* the useful work. That is what makes it so quiet: retention silently works on some hosts and silently doesn't on others, and the dashboard shows neither. ## Stack trace ``` panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8e319a] goroutine 1 [running]: example.internal/restic-manager/internal/agent/wsclient.connectOnce.func1() internal/agent/wsclient/client.go:114 +0x1a example.internal/restic-manager/internal/agent/wsclient.connectOnce(...) internal/agent/wsclient/client.go:165 +0xdda example.internal/restic-manager/internal/agent/wsclient.Run(...) internal/agent/wsclient/client.go:64 +0xd8 main.run() cmd/agent/main.go:150 +0x1090 main.main() cmd/agent/main.go:29 +0x17 restic-manager-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT ``` ## Likely cause `client.go:114` is the deferred close of the dial response: ```go conn, res, err := websocket.Dial(dialCtx, wsURL, dialOpts) cancel() if err != nil { return fmt.Errorf("dial: %w", err) } defer func() { _ = res.Body.Close() }() // <-- line 114, panics ``` With `github.com/coder/websocket v1.8.14`, the `*http.Response` returned by `Dial` is **not guaranteed non-nil even when `err == nil`**, and `res.Body` may itself be nil. The deferred closure dereferences it unconditionally, and `addr=0x18` is consistent with a nil-struct field offset rather than a nil interface. Note the panic is in the **deferred** call, so it fires when `connectOnce` returns — i.e. when the connection is torn down after handling the forget — which matches the observed ~1s delay after `accepting forget job`. Suggested fix: ```go if res != nil && res.Body != nil { defer func() { _ = res.Body.Close() }() } ``` ## Impact **High**, mostly because it is silent: 1. Forget jobs never reach a terminal state — they stay `running` indefinitely. On `host-A` the jobs from 2026-08-20, 08-21 and 08-22 are all still `running` with 0 log lines. 2. Because only `failed` raises an alert, a hung forget produces **no alert at all**. `OpenAlerts: 0` while retention has not run for months. 3. The agent process is killed and restarted each time (`restart counter is at 33` on host-B), so any other in-flight work on that agent dies too. ## Scope — fleet-wide, all on v1.1.0 | host | panics (7d) | last panic | forget job that triggered it | |---|---|---|---| | host-B | 18 | 2026-08-22 03:00:57 | accepted 03:00:56 | | host-C | 24 | 2026-08-22 03:15:57 | accepted 03:15:56 | | host-A | (every run) | 2026-08-22 04:00:58 | accepted 04:00:56 | | host-E | 0 | — | host was powered off; no forget ran | In every case the panic timestamp is exactly 1–2s after the `accepting forget job` log line. ## Not related to source-group config Worth stating explicitly, since the two were initially conflated: re-saving a host's source group changes whether *retention is applied*, but has **no effect on the panic**. `host-B` had its source group re-saved at 2026-08-21T18:21 and still panicked at 2026-08-21 18:21:57 and again at 2026-08-22 03:00:57. ## Environment - Agent `v1.1.0` (commit `0f5110f3d9b91b269684453e6b8d14dbcffb93c6`, built 2026-06-16T06:32:52Z) - restic `0.18.1`, protocol_version 1 - `github.com/coder/websocket v1.8.14` - Observed on Ubuntu 24.04 (host-A, host-B, host-C)

steve referenced a pull request that will close this issue

2026-08-22 09:45:29 +01:00

Fix agent panic on WebSocket disconnect #38

steve closed this issue

2026-08-22 09:49:37 +01:00

bobby referenced this issue

2026-08-22 09:54:47 +01:00

Snapshot projection never refreshes after forget — counts are stale until the next backup #40

steve referenced this issue

2026-08-22 10:03:19 +01:00

Snapshot projection never refreshes after forget — counts are stale until the next backup #40

steve referenced this issue