docs: status report — live validation done, header-only fetch resolved

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

specifications/PHASE1-STATUS.md

@@ -1,12 +1,28 @@
 # emcli — Phase 1 Status Report
 
 **Date:** 2026-06-22
-**Branch:** `main` (pushed to `gitea.dcglab.co.uk/steve/emcli`, HEAD `6061bd2`)
+**Branch:** `main` (pushed to `gitea.dcglab.co.uk/steve/emcli`)
 **Phase 1 scope:** Foundation + read path (encrypted config, IMAP read, agent commands, password auth)
 
+> **Update (post-Phase-1 hardening):** the read path has now been **validated end-to-end against a live IMAP account**, and the full-message-body-download issue has been **resolved within Phase 1** (`list`/`search` fetch headers only). See "Live validation" and "Resolved" sections below.
+
 ## TL;DR
 
-**Phase 1 is complete and merged to `main`.** All 14 planned tasks were implemented test-first, each reviewed (spec compliance + code quality) by a separate agent, with fixes applied and re-verified. The binary builds as a single static CGO-free executable, `go vet` is clean, and the full unit-test suite passes. An end-to-end smoke test of the real binary confirms the agent (JSON) and admin (human-readable) surfaces behave per spec.
+**Phase 1 is complete and merged to `main`.** All 14 planned tasks were implemented test-first, each reviewed (spec compliance + code quality) by a separate agent, with fixes applied and re-verified. The binary builds as a single static CGO-free executable, `go vet` is clean, and the full unit-test suite passes (including under `-race`). Validated both via a real-binary smoke test and **against a live IMAP mailbox**.
+
+## Live validation (real IMAP account)
+
+The read path was exercised against a live cPanel/MXlogin IMAP account over verified TLS (port 993, Let's Encrypt cert):
+- **Auth + connect:** email-as-username login over implicit TLS succeeded.
+- **`list`:** returned real INBOX headers; `has_attachments` correct.
+- **`get`:** returned the full plain-text body and base64 attachments that decoded to exact byte sizes (a cPanel config mail with a PNG + 3 `.mobileconfig` files — all 4 matched).
+- **`search`:** `--from`/`--subject-contains` correctly narrowed results.
+- **Seen-set:** `list --new` correctly returned 0 on first contact (baseline floor = max UID); `ack` succeeded; `folder_state`/`acked` rows correct.
+- **Audit:** every command wrote an `audit_log` row; password confirmed encrypted at rest (no plaintext in the DB).
+
+## Resolved within Phase 1: header-only fetch
+
+The previously-flagged "header listing downloads full message bodies" issue is **fixed** (`internal/mail/imap.go`). `list` and `search` now fetch `BODY.PEEK[HEADER]` + `BODYSTRUCTURE` instead of the whole RFC822 message, so listing a large mailbox no longer downloads every body and attachment. Header parsing reuses the same go-message path (RFC2047 decoding and address formatting unchanged); `has_attachments` is derived from the body-structure tree. `get` still fetches the full message. Verified: output is identical to the prior full-fetch behaviour against the live account, new unit tests (`TestParseHeaderBytes`, `TestHasAttachment`) pass, and the fetch loops were hardened to drain the IMAP channel on error (race-tested).
 
 ```
 go build (CGO_ENABLED=0)  → OK, single static binary
@@ -56,19 +72,19 @@ Executed via subagent-driven development: a fresh implementer per task (TDD, com
 - **Phase 4 — Admin TUI + `doctor`:** interactive init/reconfigure, connectivity diagnostics.
 
 ### Carry-over items to address (logged from reviews)
-1. **[Important, Phase 2 priority] Header listing fetches full message bodies.** `mail.fetchByUIDSet` downloads each whole RFC822 message even for `list`/`search`, then discards the body. Functionally correct but slow/heavy on real mailboxes. Fix = request a header-only body section / `ENVELOPE` + `BODYSTRUCTURE` when `full=false`. **Deliberately not fixed yet** because the change to the IMAP fetch path could not be validated against a live server in this environment (see below); it should be done with a real/TLS-configured IMAP server in front of it. Phase 2's `--reply-to` should build that header-only fetch path too.
+1. **[Minor] Audit completeness:** `list`/`search` don't write a per-filtered-message audit row; some IMAP-error envelope paths don't write an audit row; `imap_error` is an audit reason not enumerated in the spec.
-2. **[Minor] Audit completeness:** `list`/`search` don't write a per-filtered-message audit row; some IMAP-error envelope paths don't write an audit row; `imap_error` is an audit reason not enumerated in the spec.
+2. **[Minor] CLI polish:** `account add` opens the store before parsing its own flags (`--help` touches the DB); invalid `--since-date`/`--before-date` are silently ignored rather than erroring; `whitelist add/remove` don't reject an empty `--address` at the CLI layer.
-3. **[Minor] CLI polish:** `account add` opens the store before parsing its own flags (`--help` touches the DB); invalid `--since-date`/`--before-date` are silently ignored rather than erroring; `whitelist add/remove` don't reject an empty `--address` at the CLI layer.
+3. **[Minor] Performance:** `ack` fetches each UID's header in its own round-trip; `get` fetches the header then re-fetches the full message. Fine for a per-invocation CLI; batch ops in later phases may want to optimize.
-4. **[Minor] Performance:** `ack` fetches each UID's header in its own round-trip; `get` fetches the header then re-fetches the full message. Fine for a per-invocation CLI; batch ops in later phases may want to optimize.
 
-### Validation gap
+### Previously-open items now closed
-- **Live IMAP integration was not exercised.** The `internal/mail` integration test (build tag `integration`) compiles and self-skips without a server. An attempt to validate against a GreenMail container failed on TLS/port friction: GreenMail 2.1.0's IMAP STARTTLS didn't negotiate here and the IMAPS port wasn't exposed, while our client (correctly) requires verified TLS and the test config has no skip-verify option. The IMAP code was reviewed by inspection and compiles under the integration tag, but has **not** been run end-to-end against a real server. **Recommended follow-up:** add a dev-only `--insecure`/skip-verify option (or point the integration test at a properly-certificated server) so the read path can be validated against live IMAP before relying on it operationally.
+- ✅ **Live IMAP validation** — done against a real mailbox over verified TLS (see "Live validation" above). The earlier GreenMail attempt had failed only on local self-signed-TLS friction, not a code issue.
+- ✅ **Header-only fetch** — `list`/`search` no longer download message bodies (see "Resolved within Phase 1" above).
+- A dev-only `--insecure`/skip-verify option is still worth adding later for local self-signed test servers, but is not needed for real providers (valid certs verified fine).
 
 ## Suggested next steps
 
-1. **Validate the read path against a real IMAP account** (e.g. your own mailbox over TLS) — this is the highest-value confidence step and would also surface the full-body-fetch performance issue (#1) in practice.
+1. Proceed to **Phase 2 (send path)** — the structure is ready: `policy` is set up to host outbound checks, `Account.Mode` is loaded for RO rejection, and `MatchAddress` already implements the domain matching whitelist-out needs. `--reply-to` can reuse the new header-only fetch path to read `Message-ID`/`References` cheaply.
-2. Decide whether to fix the header-only fetch (#1) before or as part of Phase 2.
+2. Optionally clean up the Minor carry-over items above as part of Phase 2 touch-ups.
-3. Proceed to **Phase 2 (send path)** — the structure is ready: `policy` is set up to host outbound checks, `Account.Mode` is loaded for RO rejection, and `MatchAddress` already implements the domain matching whitelist-out needs.
 
 ## Repository layout