diff --git a/specifications/PHASE1-STATUS.md b/specifications/PHASE1-STATUS.md index 1209a81..2c0f13a 100644 --- a/specifications/PHASE1-STATUS.md +++ b/specifications/PHASE1-STATUS.md @@ -1,12 +1,28 @@ # emcli — Phase 1 Status Report **Date:** 2026-06-22 -**Branch:** `main` (pushed to `gitea.dcglab.co.uk/steve/emcli`, HEAD `6061bd2`) +**Branch:** `main` (pushed to `gitea.dcglab.co.uk/steve/emcli`) **Phase 1 scope:** Foundation + read path (encrypted config, IMAP read, agent commands, password auth) +> **Update (post-Phase-1 hardening):** the read path has now been **validated end-to-end against a live IMAP account**, and the full-message-body-download issue has been **resolved within Phase 1** (`list`/`search` fetch headers only). See "Live validation" and "Resolved" sections below. + ## TL;DR -**Phase 1 is complete and merged to `main`.** All 14 planned tasks were implemented test-first, each reviewed (spec compliance + code quality) by a separate agent, with fixes applied and re-verified. The binary builds as a single static CGO-free executable, `go vet` is clean, and the full unit-test suite passes. An end-to-end smoke test of the real binary confirms the agent (JSON) and admin (human-readable) surfaces behave per spec. +**Phase 1 is complete and merged to `main`.** All 14 planned tasks were implemented test-first, each reviewed (spec compliance + code quality) by a separate agent, with fixes applied and re-verified. The binary builds as a single static CGO-free executable, `go vet` is clean, and the full unit-test suite passes (including under `-race`). Validated both via a real-binary smoke test and **against a live IMAP mailbox**. + +## Live validation (real IMAP account) + +The read path was exercised against a live cPanel/MXlogin IMAP account over verified TLS (port 993, Let's Encrypt cert): +- **Auth + connect:** email-as-username login over implicit TLS succeeded. +- **`list`:** returned real INBOX headers; `has_attachments` correct. +- **`get`:** returned the full plain-text body and base64 attachments that decoded to exact byte sizes (a cPanel config mail with a PNG + 3 `.mobileconfig` files — all 4 matched). +- **`search`:** `--from`/`--subject-contains` correctly narrowed results. +- **Seen-set:** `list --new` correctly returned 0 on first contact (baseline floor = max UID); `ack` succeeded; `folder_state`/`acked` rows correct. +- **Audit:** every command wrote an `audit_log` row; password confirmed encrypted at rest (no plaintext in the DB). + +## Resolved within Phase 1: header-only fetch + +The previously-flagged "header listing downloads full message bodies" issue is **fixed** (`internal/mail/imap.go`). `list` and `search` now fetch `BODY.PEEK[HEADER]` + `BODYSTRUCTURE` instead of the whole RFC822 message, so listing a large mailbox no longer downloads every body and attachment. Header parsing reuses the same go-message path (RFC2047 decoding and address formatting unchanged); `has_attachments` is derived from the body-structure tree. `get` still fetches the full message. Verified: output is identical to the prior full-fetch behaviour against the live account, new unit tests (`TestParseHeaderBytes`, `TestHasAttachment`) pass, and the fetch loops were hardened to drain the IMAP channel on error (race-tested). ``` go build (CGO_ENABLED=0) → OK, single static binary @@ -56,19 +72,19 @@ Executed via subagent-driven development: a fresh implementer per task (TDD, com - **Phase 4 — Admin TUI + `doctor`:** interactive init/reconfigure, connectivity diagnostics. ### Carry-over items to address (logged from reviews) -1. **[Important, Phase 2 priority] Header listing fetches full message bodies.** `mail.fetchByUIDSet` downloads each whole RFC822 message even for `list`/`search`, then discards the body. Functionally correct but slow/heavy on real mailboxes. Fix = request a header-only body section / `ENVELOPE` + `BODYSTRUCTURE` when `full=false`. **Deliberately not fixed yet** because the change to the IMAP fetch path could not be validated against a live server in this environment (see below); it should be done with a real/TLS-configured IMAP server in front of it. Phase 2's `--reply-to` should build that header-only fetch path too. -2. **[Minor] Audit completeness:** `list`/`search` don't write a per-filtered-message audit row; some IMAP-error envelope paths don't write an audit row; `imap_error` is an audit reason not enumerated in the spec. -3. **[Minor] CLI polish:** `account add` opens the store before parsing its own flags (`--help` touches the DB); invalid `--since-date`/`--before-date` are silently ignored rather than erroring; `whitelist add/remove` don't reject an empty `--address` at the CLI layer. -4. **[Minor] Performance:** `ack` fetches each UID's header in its own round-trip; `get` fetches the header then re-fetches the full message. Fine for a per-invocation CLI; batch ops in later phases may want to optimize. +1. **[Minor] Audit completeness:** `list`/`search` don't write a per-filtered-message audit row; some IMAP-error envelope paths don't write an audit row; `imap_error` is an audit reason not enumerated in the spec. +2. **[Minor] CLI polish:** `account add` opens the store before parsing its own flags (`--help` touches the DB); invalid `--since-date`/`--before-date` are silently ignored rather than erroring; `whitelist add/remove` don't reject an empty `--address` at the CLI layer. +3. **[Minor] Performance:** `ack` fetches each UID's header in its own round-trip; `get` fetches the header then re-fetches the full message. Fine for a per-invocation CLI; batch ops in later phases may want to optimize. -### Validation gap -- **Live IMAP integration was not exercised.** The `internal/mail` integration test (build tag `integration`) compiles and self-skips without a server. An attempt to validate against a GreenMail container failed on TLS/port friction: GreenMail 2.1.0's IMAP STARTTLS didn't negotiate here and the IMAPS port wasn't exposed, while our client (correctly) requires verified TLS and the test config has no skip-verify option. The IMAP code was reviewed by inspection and compiles under the integration tag, but has **not** been run end-to-end against a real server. **Recommended follow-up:** add a dev-only `--insecure`/skip-verify option (or point the integration test at a properly-certificated server) so the read path can be validated against live IMAP before relying on it operationally. +### Previously-open items now closed +- ✅ **Live IMAP validation** — done against a real mailbox over verified TLS (see "Live validation" above). The earlier GreenMail attempt had failed only on local self-signed-TLS friction, not a code issue. +- ✅ **Header-only fetch** — `list`/`search` no longer download message bodies (see "Resolved within Phase 1" above). +- A dev-only `--insecure`/skip-verify option is still worth adding later for local self-signed test servers, but is not needed for real providers (valid certs verified fine). ## Suggested next steps -1. **Validate the read path against a real IMAP account** (e.g. your own mailbox over TLS) — this is the highest-value confidence step and would also surface the full-body-fetch performance issue (#1) in practice. -2. Decide whether to fix the header-only fetch (#1) before or as part of Phase 2. -3. Proceed to **Phase 2 (send path)** — the structure is ready: `policy` is set up to host outbound checks, `Account.Mode` is loaded for RO rejection, and `MatchAddress` already implements the domain matching whitelist-out needs. +1. Proceed to **Phase 2 (send path)** — the structure is ready: `policy` is set up to host outbound checks, `Account.Mode` is loaded for RO rejection, and `MatchAddress` already implements the domain matching whitelist-out needs. `--reply-to` can reuse the new header-only fetch path to read `Message-ID`/`References` cheaply. +2. Optionally clean up the Minor carry-over items above as part of Phase 2 touch-ups. ## Repository layout