109 lines
8.4 KiB
Markdown
109 lines
8.4 KiB
Markdown
# emcli — Phase 1 Status Report
|
|
|
|
**Date:** 2026-06-22
|
|
**Branch:** `main` (pushed to `gitea.dcglab.co.uk/steve/emcli`)
|
|
**Phase 1 scope:** Foundation + read path (encrypted config, IMAP read, agent commands, password auth)
|
|
|
|
> **Update (post-Phase-1 hardening):** the read path has now been **validated end-to-end against a live IMAP account**, and the full-message-body-download issue has been **resolved within Phase 1** (`list`/`search` fetch headers only). See "Live validation" and "Resolved" sections below.
|
|
|
|
## TL;DR
|
|
|
|
**Phase 1 is complete and merged to `main`.** All 14 planned tasks were implemented test-first, each reviewed (spec compliance + code quality) by a separate agent, with fixes applied and re-verified. The binary builds as a single static CGO-free executable, `go vet` is clean, and the full unit-test suite passes (including under `-race`). Validated both via a real-binary smoke test and **against a live IMAP mailbox**.
|
|
|
|
## Live validation (real IMAP account)
|
|
|
|
The read path was exercised against a live cPanel/MXlogin IMAP account over verified TLS (port 993, Let's Encrypt cert):
|
|
- **Auth + connect:** email-as-username login over implicit TLS succeeded.
|
|
- **`list`:** returned real INBOX headers; `has_attachments` correct.
|
|
- **`get`:** returned the full plain-text body and base64 attachments that decoded to exact byte sizes (a cPanel config mail with a PNG + 3 `.mobileconfig` files — all 4 matched).
|
|
- **`search`:** `--from`/`--subject-contains` correctly narrowed results.
|
|
- **Seen-set:** `list --new` correctly returned 0 on first contact (baseline floor = max UID); `ack` succeeded; `folder_state`/`acked` rows correct.
|
|
- **Audit:** every command wrote an `audit_log` row; password confirmed encrypted at rest (no plaintext in the DB).
|
|
|
|
## Resolved within Phase 1: header-only fetch
|
|
|
|
The previously-flagged "header listing downloads full message bodies" issue is **fixed** (`internal/mail/imap.go`). `list` and `search` now fetch `BODY.PEEK[HEADER]` + `BODYSTRUCTURE` instead of the whole RFC822 message, so listing a large mailbox no longer downloads every body and attachment. Header parsing reuses the same go-message path (RFC2047 decoding and address formatting unchanged); `has_attachments` is derived from the body-structure tree. `get` still fetches the full message. Verified: output is identical to the prior full-fetch behaviour against the live account, new unit tests (`TestParseHeaderBytes`, `TestHasAttachment`) pass, and the fetch loops were hardened to drain the IMAP channel on error (race-tested).
|
|
|
|
```
|
|
go build (CGO_ENABLED=0) → OK, single static binary
|
|
go vet ./... → clean
|
|
go test ./... → all packages pass
|
|
```
|
|
|
|
## What was built
|
|
|
|
| Package | Responsibility | Status |
|
|
|---|---|---|
|
|
| `internal/version` | Build version string + `version` command | ✅ |
|
|
| `internal/crypto` | AES-256-GCM field encryption keyed from `EMCLI_KEY` (fail-closed) | ✅ |
|
|
| `internal/store` | Pure-Go SQLite: schema v1, accounts CRUD (encrypted secrets), whitelists, **seen-set read state** (floor + acked + compaction + UIDVALIDITY reset), audit log + retention purge | ✅ |
|
|
| `internal/policy` | Pure enforcement: case-insensitive address/domain matching, inbound whitelist + subject-regex filter | ✅ |
|
|
| `internal/mail` | RFC822 parsing (headers/body/attachments) + IMAP client (select/fetch/search, read-only, Peek) | ✅ |
|
|
| `internal/cli` | JSON envelope, agent commands `list`/`get`/`search`/`ack` (policy-gated), flag-based admin (`account`, `whitelist`), command router | ✅ |
|
|
| `cmd/emcli` | Entry point routing to `cli.Run` | ✅ |
|
|
|
|
### Working commands (verified via smoke test)
|
|
- **Agent (JSON-only output, exit code mirrors error):**
|
|
- `emcli list --account <a> [--new] [--before/--since <uid>] [--limit N]`
|
|
- `emcli get --account <a> --uid <uid>`
|
|
- `emcli search --account <a> [--from --subject-contains --text --since-date --before-date]`
|
|
- `emcli ack --account <a> --uid-list 1,2,3`
|
|
- **Admin (human-readable):**
|
|
- `emcli account add|list`, `emcli whitelist in|out add|remove|list`
|
|
- `emcli version`
|
|
|
|
### Security invariant — independently verified in the final review
|
|
Mail that fails the inbound policy (`whitelist_in` / `subject_regex`) is **invisible** on every path: excluded from `list`/`search`, returns generic `not_found` on `get` (body never fetched), and **cannot be `ack`ed** (no state manipulation for unseen mail). Filtered vs. genuinely-absent UIDs are indistinguishable to the agent. Credentials/secrets never appear in stdout, the JSON envelope, or the audit log; passwords are sealed at rest (ciphertext ≠ plaintext confirmed by test).
|
|
|
|
## Process
|
|
|
|
Executed via subagent-driven development: a fresh implementer per task (TDD, commit per task), a separate reviewer per task (spec + quality gates), fix sub-agents for Critical/Important findings, and a final whole-branch review on the most capable model. Durable progress tracked in `.superpowers/sdd/progress.md`. 20 commits on `main` (`04d3b61..6061bd2`).
|
|
|
|
**Fixes applied during review (all re-verified):**
|
|
- `store`: pinned SQLite pool to one connection so `PRAGMA foreign_keys` (cascade deletes) stays effective; added a cascade test.
|
|
- `mail`: propagate `io.ReadAll` errors when parsing message parts and reading bodies off the network (prevents silent truncation); apply the `search` limit.
|
|
- `cli`: `AckCmd` reuses the folder UID-validity from `setup` (was swallowing a second `SelectFolder` error → epoch-0 acks); agent commands now **exit non-zero** when the JSON envelope reports an error (spec §8); `search` limit now counts *visible* results (filter before cap), consistent with `list`.
|
|
|
|
## Known limitations / deferred (not defects)
|
|
|
|
### Deliberately out of scope for Phase 1 (have their own future plans)
|
|
- **Phase 2 — Send path:** SMTP, MIME building, reply threading, outbound policy (RO/RW + whitelist-out), `send`.
|
|
- **Phase 3 — OAuth2:** Gmail loopback consent + token refresh (schema columns already present).
|
|
- **Phase 4 — Admin TUI + `doctor`:** interactive init/reconfigure, connectivity diagnostics.
|
|
|
|
### Carry-over items to address (logged from reviews)
|
|
1. **[Minor] Audit completeness:** `list`/`search` don't write a per-filtered-message audit row; some IMAP-error envelope paths don't write an audit row; `imap_error` is an audit reason not enumerated in the spec.
|
|
2. **[Minor] CLI polish:** `account add` opens the store before parsing its own flags (`--help` touches the DB); invalid `--since-date`/`--before-date` are silently ignored rather than erroring; `whitelist add/remove` don't reject an empty `--address` at the CLI layer.
|
|
3. **[Minor] Performance:** `ack` fetches each UID's header in its own round-trip; `get` fetches the header then re-fetches the full message. Fine for a per-invocation CLI; batch ops in later phases may want to optimize.
|
|
|
|
### Previously-open items now closed
|
|
- ✅ **Live IMAP validation** — done against a real mailbox over verified TLS (see "Live validation" above). The earlier GreenMail attempt had failed only on local self-signed-TLS friction, not a code issue.
|
|
- ✅ **Header-only fetch** — `list`/`search` no longer download message bodies (see "Resolved within Phase 1" above).
|
|
- A dev-only `--insecure`/skip-verify option is still worth adding later for local self-signed test servers, but is not needed for real providers (valid certs verified fine).
|
|
|
|
## Suggested next steps
|
|
|
|
1. Proceed to **Phase 2 (send path)** — the structure is ready: `policy` is set up to host outbound checks, `Account.Mode` is loaded for RO rejection, and `MatchAddress` already implements the domain matching whitelist-out needs. `--reply-to` can reuse the new header-only fetch path to read `Message-ID`/`References` cheaply.
|
|
2. Optionally clean up the Minor carry-over items above as part of Phase 2 touch-ups.
|
|
|
|
## Repository layout
|
|
|
|
```
|
|
emcli/
|
|
├── Makefile # build/test/vet (CGO_ENABLED=0)
|
|
├── go.mod / go.sum
|
|
├── cmd/emcli/main.go
|
|
├── internal/
|
|
│ ├── version/
|
|
│ ├── crypto/
|
|
│ ├── store/ # store, schema, settings, account, whitelist, seenset, audit
|
|
│ ├── policy/ # policy (address), inbound
|
|
│ ├── mail/ # message, imap, testdata/*.eml
|
|
│ └── cli/ # envelope, agent, dispatch, run, admin
|
|
└── specifications/
|
|
├── PRD.md
|
|
├── SPEC.md
|
|
├── PHASE1-STATUS.md # this file
|
|
└── plans/2026-06-21-phase1-foundation-and-read-path.md
|
|
```
|