From d023df1b4ad110a0ff4b0c2f49e5e97fb7003d82 Mon Sep 17 00:00:00 2001 From: Steve Cliff Date: Tue, 7 Jul 2026 14:28:38 +0100 Subject: [PATCH] feat(cli): folders command, search --all-folders, empty-search hint Fixes from testing email search (docs/enhancements-2026-07-07.md): - New `folders` agent command lists the account's mailboxes (name, delimiter, selectable), INBOX first, so agents can discover archived mail outside INBOX. - `search --all-folders` sweeps every selectable mailbox; each hit carries a `folder` field, `skipped_folders` reports mailboxes the server refused, and --limit caps visible results across the sweep. The sweep deliberately skips EnsureFolderBaseline so a read-only search never mutates list --new state. - Empty search results include a generic `data.hint` with next steps. The hint is a fixed constant per mode, so the invisibility invariant holds: absent and policy-filtered mail produce byte-identical envelopes (codified in TestSearchEmptyHintIndistinguishableFromFiltered). - Skill and user docs: document `--text` full-text search as best-effort (server-dependent); recommend --subject-contains/--from. Co-Authored-By: Claude Fable 5 --- USER-MANUAL.md | 42 +++- docs/enhancements-2026-07-07.md | 25 +++ internal/cli/agent.go | 122 ++++++++++-- internal/cli/agent_test.go | 253 ++++++++++++++++++++++++- internal/cli/help.go | 3 +- internal/cli/help_test.go | 2 +- internal/cli/role_test.go | 2 +- internal/cli/run.go | 55 +++++- internal/cli/run_test.go | 44 +++++ internal/mail/imap.go | 39 ++++ internal/mail/imap_integration_test.go | 19 ++ skills/emcli/AGENTIC-MANUAL.md | 2 +- skills/emcli/SKILL.md | 20 +- skills/emcli/references/commands.md | 51 ++++- 14 files changed, 645 insertions(+), 34 deletions(-) create mode 100644 docs/enhancements-2026-07-07.md diff --git a/USER-MANUAL.md b/USER-MANUAL.md index dbd3327..309c4a6 100644 --- a/USER-MANUAL.md +++ b/USER-MANUAL.md @@ -36,7 +36,7 @@ This manual is for **using and administering** `emcli`. It assumes you have the - **Admin commands** (`init`, `account add/edit/remove`, `whitelist`, `config`, `audit`) require `EMCLI_ADMIN_KEY` and are for *you*, the human. They print human-readable text or open an interactive form. (`account list` is the one exception — it is also an agent command; see below.) -- **Agent commands** (`list`, `get`, `search`, `ack`, `send`, `doctor`) require `EMCLI_KEY` (or +- **Agent commands** (`list`, `get`, `search`, `folders`, `ack`, `send`, `doctor`) require `EMCLI_KEY` (or `EMCLI_ADMIN_KEY` as a superset) and are for the *agent*. They print one line of JSON and nothing else, so a program can consume them reliably. (`doctor` prints human-readable text but is authorised by the agent key — `EMCLI_KEY` alone is sufficient; `EMCLI_ADMIN_KEY` also works @@ -103,7 +103,7 @@ reconfigure accounts, whitelists, or audit settings. | Key | Holder | Authorises | |---|---|---| | `EMCLI_ADMIN_KEY` | Human / secrets manager | ALL commands (`account`, `whitelist`, `config`, `audit`, `init`, plus all agent commands) | -| `EMCLI_KEY` | Agent orchestrator | Agent commands only (`list`, `get`, `search`, `ack`, `send`, `doctor`) | +| `EMCLI_KEY` | Agent orchestrator | Agent commands only (`list`, `get`, `search`, `folders`, `ack`, `send`, `doctor`) | `EMCLI_ADMIN_KEY` is a strict superset: a process with only the admin key can run agent commands too. A process with only `EMCLI_KEY` is refused with `emcli: this command requires EMCLI_ADMIN_KEY @@ -125,7 +125,7 @@ DEK for an admin command, even if it somehow knows the agent key. | Command | Role required | |---|---| -| `list`, `get`, `search`, `ack`, `send`, `doctor`, `account list` | Agent (`EMCLI_KEY` or `EMCLI_ADMIN_KEY`) | +| `list`, `get`, `search`, `folders`, `ack`, `send`, `doctor`, `account list` | Agent (`EMCLI_KEY` or `EMCLI_ADMIN_KEY`) | | `account add/edit/remove`, `whitelist`, `config`, `audit` | Admin (`EMCLI_ADMIN_KEY` required) | `account list` is dual-role: with the admin key it prints the full `NAME MODE IMAP USER` table; @@ -353,7 +353,7 @@ Clear it by passing an empty string (`--subject-regex ''`) or using the interact These are what the agent runs. Each prints exactly one JSON object (see section 8). They all take `--account` and most take `--folder` (default `INBOX`). -> **Reading never changes state.** `list`, `get`, and `search` are read-only. Only `ack` advances +> **Reading never changes state.** `list`, `get`, `search`, and `folders` are read-only. Only `ack` advances > "what's been processed." ### `list` — message headers @@ -392,19 +392,44 @@ emcli search --account gmail --from boss@example.com emcli search --account gmail --subject-contains invoice emcli search --account gmail --text "quarterly report" emcli search --account gmail --since-date 2026-01-01T00:00:00Z --before-date 2026-02-01T00:00:00Z +emcli search --account gmail --subject-contains invoice --all-folders # sweep every mailbox ``` | Flag | Meaning | |---|---| | `--from` | Sender contains | | `--subject-contains` | Subject contains | -| `--text` | Full-text search | +| `--text` | Full-text search (best-effort — see below) | +| `--all-folders` | Search every selectable mailbox (mutually exclusive with `--folder`) | | `--since-date` / `--before-date` | Date bounds, RFC 3339 (e.g. `2026-06-01T00:00:00Z`) | | `--limit` | Max results (default `50`) | Returns the same headers-only shape as `list`. Searches the whole folder, regardless of new/acked state. Filtered (whitelisted-out) mail never appears. +With `--all-folders` each returned message also carries a `folder` field (UIDs are only unique +per folder — pass it back via `--folder` when using `get`/`ack` on a hit), and +`data.skipped_folders` lists any mailboxes the server refused to search. `--limit` caps total +results across the sweep. + +`--text` depends on the server having a full-text index; some IMAP servers return zero results +for mail that `--subject-contains` finds. Prefer `--subject-contains`/`--from`; treat `--text` as +best-effort. + +When a search returns no messages, `data.hint` suggests next steps (list folders, try +`--all-folders`, adjust criteria). The hint is deliberately generic: it is identical whether +matching mail is absent, in another folder, or hidden by the inbound whitelist. + +### `folders` — list mailboxes + +```bash +emcli folders --account gmail +``` + +Returns every mailbox the account can see: `name`, `delimiter`, and `selectable` (folders with +`selectable: false` are hierarchy placeholders and can't be used with `--folder`). INBOX sorts +first. Useful before searching mail that may be archived outside INBOX. + ### `ack` — mark messages processed ```bash @@ -543,7 +568,7 @@ server settings are right. ### Audit log -Every agent action (`list`, `get`, `search`, `ack`, `send`) — allowed or blocked — is recorded. +Every agent action (`list`, `get`, `search`, `folders`, `ack`, `send`) — allowed or blocked — is recorded. ```bash emcli audit list # most recent 50 @@ -573,7 +598,7 @@ are rejected by `config get` and `config set`. ## 11. Troubleshooting **"EMCLI_KEY is not set" / "must be base64 of exactly 32 bytes".** Set `EMCLI_KEY` to a valid -base64-encoded 32-byte key (section 2). Agent commands (`list`, `get`, `search`, `ack`, `send`, +base64-encoded 32-byte key (section 2). Agent commands (`list`, `get`, `search`, `folders`, `ack`, `send`, `doctor`) need this key. **"this command requires EMCLI_ADMIN_KEY (admin privilege)".** Set `EMCLI_ADMIN_KEY` (section 2). @@ -633,7 +658,8 @@ emcli version emcli doctor [N | --account N] # connectivity/auth check emcli list --account N [--folder F] [--new] [--limit K] [--before U] [--since U] emcli get --account N [--folder F] --uid U -emcli search --account N [--folder F] [--from A] [--subject-contains S] [--text S] [--since-date D] [--before-date D] [--limit K] +emcli search --account N [--folder F | --all-folders] [--from A] [--subject-contains S] [--text S] [--since-date D] [--before-date D] [--limit K] +emcli folders --account N emcli ack --account N [--folder F] --uid-list U1,U2,U3 emcli send --account N --to A [--cc A] [--bcc A] --subject S --body B [--attach P]… [--reply-to U [--folder F]] ``` diff --git a/docs/enhancements-2026-07-07.md b/docs/enhancements-2026-07-07.md new file mode 100644 index 0000000..96e1cae --- /dev/null +++ b/docs/enhancements-2026-07-07.md @@ -0,0 +1,25 @@ +Fixes identified from testing emcli email search (Morphy Richards purchase investigation). + +1. No folder discovery command + +emcli has no folders command, so the agent cannot enumerate available mailboxes. Searches default to INBOX only and miss archived mail (e.g. INBOX.Archive). + +Fix: Add a folders agent command to list available mailboxes. + +2. INBOX-only default too narrow + +When searching for a specific email, the agent should ideally search across all folders, but cannot without knowing folder names. + +Fix: Add an --all-folders flag to search to sweep all mailboxes at once. + +3. --text full-text search unreliable on some IMAP servers + +On myfamilyemail.co.uk, --text returned zero results for emails that --subject-contains found fine. The IMAP server either does not support full-text search or has an incomplete index. + +Fix: Document this limitation in the skill; recommend --subject-contains as the primary search method and treat --text as best-effort. + +4. No way to distinguish filtered vs missing vs wrong-folder + +All three cases return empty results with no error, making it impossible for the agent to know whether mail is absent, hidden by a whitelist, or simply in another folder. + +Fix: emcli (or the skill) should surface a diagnostic hint when results are empty — e.g. suggest running folders or retrying with --all-folders. diff --git a/internal/cli/agent.go b/internal/cli/agent.go index e5764b7..5a2d10f 100644 --- a/internal/cli/agent.go +++ b/internal/cli/agent.go @@ -21,6 +21,7 @@ type Mailer interface { FetchHeadersRange(folder string, since, before uint32, limit int) ([]mail.Header, error) FetchFull(folder string, uid uint32) (mail.Message, error) Search(folder string, sc mail.SearchCriteria, limit int) ([]mail.Header, error) + ListFolders() ([]mail.FolderInfo, error) Logout() error } @@ -50,18 +51,18 @@ func (d Deps) audit(account, action, target, result, reason string) { }) } -// setup loads the account, builds the inbound rule, dials IMAP, and selects the -// folder (establishing the baseline). Returns a cleanup func. -func (d Deps) setup(account, folder string) (store.Account, policy.InboundRule, Mailer, uint32, func(), *Envelope) { +// connect loads the account, builds the inbound rule, and dials IMAP — no +// folder selection and no baseline side effects. +func (d Deps) connect(account string) (store.Account, policy.InboundRule, Mailer, *Envelope) { acc, err := d.Store.GetAccount(account) if err != nil { e := Failure(CodeNotFound, "account not found: "+account) - return acc, policy.InboundRule{}, nil, 0, nil, &e + return acc, policy.InboundRule{}, nil, &e } re, err := policy.CompileSubject(acc.SubjectRegex) if err != nil { e := Failure(CodeConfig, "invalid subject_regex: "+err.Error()) - return acc, policy.InboundRule{}, nil, 0, nil, &e + return acc, policy.InboundRule{}, nil, &e } wlIn, _ := d.Store.ListWhitelist(account, store.DirIn) rule := policy.InboundRule{ @@ -72,7 +73,17 @@ func (d Deps) setup(account, folder string) (store.Account, policy.InboundRule, m, err := d.Dial(acc) if err != nil { e := Failure(CodeNetwork, "imap connect failed: "+err.Error()) - return acc, rule, nil, 0, nil, &e + return acc, rule, nil, &e + } + return acc, rule, m, nil +} + +// setup loads the account, builds the inbound rule, dials IMAP, and selects the +// folder (establishing the baseline). Returns a cleanup func. +func (d Deps) setup(account, folder string) (store.Account, policy.InboundRule, Mailer, uint32, func(), *Envelope) { + acc, rule, m, fail := d.connect(account) + if fail != nil { + return acc, rule, nil, 0, nil, fail } uidv, maxUID, err := m.SelectFolder(folder) if err != nil { @@ -184,6 +195,41 @@ func GetCmd(d Deps, account, folder string, uid uint32) error { })) } +// Empty-search hints. These MUST stay constant per command variant: the same +// text is emitted whether matching mail is absent, in another folder, or +// hidden by inbound policy, so an empty result reveals nothing about +// filtering (see the invisibility invariant in the skill docs). +const ( + searchEmptyHint = "no matches in this folder; mail may be in another mailbox — " + + "run 'folders' to list mailboxes or retry with --all-folders; " + + "note the account's inbound policy may hide some messages " + + "(hidden and non-existent messages are indistinguishable)" + searchAllEmptyHint = "no matches in any folder; try adjusting criteria — " + + "--subject-contains is more reliable than --text on some servers; " + + "note the account's inbound policy may hide some messages " + + "(hidden and non-existent messages are indistinguishable)" +) + +// appendVisible appends policy-visible headers to out, up to limit total +// visible results (0 = uncapped). A non-empty folder tags each message with +// its mailbox (needed when sweeping: UIDs are only unique per folder). +func appendVisible(out []map[string]any, rule policy.InboundRule, headers []mail.Header, limit int, folder string) []map[string]any { + for _, h := range headers { + if !rule.Allows(h.From, h.Subject) { + continue + } + m := headerMap(h) + if folder != "" { + m["folder"] = folder + } + out = append(out, m) + if limit > 0 && len(out) >= limit { + break + } + } + return out +} + func SearchCmd(d Deps, account, folder string, sc mail.SearchCriteria, limit int) error { _, rule, m, _, done, fail := d.setup(account, folder) if fail != nil { @@ -194,18 +240,70 @@ func SearchCmd(d Deps, account, folder string, sc mail.SearchCriteria, limit int if err != nil { return d.emit(Failure(CodeNetwork, err.Error())) } - out := make([]map[string]any, 0, len(headers)) - for _, h := range headers { - if !rule.Allows(h.From, h.Subject) { + out := appendVisible(make([]map[string]any, 0, len(headers)), rule, headers, limit, "") + d.audit(account, "search", folder, "allowed", "") + data := map[string]any{"messages": out} + if len(out) == 0 { + data["hint"] = searchEmptyHint + } + return d.emit(Success(data)) +} + +// SearchAllCmd sweeps every selectable folder. It deliberately skips +// EnsureFolderBaseline: baselining folders the agent never listed would mark +// their existing mail "not new" as a side effect of a read command. +func SearchAllCmd(d Deps, account string, sc mail.SearchCriteria, limit int) error { + _, rule, m, fail := d.connect(account) + if fail != nil { + return d.emit(*fail) + } + defer m.Logout() + folders, err := m.ListFolders() + if err != nil { + return d.emit(Failure(CodeNetwork, "list folders failed: "+err.Error())) + } + out := make([]map[string]any, 0) + skipped := make([]string, 0) + for _, f := range folders { + if !f.Selectable { continue } - out = append(out, headerMap(h)) if limit > 0 && len(out) >= limit { break } + headers, err := m.Search(f.Name, sc, 0) + if err != nil { + skipped = append(skipped, f.Name) + continue + } + out = appendVisible(out, rule, headers, limit, f.Name) } - d.audit(account, "search", folder, "allowed", "") - return d.emit(Success(map[string]any{"messages": out})) + d.audit(account, "search", "*", "allowed", "") + data := map[string]any{"messages": out, "skipped_folders": skipped} + if len(out) == 0 { + data["hint"] = searchAllEmptyHint + } + return d.emit(Success(data)) +} + +func FoldersCmd(d Deps, account string) error { + _, _, m, fail := d.connect(account) + if fail != nil { + return d.emit(*fail) + } + defer m.Logout() + folders, err := m.ListFolders() + if err != nil { + return d.emit(Failure(CodeNetwork, err.Error())) + } + out := make([]map[string]any, 0, len(folders)) + for _, f := range folders { + out = append(out, map[string]any{ + "name": f.Name, "delimiter": f.Delimiter, "selectable": f.Selectable, + }) + } + d.audit(account, "folders", "", "allowed", "") + return d.emit(Success(map[string]any{"folders": out})) } func AckCmd(d Deps, account, folder string, uids []uint32) error { diff --git a/internal/cli/agent_test.go b/internal/cli/agent_test.go index f1b4582..0e0f7e4 100644 --- a/internal/cli/agent_test.go +++ b/internal/cli/agent_test.go @@ -16,6 +16,13 @@ type fakeMailer struct { maxUID uint32 headers []mail.Header full map[uint32]mail.Message + + // Multi-folder fields, used by folders/--all-folders tests. When + // headersByFolder is non-nil Search consults it instead of headers. + folders []mail.FolderInfo + headersByFolder map[string][]mail.Header + searchErr map[string]error + searched []string } func (f *fakeMailer) SelectFolder(string) (uint32, uint32, error) { @@ -43,9 +50,19 @@ func (f *fakeMailer) FetchHeadersRange(string, uint32, uint32, int) ([]mail.Head func (f *fakeMailer) FetchFull(_ string, uid uint32) (mail.Message, error) { return f.full[uid], nil } -func (f *fakeMailer) Search(string, mail.SearchCriteria, int) ([]mail.Header, error) { +func (f *fakeMailer) Search(folder string, _ mail.SearchCriteria, _ int) ([]mail.Header, error) { + f.searched = append(f.searched, folder) + if err := f.searchErr[folder]; err != nil { + return nil, err + } + if f.headersByFolder != nil { + return f.headersByFolder[folder], nil + } return f.headers, nil } +func (f *fakeMailer) ListFolders() ([]mail.FolderInfo, error) { + return f.folders, nil +} func (f *fakeMailer) Logout() error { return nil } func testKey() []byte { @@ -219,3 +236,237 @@ func TestAckAdvancesStateAndFiltered(t *testing.T) { t.Fatalf("want 0 new messages, got %d", len(msgs)) } } + +func TestFoldersListsMailboxes(t *testing.T) { + fm := &fakeMailer{ + folders: []mail.FolderInfo{ + {Name: "INBOX", Delimiter: ".", Selectable: true}, + {Name: "INBOX.Archive", Delimiter: ".", Selectable: true}, + {Name: "Public", Delimiter: ".", Selectable: false}, + }, + } + d, buf := newDeps(t, fm) + if err := FoldersCmd(d, "work"); err != nil { + t.Fatalf("FoldersCmd: %v", err) + } + res := decode(t, buf.Bytes()) + if res["error"] != false { + t.Fatalf("unexpected error envelope: %v", res) + } + data := res["data"].(map[string]any) + folders := data["folders"].([]any) + if len(folders) != 3 { + t.Fatalf("want 3 folders, got %d: %v", len(folders), folders) + } + first := folders[0].(map[string]any) + if first["name"] != "INBOX" || first["delimiter"] != "." || first["selectable"] != true { + t.Fatalf("unexpected first folder: %v", first) + } + last := folders[2].(map[string]any) + if last["name"] != "Public" || last["selectable"] != false { + t.Fatalf("unexpected last folder: %v", last) + } +} + +func TestSearchAllFoldersTagsFolderAndFilters(t *testing.T) { + fm := &fakeMailer{ + folders: []mail.FolderInfo{ + {Name: "INBOX", Delimiter: ".", Selectable: true}, + {Name: "INBOX.Archive", Delimiter: ".", Selectable: true}, + }, + headersByFolder: map[string][]mail.Header{ + "INBOX": { + {UID: 1, From: "a@trusted.com", Subject: "one"}, + {UID: 2, From: "x@evil.com", Subject: "spam"}, // filtered + }, + "INBOX.Archive": { + {UID: 7, From: "b@trusted.com", Subject: "two"}, + }, + }, + } + d, buf := newDeps(t, fm) + if err := SearchAllCmd(d, "work", mail.SearchCriteria{}, 50); err != nil { + t.Fatalf("SearchAllCmd: %v", err) + } + res := decode(t, buf.Bytes()) + if res["error"] != false { + t.Fatalf("unexpected error envelope: %v", res) + } + data := res["data"].(map[string]any) + msgs := data["messages"].([]any) + if len(msgs) != 2 { + t.Fatalf("want 2 visible messages, got %d: %v", len(msgs), msgs) + } + m0 := msgs[0].(map[string]any) + m1 := msgs[1].(map[string]any) + if m0["folder"] != "INBOX" || m1["folder"] != "INBOX.Archive" { + t.Fatalf("wrong folder tags: %v / %v", m0["folder"], m1["folder"]) + } + skipped := data["skipped_folders"].([]any) + if len(skipped) != 0 { + t.Fatalf("want no skipped folders, got %v", skipped) + } +} + +func TestSearchAllFoldersSkipsNoselectAndErrors(t *testing.T) { + fm := &fakeMailer{ + folders: []mail.FolderInfo{ + {Name: "INBOX", Delimiter: ".", Selectable: true}, + {Name: "Broken", Delimiter: ".", Selectable: true}, + {Name: "Public", Delimiter: ".", Selectable: false}, + }, + headersByFolder: map[string][]mail.Header{ + "INBOX": {{UID: 1, From: "a@trusted.com", Subject: "one"}}, + }, + searchErr: map[string]error{"Broken": errCommandFailed}, + } + d, buf := newDeps(t, fm) + if err := SearchAllCmd(d, "work", mail.SearchCriteria{}, 50); err != nil { + t.Fatalf("SearchAllCmd: %v", err) + } + res := decode(t, buf.Bytes()) + if res["error"] != false { + t.Fatalf("unexpected error envelope: %v", res) + } + for _, f := range fm.searched { + if f == "Public" { + t.Fatal("searched a \\Noselect folder") + } + } + data := res["data"].(map[string]any) + if got := len(data["messages"].([]any)); got != 1 { + t.Fatalf("want 1 message, got %d", got) + } + skipped := data["skipped_folders"].([]any) + if len(skipped) != 1 || skipped[0] != "Broken" { + t.Fatalf("want skipped_folders [Broken], got %v", skipped) + } +} + +func TestSearchAllFoldersLimitCountsVisibleAcrossFolders(t *testing.T) { + fm := &fakeMailer{ + folders: []mail.FolderInfo{ + {Name: "INBOX", Delimiter: ".", Selectable: true}, + {Name: "INBOX.Archive", Delimiter: ".", Selectable: true}, + {Name: "INBOX.Sent", Delimiter: ".", Selectable: true}, + }, + headersByFolder: map[string][]mail.Header{ + "INBOX": { + {UID: 1, From: "a@trusted.com", Subject: "one"}, + {UID: 2, From: "x@evil.com", Subject: "spam"}, // filtered + }, + "INBOX.Archive": { + {UID: 3, From: "y@evil.com", Subject: "spam"}, // filtered + {UID: 4, From: "b@trusted.com", Subject: "two"}, + }, + "INBOX.Sent": { + {UID: 5, From: "c@trusted.com", Subject: "three"}, + }, + }, + } + d, buf := newDeps(t, fm) + if err := SearchAllCmd(d, "work", mail.SearchCriteria{}, 2); err != nil { + t.Fatalf("SearchAllCmd: %v", err) + } + res := decode(t, buf.Bytes()) + data := res["data"].(map[string]any) + msgs := data["messages"].([]any) + if len(msgs) != 2 { + t.Fatalf("want exactly 2 visible messages, got %d: %v", len(msgs), msgs) + } + for _, f := range fm.searched { + if f == "INBOX.Sent" { + t.Fatal("searched a folder after the visible limit was reached") + } + } +} + +func TestSearchAllFoldersDoesNotBaseline(t *testing.T) { + fm := &fakeMailer{ + folders: []mail.FolderInfo{ + {Name: "INBOX.Archive", Delimiter: ".", Selectable: true}, + }, + headersByFolder: map[string][]mail.Header{ + "INBOX.Archive": {{UID: 7, From: "b@trusted.com", Subject: "two"}}, + }, + } + d, _ := newDeps(t, fm) + if err := SearchAllCmd(d, "work", mail.SearchCriteria{}, 50); err != nil { + t.Fatalf("SearchAllCmd: %v", err) + } + // IsNew errors (no folder_state row) iff the sweep did not baseline. + if _, err := d.Store.IsNew("work", "INBOX.Archive", 7); err == nil { + t.Fatal("sweep must not create a folder_state baseline for swept folders") + } +} + +func TestSearchEmptyEmitsHint(t *testing.T) { + fm := &fakeMailer{uidValidity: 1, maxUID: 5} + d, buf := newDeps(t, fm) + if err := SearchCmd(d, "work", "INBOX", mail.SearchCriteria{}, 50); err != nil { + t.Fatalf("SearchCmd: %v", err) + } + data := decode(t, buf.Bytes())["data"].(map[string]any) + if hint, ok := data["hint"].(string); !ok || hint == "" { + t.Fatalf("want non-empty hint on empty result, got %v", data["hint"]) + } + + // Non-empty results must omit the hint. + fm2 := &fakeMailer{ + uidValidity: 1, maxUID: 5, + headers: []mail.Header{{UID: 1, From: "a@trusted.com", Subject: "one"}}, + } + d2, buf2 := newDeps(t, fm2) + if err := SearchCmd(d2, "work", "INBOX", mail.SearchCriteria{}, 50); err != nil { + t.Fatalf("SearchCmd: %v", err) + } + data2 := decode(t, buf2.Bytes())["data"].(map[string]any) + if _, ok := data2["hint"]; ok { + t.Fatalf("hint must be absent when messages exist, got %v", data2["hint"]) + } +} + +func TestSearchAllFoldersEmptyEmitsHint(t *testing.T) { + fm := &fakeMailer{ + folders: []mail.FolderInfo{{Name: "INBOX", Delimiter: ".", Selectable: true}}, + headersByFolder: map[string][]mail.Header{}, + } + d, buf := newDeps(t, fm) + if err := SearchAllCmd(d, "work", mail.SearchCriteria{}, 50); err != nil { + t.Fatalf("SearchAllCmd: %v", err) + } + data := decode(t, buf.Bytes())["data"].(map[string]any) + if hint, ok := data["hint"].(string); !ok || hint == "" { + t.Fatalf("want non-empty hint on empty sweep, got %v", data["hint"]) + } +} + +// TestSearchEmptyHintIndistinguishableFromFiltered codifies the invisibility +// invariant for the empty-search hint: the envelope for "no mail at all" must +// be byte-identical to the envelope for "mail exists but is all filtered", so +// an empty result reveals nothing about inbound policy. Do not weaken this to +// a structural comparison. +func TestSearchEmptyHintIndistinguishableFromFiltered(t *testing.T) { + absent := &fakeMailer{uidValidity: 1, maxUID: 5} + dAbsent, bufAbsent := newDeps(t, absent) + if err := SearchCmd(dAbsent, "work", "INBOX", mail.SearchCriteria{}, 50); err != nil { + t.Fatalf("SearchCmd (absent): %v", err) + } + + filtered := &fakeMailer{ + uidValidity: 1, maxUID: 5, + headers: []mail.Header{ + {UID: 1, From: "x@evil.com", Subject: "spam1"}, + {UID: 2, From: "y@evil.com", Subject: "spam2"}, + }, + } + dFiltered, bufFiltered := newDeps(t, filtered) + if err := SearchCmd(dFiltered, "work", "INBOX", mail.SearchCriteria{}, 50); err != nil { + t.Fatalf("SearchCmd (filtered): %v", err) + } + + if !bytes.Equal(bufAbsent.Bytes(), bufFiltered.Bytes()) { + t.Fatalf("empty-vs-filtered envelopes differ:\nabsent: %s\nfiltered: %s", + bufAbsent.Bytes(), bufFiltered.Bytes()) + } +} diff --git a/internal/cli/help.go b/internal/cli/help.go index 6fd96e3..5f60bbf 100644 --- a/internal/cli/help.go +++ b/internal/cli/help.go @@ -16,7 +16,8 @@ type cmdHelp struct { var agentCmds = []cmdHelp{ {"list", "list --account [--folder F] [--new] [--limit N] [--before U] [--since U]", "List message headers, newest first."}, {"get", "get --account [--folder F] --uid ", "Fetch one full message (body + attachments)."}, - {"search", "search --account [--folder F] [--from A] [--subject-contains S] [--text S] [--since-date D] [--before-date D] [--limit N]", "Server-side IMAP search."}, + {"search", "search --account [--folder F | --all-folders] [--from A] [--subject-contains S] [--text S] [--since-date D] [--before-date D] [--limit N]", "Server-side IMAP search."}, + {"folders", "folders --account ", "List the account's mailboxes/folders."}, {"ack", "ack --account [--folder F] --uid-list U1,U2,…", "Mark message(s) processed."}, {"send", "send --account --to A… [--cc A…] [--bcc A…] --subject S --body B [--attach P]… [--reply-to U [--folder F]]", "Send or reply (RW accounts only)."}, } diff --git a/internal/cli/help_test.go b/internal/cli/help_test.go index 740429a..71293f9 100644 --- a/internal/cli/help_test.go +++ b/internal/cli/help_test.go @@ -15,7 +15,7 @@ func TestMainHelpListsAllCommands(t *testing.T) { t.Fatalf("%v: want exit 0, got %d\n%s", args, code, text) } for _, want := range []string{ - "Usage", "list", "get", "search", "ack", "send", + "Usage", "list", "get", "search", "ack", "send", "folders", "account", "whitelist", "config", "audit", "doctor", "version", "EMCLI_KEY", "EMCLI_DB", } { diff --git a/internal/cli/role_test.go b/internal/cli/role_test.go index c62bcfa..e473926 100644 --- a/internal/cli/role_test.go +++ b/internal/cli/role_test.go @@ -14,7 +14,7 @@ func TestCommandRole(t *testing.T) { {"account"}, {"account", "add"}, {"account", "edit"}, {"account", "remove"}, } agentCmds := [][]string{ - {"list"}, {"get"}, {"search"}, {"ack"}, {"send"}, {"doctor"}, + {"list"}, {"get"}, {"search"}, {"ack"}, {"send"}, {"doctor"}, {"folders"}, {"account", "list"}, } for _, c := range adminCmds { diff --git a/internal/cli/run.go b/internal/cli/run.go index 23b79c7..638102b 100644 --- a/internal/cli/run.go +++ b/internal/cli/run.go @@ -165,6 +165,8 @@ func Run(args []string, out, errOut io.Writer) int { switch normalizeVerb(cmd) { case "list", "get", "search", "ack": return runAgent(normalizeVerb(cmd), rest, role, out, errOut) + case "folders": + return runFolders(rest, role, out, errOut) case "send": return runSend(rest, role, out, errOut) case "account": @@ -185,6 +187,37 @@ func Run(args []string, out, errOut io.Writer) int { } } +// runFolders handles `folders --account ` — list the account's mailboxes. +func runFolders(args []string, role store.Role, out, errOut io.Writer) int { + fs := flag.NewFlagSet("folders", flag.ContinueOnError) + fs.SetOutput(errOut) + usageFlags(fs, "folders", errOut) + account := fs.String("account", "", "account name") + if err := fs.Parse(args); err != nil { + if errors.Is(err, flag.ErrHelp) { + return 0 + } + _ = Failure(CodeUsage, err.Error()).Write(out) + return 2 + } + if *account == "" { + _ = Failure(CodeUsage, "--account is required").Write(out) + return 2 + } + st, err := openStore(role) + if err != nil { + _ = Failure(CodeConfig, err.Error()).Write(out) + return 1 + } + defer st.Close() + _, _ = st.PurgeAudit(time.Now()) + d := newDepsLive(st, out) + if err := FoldersCmd(d, *account); err != nil { + return 1 + } + return 0 +} + // runAgent handles JSON-emitting commands. Errors are emitted as JSON envelopes. func runAgent(cmd string, args []string, role store.Role, out, errOut io.Writer) int { fs := flag.NewFlagSet(cmd, flag.ContinueOnError) @@ -203,6 +236,10 @@ func runAgent(cmd string, args []string, role store.Role, out, errOut io.Writer) sinceDate := fs.String("since-date", "", "search: RFC3339 date lower bound") beforeDate := fs.String("before-date", "", "search: RFC3339 date upper bound") ackUIDs := fs.String("uid-list", "", "ack: comma-separated UIDs") + allFolders := new(bool) + if cmd == "search" { + allFolders = fs.Bool("all-folders", false, "search: sweep all mailboxes") + } if err := fs.Parse(args); err != nil { if errors.Is(err, flag.ErrHelp) { return 0 // usage already printed to stderr; help isn't an error @@ -217,6 +254,18 @@ func runAgent(cmd string, args []string, role store.Role, out, errOut io.Writer) _ = Failure(CodeUsage, "--account is required").Write(out) return 2 } + if *allFolders { + folderSet := false + fs.Visit(func(f *flag.Flag) { + if f.Name == "folder" { + folderSet = true + } + }) + if folderSet { + _ = Failure(CodeUsage, "--folder and --all-folders are mutually exclusive").Write(out) + return 2 + } + } st, err := openStore(role) if err != nil { _ = Failure(CodeConfig, err.Error()).Write(out) @@ -251,7 +300,11 @@ func runAgent(cmd string, args []string, role store.Role, out, errOut io.Writer) sc.Before = tm } } - if err := SearchCmd(d, *account, *folder, sc, *limit); err != nil { + if *allFolders { + if err := SearchAllCmd(d, *account, sc, *limit); err != nil { + return 1 + } + } else if err := SearchCmd(d, *account, *folder, sc, *limit); err != nil { return 1 } case "ack": diff --git a/internal/cli/run_test.go b/internal/cli/run_test.go index e9a04ec..cc8fa25 100644 --- a/internal/cli/run_test.go +++ b/internal/cli/run_test.go @@ -121,3 +121,47 @@ func TestAccountLsAliasAgentRole(t *testing.T) { t.Fatalf("account ls leaked host/username:\n%s", out) } } + +func TestSearchAllFoldersConflictsWithFolder(t *testing.T) { + adminEnv(t) + code, out, _ := run(t, "search", "--account", "x", "--folder", "F", "--all-folders") + if code != 2 { + t.Fatalf("want usage exit 2, got %d (out=%q)", code, out) + } + var env map[string]any + if err := json.Unmarshal([]byte(out), &env); err != nil { + t.Fatalf("usage error must be a JSON envelope, got %q", out) + } + if env["error"] != true { + t.Fatalf("want error envelope: %v", env) + } +} + +func TestAllFoldersRejectedForList(t *testing.T) { + adminEnv(t) + // --all-folders is only registered for search; list must reject it at + // flag-parse time with a JSON usage envelope. + code, out, _ := run(t, "list", "--account", "x", "--all-folders") + if code != 2 { + t.Fatalf("want usage exit 2, got %d (out=%q)", code, out) + } + var env map[string]any + if err := json.Unmarshal([]byte(out), &env); err != nil { + t.Fatalf("usage error must be a JSON envelope, got %q", out) + } + if env["error"] != true { + t.Fatalf("want error envelope: %v", env) + } +} + +func TestFoldersRequiresAccount(t *testing.T) { + adminEnv(t) + code, out, _ := run(t, "folders") + if code != 2 { + t.Fatalf("want usage exit 2, got %d (out=%q)", code, out) + } + var env map[string]any + if err := json.Unmarshal([]byte(out), &env); err != nil { + t.Fatalf("usage error must be a JSON envelope, got %q", out) + } +} diff --git a/internal/mail/imap.go b/internal/mail/imap.go index bf1ac07..ed94741 100644 --- a/internal/mail/imap.go +++ b/internal/mail/imap.go @@ -52,6 +52,45 @@ func Dial(cfg IMAPConfig) (*Client, error) { func (c *Client) Logout() error { return c.c.Logout() } +// FolderInfo describes one mailbox as reported by the server. Selectable is +// false for hierarchy-only entries (\Noselect) that cannot be passed to SELECT. +type FolderInfo struct { + Name string + Delimiter string + Selectable bool +} + +// ListFolders enumerates all mailboxes, INBOX first, then case-insensitive +// name order. +func (c *Client) ListFolders() ([]FolderInfo, error) { + infoCh := make(chan *imap.MailboxInfo, 16) + done := make(chan error, 1) + go func() { done <- c.c.List("", "*", infoCh) }() + + var out []FolderInfo + for info := range infoCh { + selectable := true + for _, attr := range info.Attributes { + if strings.EqualFold(attr, imap.NoSelectAttr) { + selectable = false + break + } + } + out = append(out, FolderInfo{Name: info.Name, Delimiter: info.Delimiter, Selectable: selectable}) + } + if err := <-done; err != nil { + return nil, err + } + sort.Slice(out, func(i, j int) bool { + ii, ji := out[i].Name == "INBOX", out[j].Name == "INBOX" + if ii != ji { + return ii + } + return strings.ToLower(out[i].Name) < strings.ToLower(out[j].Name) + }) + return out, nil +} + func (c *Client) SelectFolder(folder string) (uint32, uint32, error) { mbox, err := c.c.Select(folder, true) // read-only select if err != nil { diff --git a/internal/mail/imap_integration_test.go b/internal/mail/imap_integration_test.go index b5e7cca..3669187 100644 --- a/internal/mail/imap_integration_test.go +++ b/internal/mail/imap_integration_test.go @@ -49,6 +49,25 @@ func TestSelectAndFetch(t *testing.T) { t.Logf("inbox has %d messages, maxUID=%d", len(headers), maxUID) } +func TestListFolders(t *testing.T) { + c, err := Dial(testCfg(t)) + if err != nil { + t.Fatalf("Dial: %v", err) + } + defer c.Logout() + + folders, err := c.ListFolders() + if err != nil { + t.Fatalf("ListFolders: %v", err) + } + if len(folders) == 0 { + t.Fatal("expected at least one folder") + } + if folders[0].Name != "INBOX" || !folders[0].Selectable { + t.Fatalf("expected selectable INBOX first, got %+v", folders[0]) + } +} + func atoiEnv(t *testing.T, k string) int { t.Helper() n, err := strconv.Atoi(os.Getenv(k)) diff --git a/skills/emcli/AGENTIC-MANUAL.md b/skills/emcli/AGENTIC-MANUAL.md index e0a6cca..806d62b 100644 --- a/skills/emcli/AGENTIC-MANUAL.md +++ b/skills/emcli/AGENTIC-MANUAL.md @@ -68,7 +68,7 @@ CGO_ENABLED=0 go build -o emcli ./cmd/emcli # then move ./emcli onto your PATH ## 3. Confirm the agent key is present emcli uses two keys; **you (the agent) are given only `EMCLI_KEY`** (the agent key). It authorises -`list`, `get`, `search`, `ack`, `send`, and `doctor`. Admin commands require `EMCLI_ADMIN_KEY`, +`list`, `get`, `search`, `folders`, `ack`, `send`, and `doctor`. Admin commands require `EMCLI_ADMIN_KEY`, which the human holds — attempting admin commands with only `EMCLI_KEY` is refused by `emcli`. For agent use, **the orchestrator that launched you provides `EMCLI_KEY`** in the environment. diff --git a/skills/emcli/SKILL.md b/skills/emcli/SKILL.md index f99139d..026c9c2 100644 --- a/skills/emcli/SKILL.md +++ b/skills/emcli/SKILL.md @@ -17,8 +17,8 @@ sets its exit code to match. ## Security model — read this first -- **You only run agent commands:** `list`, `get`, `search`, `ack`, `send`, `doctor`, and - `account list` (to discover accounts). You are provided only `EMCLI_KEY` (the agent key), which +- **You only run agent commands:** `list`, `get`, `search`, `folders`, `ack`, `send`, `doctor`, + and `account list` (to discover accounts). You are provided only `EMCLI_KEY` (the agent key), which authorises these and nothing else. Account *setup* (`account add/edit/remove`), passwords, whitelists, and config are the **user's** job (admin commands that require `EMCLI_ADMIN_KEY`) — do not run or suggest running `account add/edit/remove`, `whitelist`, `config`, `audit`, or @@ -117,7 +117,8 @@ read-only — tell the user; do not attempt another account without their say-so |---|---| | `emcli list --account A [--folder F] [--new] [--limit N] [--before U] [--since U]` | Message headers, newest first | | `emcli get --account A [--folder F] --uid U` | One full message (body + attachments) | -| `emcli search --account A [--folder F] [--from X] [--subject-contains X] [--text X] [--since-date D] [--before-date D]` | Server-side search | +| `emcli search --account A [--folder F \| --all-folders] [--from X] [--subject-contains X] [--text X] [--since-date D] [--before-date D]` | Server-side search | +| `emcli folders --account A` | List the account's mailboxes/folders | | `emcli ack --account A [--folder F] --uid-list U1,U2` | Mark message(s) processed | | `emcli send --account A --to X [--cc X] [--bcc X] --subject S --body B [--attach P]… [--reply-to U]` | Send / reply | | `emcli account list` | Discover accounts: JSON `name` / `from` / `can_send` per account | @@ -125,6 +126,19 @@ read-only — tell the user; do not attempt another account without their say-so Defaults: `--folder INBOX`, `--limit 50` (max 500). Dates are RFC 3339 (e.g. `2026-06-01T00:00:00Z`). UIDs come from `list`/`search` output. +### Searching effectively + +- `list`/`search` cover **one folder** (INBOX by default). To find mail that may be archived or + sorted elsewhere, run `emcli folders --account A` to see the mailboxes, or search with + `--all-folders` to sweep them all (each hit then carries a `folder` field — pass it back via + `--folder` when you `get`/`ack` that message). +- Prefer `--subject-contains` and `--from`. `--text` (full-text) is **best-effort**: some IMAP + servers have no body index and return zero results for mail that `--subject-contains` finds — + a `--text` miss does not mean the mail is absent. +- An empty search result includes a `data.hint` with next steps — follow it. Some mail may be + hidden by the user's inbound policy; that's normal and indistinguishable from mail that doesn't + exist. Don't probe for hidden mail. + **Full reference** (every flag, exact JSON shapes for each command, attachment encoding, error codes, and the enforcement rules): `references/commands.md` — read it from disk, or fetch it from the raw base URL in "Files & first run" above if you don't have it locally. diff --git a/skills/emcli/references/commands.md b/skills/emcli/references/commands.md index 94748b4..34f29d4 100644 --- a/skills/emcli/references/commands.md +++ b/skills/emcli/references/commands.md @@ -1,6 +1,6 @@ # emcli agent command reference -The five agent commands you may use. Each prints **one** JSON object to stdout and sets a matching +The six agent commands you may use. Each prints **one** JSON object to stdout and sets a matching exit code (0 success, non-zero error). All take `--account `; most take `--folder` (default `INBOX`). @@ -76,10 +76,30 @@ emcli get --account A [--folder F] --uid U --- -## `search` — server-side search (whole folder) +## `folders` — list mailboxes ``` -emcli search --account A [--folder F] [--from X] [--subject-contains X] [--text X] \ +emcli folders --account A +``` + +`data`: +```json +{ "folders": [ + { "name": "INBOX", "delimiter": ".", "selectable": true }, + { "name": "INBOX.Archive", "delimiter": ".", "selectable": true } ] } +``` + +- INBOX first, then case-insensitive name order. +- `selectable: false` marks hierarchy-only entries — don't pass those to `--folder`. +- Use this before searching outside INBOX: archived mail often lives in folders like + `INBOX.Archive` that `list`/`search` won't touch by default. + +--- + +## `search` — server-side search (one folder, or all) + +``` +emcli search --account A [--folder F | --all-folders] [--from X] [--subject-contains X] [--text X] \ [--since-date D] [--before-date D] [--limit N] ``` @@ -87,13 +107,33 @@ emcli search --account A [--folder F] [--from X] [--subject-contains X] [--text |---|---| | `--from` | Sender contains | | `--subject-contains` | Subject contains | -| `--text` | Full-text | +| `--text` | Full-text (**best-effort** — see below) | +| `--all-folders` | Sweep every selectable mailbox instead of one folder | | `--since-date` / `--before-date` | RFC 3339 bounds, e.g. `2026-06-01T00:00:00Z` | | `--limit` | Max results (default 50) | `data` shape is identical to `list` (`{ "messages": [ … ] }`). Searches the whole folder regardless of new/acked state. Filtered mail never appears. +**`--text` is unreliable on some servers.** IMAP full-text search depends on the server having a +body index; some servers (observed in the wild) return **zero results** for mail that +`--subject-contains` finds fine. Prefer `--subject-contains` and `--from` as your primary search +methods and treat `--text` as best-effort — a `--text` miss does not mean the mail is absent. + +**`--all-folders`** sweeps every selectable mailbox (mutually exclusive with `--folder`): +- Each message in the result gains a `"folder"` field. UIDs are only unique per folder, so pass + that folder back via `--folder` when you `get`/`ack` a hit. +- `data.skipped_folders` lists mailboxes the server refused to search (always present, `[]` when + none) — transport errors, not policy. +- `--limit` caps total visible results across the sweep. +- On Gmail-style servers the same message can appear under several folders with different UIDs; + dedupe by `message_id` if that matters. + +**Empty results include a `hint`.** When `messages` is `[]`, `data.hint` suggests next steps +(run `folders`, retry with `--all-folders`, or adjust criteria). The hint text is fixed per mode +and identical whether matching mail is absent, in another folder, or hidden by policy — it tells +you what to try, not why the result is empty. + --- ## `ack` — mark message(s) processed @@ -150,7 +190,8 @@ is subject to the inbound filter: a filtered/missing source returns `not_found`. - **Mode:** `RO` accounts reject `send`. `RW` can read and send. - **Inbound whitelist / subject filter:** disallowed mail is invisible everywhere (`list`/`search` omit it; `get`/`ack` return `not_found`). You can't tell a filtered message from a non-existent - one — by design. + one — by design. The empty-search `hint` is deliberately generic for the same reason: it never + reveals whether anything was filtered. - **Outbound whitelist:** every recipient (to+cc+bcc) must match, or the send is blocked whole. - **Address matching:** case-insensitive; an entry `@domain.com` matches any address at that domain; otherwise an exact-address match.