Bump client version to 3.3.0

Split the torch layer and drop 7.8GB of orphaned CUDA libs from the CPU image
The CPU image was 11.2GB — larger than the CUDA one — and its single 6.72GB torch layer could not be pushed at all: the registry drops any blob upload taking longer than 60s, and that layer needed ~61s. Both problems came from install ordering. Dockerfile.cpu installed the project and sentence-transformers first, which resolved the default CUDA torch and pulled ~2.7GB of nvidia-* wheels, then reinstalled torch from the CPU index. Reinstalling replaces torch but leaves its transitive CUDA dependencies behind, orphaned and unused (torch reports 2.13.0+cpu, cuda.is_available() False). Installing CPU torch first, from the CPU index, means nothing ever requests a CUDA build. Everything else then resolves against the torch already present. CPU image 11.2GB -> 3.43GB largest layer 6.72GB -> 896MB push failed 5/5 at 60s -> succeeds in 8.6s Dockerfile.nvidia gets the same split. Its torch layer is unavoidably large (~2.8GiB compressed, ~52s at current throughput) so it stays the tightest thing in the stack, but splitting the application install off keeps that layer cached and the app layer small. Both now install torch before the source COPYs, so editing application code no longer invalidates the multi-GB layer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 18:37:16 +01:00 · 2026-08-21 18:33:34 +01:00 · 2026-08-21 17:30:33 +01:00 · 2026-08-21 17:20:41 +01:00 · 2026-08-21 17:15:42 +01:00 · 2026-08-21 09:58:13 +01:00
86 changed files with 4908 additions and 362 deletions
@@ -4,3 +4,6 @@ __pycache__/
 engine/data/

 TMP/
+.env
+.venv/
+test_mcp_client.py
@@ -11,9 +11,6 @@ cd engine

 # NVIDIA GPU
 KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d
-
-# AMD GPU (ROCm)
-KB_DATA_PATH=~/kb-data docker compose -f compose.rocm.yaml up -d
 ```

 ### Client
@@ -24,6 +21,35 @@ make build    # produces ./kb binary
 make all      # or cross-compile: dist/kb-{os}-{arch}
 ```

+## Running tests
+
+### Engine
+
+Engine tests run against SQLite (with sqlite-vec) and stub out the embedding
+model, so they only need lightweight dependencies — no torch/docling install:
+
+```bash
+uv venv /tmp/kb-test-venv
+uv pip install --python /tmp/kb-test-venv/bin/python pytest pytest-asyncio fastapi httpx sqlite-vec
+cd engine && /tmp/kb-test-venv/bin/python -m pytest
+```
+
+### Client
+
+```bash
+cd client && go test ./...
+```
+
+## Search-quality benchmarking
+
+`kb bench fixture.json` runs a fixture of queries with known-relevant documents
+against each backend (fts, vec, hybrid, hybrid+rerank) and reports precision@k,
+recall, and MRR. See `docs/bench-example.json` for the fixture format.
+
+Run a bench before and after any ranking change (RRF weights, reranker, model
+swap) and compare — keep a 20-30 query fixture against your real corpus outside
+the repo.
+
 ## Building and releasing

 Client and engine are versioned independently via `client/VERSION` and `engine/VERSION`. Each has its own release script and git tag prefix.
@@ -50,7 +76,7 @@ The client embeds a `MinEngineVersion` (from `client/MIN_ENGINE_VERSION`) and wi
 ./release-engine.sh --gitea --dry-run    # preview without doing anything
 ```

-Creates tag `engine-vX.Y.Z`, builds NVIDIA and ROCm Docker images, creates a Gitea/GitHub release, and pushes images to the registry.
+Creates tag `engine-vX.Y.Z`, builds NVIDIA and CPU Docker images, creates a Gitea/GitHub release, and pushes images to the registry.

 ### Checking versions

@@ -64,10 +90,10 @@ curl http://localhost:8000/api/v1/status | jq .version

 ### Docker images

-Images are pushed to `docker.dcglab.co.uk/dcg/kb/engine` with tags:
+Images are pushed to `docker.dcglab.co.uk/kb/engine` with tags:

- `engine-v2.0.6-nvidia` / `engine-v2.0.6-rocm` — versioned
- `latest-nvidia` / `latest-rocm` — latest release
+- `engine-v2.0.6-nvidia` / `engine-v2.0.6-cpu` — versioned
+- `latest-nvidia` / `latest-cpu` — latest release

 Override the registry and org via environment variables:

@@ -75,6 +101,15 @@ Override the registry and org via environment variables:
 REGISTRY=ghcr.io IMAGE_ORG=myorg ./release-engine.sh --github
 ```

+Pushes are retried on transient registry failures. The engine images carry a
+~5.6GB torch layer, and uploading it can fail with a 502 from the proxy in
+front of the registry (or a 500 on the manifest PUT that follows), which
+clears on a retry. Tune with:
+
+```bash
+PUSH_RETRIES=8 PUSH_RETRY_DELAY=20 ./release-engine.sh --gitea
+```
+
 ## API reference

 All endpoints are under `/api/v1/`. Requires `Authorization: Bearer <key>` header when `KB_API_KEY` is set.
@@ -91,10 +126,10 @@ All endpoints are under `/api/v1/`. Requires `Authorization: Bearer <key>` heade
 | `GET` | `/documents/{id}/file` | Download original file |
 | `DELETE` | `/documents/{id}` | Remove a document (and stored file) |
 | `PUT` | `/documents/{id}/tags` | Add/remove tags |
-| `GET` | `/tags` | List all tags |
-| `GET` | `/status` | Engine status, GPU info, DB stats |
+| `GET` | `/tags` | List all tags (with descriptions) |
+| `PUT` | `/tags/{name}/description` | Set/clear a tag context description |
+| `GET` | `/status` | Engine status, GPU info, DB stats, rerank state |
 | `POST` | `/reindex` | Re-embed all chunks |
-
-## Future: ROCm runtime migration
-
-The `onnxruntime-rocm` execution provider was removed from onnxruntime as of v1.23. AMD is pushing toward the **MIGraphX execution provider** as the replacement for ROCm GPU inference. When upgrading onnxruntime beyond v1.22, the ROCm Dockerfile will need to switch from `onnxruntime-rocm` to `onnxruntime` with the MIGraphX EP and install the `migraphx` runtime libraries instead.
+| `POST` | `/bulk/delete` | Bulk delete documents by filter |
+| `POST` | `/bulk/tags` | Bulk add/remove tags by filter |
+| `POST` | `/bulk/set-tags` | Bulk replace tags by filter |
@@ -0,0 +1,174 @@
+# MCP Server (Agent Integration)
+
+The MCP server exposes kb operations as native MCP tools, so agents can search, add notes, upload files, and manage documents without shelling out to the CLI. `kb_search` is hybrid: dense vector embeddings (semantic similarity) fused with BM25 full-text ranking via Reciprocal Rank Fusion, so agents can ask natural-language questions and find conceptually related content even when the exact words don't match.
+
+## Start the MCP server
+
+The compose files include a `kb-mcp` service alongside the engine. Set `KB_MCP_API_KEY` to require Bearer token auth from connecting agents:
+
+```bash
+KB_API_KEY=your-engine-key KB_MCP_API_KEY=your-agent-key \
+  docker compose -f engine/compose.nvidia.yaml up -d
+```
+
+Or run the MCP server standalone:
+
+```bash
+docker run -d --name kb-mcp \
+  -p 3000:3000 \
+  -e KB_ENGINE_URL=http://your-engine-host:8000 \
+  -e KB_API_KEY=your-engine-key \
+  -e KB_MCP_API_KEY=your-agent-key \
+  --restart unless-stopped \
+  docker.dcglab.co.uk/kb/mcp:latest
+```
+
+## MCP tools
+
+| Tool | Description |
+|---|---|
+| `kb_search` | Hybrid semantic (vector) + full-text search with tag/type filters |
+| `kb_addnote` | Add a text note (queued for async ingestion) |
+| `kb_update_note` | Update an existing note in place |
+| `kb_get` | Get document details by ID or source path |
+| `kb_delete` | Permanently delete a document by ID |
+| `kb_status` | Engine health and statistics |
+| `kb_jobs` | Ingestion queue status |
+| `kb_upload_start` | Start a chunked file upload |
+| `kb_upload_chunk` | Upload a base64-encoded file chunk |
+| `kb_upload_finish` | Finish upload and submit for ingestion |
+| `kb_bulk_delete` | Delete multiple documents matching a filter |
+| `kb_bulk_tags` | Add/remove tags on multiple documents |
+| `kb_bulk_set_tags` | Replace all tags on multiple documents |
+
+## Organising with tags
+
+Use tags to separate agent data from user documents. For example, an agent can tag all its notes with `agent:mybot` and filter by that tag when searching. This is a naming convention — configure it in your agent's system prompt. No special server-side enforcement is needed.
+
+Bulk tools accept filter-based selection (by tags, doc_type, ID list, or ID range) so agents can manage thousands of documents in a single call instead of looping. A safety threshold (default 70%, configurable via engine env var `KB_BULK_SAFETY_PERCENT`) prevents accidental mass operations unless `force: true` is set.
+
+## MCP server configuration
+
+| Variable | Default | Description |
+|---|---|---|
+| `KB_ENGINE_URL` | `http://localhost:8000` | Engine API URL |
+| `KB_API_KEY` | (none) | Engine API key |
+| `KB_MCP_API_KEY` | (none) | Bearer token required from agents (disabled if unset) |
+| `KB_MCP_PORT` | `3000` | Port to listen on |
+
+## Connecting AI coding tools
+
+The kb MCP server uses **Streamable HTTP** transport at `http://your-host:3000/mcp`. Below are configuration examples for popular AI coding tools.
+
+### Claude Code (CLI / Desktop / Web)
+
+Add the server to your project or user settings:
+
+```bash
+claude mcp add kb-server --transport http http://localhost:3000/mcp
+```
+
+Or add it manually to `.claude/settings.json` (project) or `~/.claude/settings.json` (global):
+
+```json
+{
+  "mcpServers": {
+    "kb-server": {
+      "type": "http",
+      "url": "http://localhost:3000/mcp",
+      "headers": {
+        "Authorization": "Bearer your-agent-key"
+      }
+    }
+  }
+}
+```
+
+### VS Code (GitHub Copilot)
+
+Add to your `.vscode/settings.json` (workspace) or user settings:
+
+```json
+{
+  "mcp": {
+    "servers": {
+      "kb-server": {
+        "type": "http",
+        "url": "http://localhost:3000/mcp",
+        "headers": {
+          "Authorization": "Bearer your-agent-key"
+        }
+      }
+    }
+  }
+}
+```
+
+Or add to `.vscode/mcp.json` in your workspace:
+
+```json
+{
+  "servers": {
+    "kb-server": {
+      "type": "http",
+      "url": "http://localhost:3000/mcp",
+      "headers": {
+        "Authorization": "Bearer your-agent-key"
+      }
+    }
+  }
+}
+```
+
+### Cursor
+
+Add to `.cursor/mcp.json` in your project root:
+
+```json
+{
+  "mcpServers": {
+    "kb-server": {
+      "type": "streamable-http",
+      "url": "http://localhost:3000/mcp",
+      "headers": {
+        "Authorization": "Bearer your-agent-key"
+      }
+    }
+  }
+}
+```
+
+### Windsurf
+
+Add to `~/.codeium/windsurf/mcp_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "kb-server": {
+      "serverUrl": "http://localhost:3000/mcp",
+      "headers": {
+        "Authorization": "Bearer your-agent-key"
+      }
+    }
+  }
+}
+```
+
+### JetBrains IDEs (IntelliJ, WebStorm, PyCharm, etc.)
+
+Add to `.junie/mcp.json` in your project root, or configure via **Settings > Tools > AI Assistant > MCP Servers**:
+
+```json
+{
+  "servers": {
+    "kb-server": {
+      "type": "http",
+      "url": "http://localhost:3000/mcp",
+      "headers": {
+        "Authorization": "Bearer your-agent-key"
+      }
+    }
+  }
+}
+```
@@ -2,16 +2,19 @@

 Personal knowledge base with hybrid search (full-text + semantic vector search).

-v2 uses a client-server architecture: a **FastAPI engine** running in Docker (with optional GPU acceleration) and a lightweight **Go CLI client** that talks to it over HTTP.
+Client-server architecture: a **FastAPI engine** running in Docker (with optional GPU acceleration), a lightweight **Go CLI client**, and an **MCP server** for native agent integration.

 ## Architecture

 ```
 Go CLI (kb) ──HTTP──▶ FastAPI Engine (Docker) ──▶ SQLite + GPU
+                            ▲
+MCP Agents  ──MCP/HTTP──▶ MCP Server (Docker) ──┘
 ```

- **Engine**: Keeps the embedding model warm in memory. Handles search, ingestion, and document management via REST API. Runs in Docker with NVIDIA GPU, AMD GPU (ROCm), or CPU-only support.
+- **Engine**: Keeps the embedding model warm in memory. Handles search, ingestion, document management, and note mutation via REST API. Runs in Docker with NVIDIA GPU or CPU-only support.
 - **Client**: Single static Go binary. No Python, no ML dependencies, instant startup. Talks to the engine over HTTP.
+- **MCP Server**: Exposes kb operations as native MCP tools over Streamable HTTP. Runs as a separate Docker container alongside the engine. Use tags to scope agent data from user documents.
 - **Storage**: Single SQLite database with FTS5 (keyword search) and sqlite-vec (vector search). Portable via bind mount — just copy the data directory between hosts.

 ## Quick start
@@ -30,19 +33,7 @@ docker run -d --name kb-engine \
  -e KB_DEVICE=auto \
  -e KB_API_KEY=your-secret-key \
  --restart unless-stopped \
-  docker.dcglab.co.uk/dcg/kb/engine:latest-nvidia
-
-# AMD GPU (ROCm)
-docker run -d --name kb-engine \
-  --device /dev/kfd --device /dev/dri \
-  --group-add video \
-  -p 8000:8000 \
-  -v ~/kb-data:/data \
-  -e KB_MODEL=all-MiniLM-L6-v2 \
-  -e KB_DEVICE=auto \
-  -e KB_API_KEY=your-secret-key \
-  --restart unless-stopped \
-  docker.dcglab.co.uk/dcg/kb/engine:latest-rocm
+  docker.dcglab.co.uk/kb/engine:latest-nvidia

 # CPU only (no GPU required — smaller image)
 docker run -d --name kb-engine \
@@ -51,7 +42,7 @@ docker run -d --name kb-engine \
  -e KB_MODEL=all-MiniLM-L6-v2 \
  -e KB_API_KEY=your-secret-key \
  --restart unless-stopped \
-  docker.dcglab.co.uk/dcg/kb/engine:latest-cpu
+  docker.dcglab.co.uk/kb/engine:latest-cpu
 ```

 Or use a compose file from the repo:
@@ -60,9 +51,6 @@ Or use a compose file from the repo:
 # NVIDIA GPU
 KB_DATA_PATH=~/kb-data docker compose -f engine/compose.nvidia.yaml up -d

-# AMD GPU (ROCm)
-KB_DATA_PATH=~/kb-data docker compose -f engine/compose.rocm.yaml up -d
-
 # CPU only
 KB_DATA_PATH=~/kb-data docker compose -f engine/compose.cpu.yaml up -d
 ```
@@ -84,7 +72,7 @@ Check [releases](https://gitea.dcglab.co.uk/steve/kb/releases) for the latest cl

 ```bash
 # Set the version tag
-TAG=client-v2.1.0
+TAG=client-v3.0.0

 # Linux (amd64)
 curl -L -o kb https://gitea.dcglab.co.uk/steve/kb/releases/download/${TAG}/kb-linux-amd64
@@ -135,6 +123,9 @@ kb jobs
 kb search "how to install git"
 kb search "deploy process" --tags ops --type pdf

+# Update a note in place
+kb updatenote 42 "revised note content"
+
 # Manage
 kb list
 kb info 1
@@ -143,12 +134,18 @@ kb tag 1 --add important
 kb export 1 -o manual.pdf    # download original file
 kb remove 3 --yes
 kb status
+
+# Bulk operations
+kb bulk-remove --tags "draft,old" --type note --yes
+kb bulk-tag --type note --add "archived" --yes
+kb bulk-set-tags --tags "old-scheme" --set "new-scheme" --yes
 ```

 ## How it works

- **Ingestion**: Files are uploaded to the engine and queued for async processing. The engine chunks documents (PDFs via Docling, markdown by headers, code by AST/functions, notes as whole text), generates embeddings on GPU, and stores everything in SQLite.
- **Search**: Hybrid retrieval combining BM25 keyword scoring (FTS5) and vector similarity (sqlite-vec), merged via Reciprocal Rank Fusion. Sub-100ms with a warm model.
+- **Ingestion**: Files are uploaded to the engine and queued for async processing. The engine chunks documents (PDFs via Docling, markdown by headers, code by AST/functions, notes as whole text, JSON/YAML/TOML as pretty-printed text), generates embeddings on GPU, and stores everything in SQLite.
+- **Search**: Hybrid retrieval combining BM25 keyword scoring (FTS5) and vector similarity (sqlite-vec), merged via Reciprocal Rank Fusion with a top-rank bonus. Optionally reranked by a local cross-encoder (`KB_RERANK_ENABLED`). Sub-100ms with a warm model (without reranking). Add `--explain` to any search for a per-result score breakdown.
+- **Quality measurement**: `kb bench fixture.json` runs a fixture of queries with known-relevant documents and reports precision@k / recall / MRR per backend (fts, vec, hybrid, hybrid+rerank). See `docs/bench-example.json`.
 - **Output**: JSON (for scripts/LLM tool use) or human-readable terminal format. Use `--format json` on any command.

 ## Engine configuration
@@ -163,6 +160,10 @@ The engine is configured via environment variables (set in the compose file or v
 | `KB_INGEST_DEVICE` | `auto` | Docling layout detection device: `auto`, `cpu`, or `cuda` |
 | `KB_API_KEY` | (none) | Optional Bearer token for API authentication |
 | `KB_SEARCH_THRESHOLD` | `0.01` | Minimum score for search results (filters noise) |
+| `KB_RERANK_ENABLED` | `false` (`true` in nvidia compose) | Load a cross-encoder and rerank hybrid results server-side |
+| `KB_RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | Cross-encoder model for reranking |
+| `KB_RERANK_CANDIDATES` | `40` | Hybrid candidates scored by the reranker per query |
+| `KB_BULK_SAFETY_PERCENT` | `70` | Bulk operations affecting more than this % of documents are rejected unless `force` is set (0 disables) |
 | `KB_PORT` | `8000` | Port to expose |
 | `KB_HOST` | `0.0.0.0` | Host to bind to |
 | `HF_HUB_OFFLINE` | (none) | Set to `1` to prevent model downloads (use cached only) |
@@ -180,8 +181,14 @@ rsync -a ~/kb-data/ user@target:/home/user/kb-data/
 KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d
 ```

-Data is device-agnostic — you can ingest on NVIDIA and serve from AMD or CPU (or any combination) with the same data directory.
+Data is device-agnostic — you can ingest on NVIDIA and serve from CPU (or vice versa) with the same data directory.

-## Claude Code skill
+## MCP server (agent integration)

-This tool is designed to be wrapped as a Claude Code skill. See `SKILL.md` for the skill definition.
+The MCP server exposes kb operations as native MCP tools over Streamable HTTP, so agents can search, add notes, upload files, and manage documents without shelling out to the CLI. Includes setup guides for Claude Code, VS Code, Cursor, Windsurf, and JetBrains IDEs.
+
+See **[MCP.md](MCP.md)** for full details — server setup, available tools, tag-based organisation, configuration, and client examples.
+
+## Agent skill
+
+If you are restricted from using MCP server, or you just prefer to utilise Agent SKILLS, please also see `SKILL.md` for the skill definition.
@@ -30,11 +30,13 @@ Returns JSON with ranked results combining full-text and semantic search.
 **Flags:**
 - `-n, --top N` — number of results (default: 10)
 - `--tags tag1,tag2` — filter by tags (AND logic)
- `--type pdf|markdown|code|note` — filter by document type
+- `--type pdf|markdown|code|note|data` — filter by document type
 - `--format json|human` — output format (always use json for parsing)
 - `--fts-only` — keyword search only (skip semantic)
 - `--vec-only` — semantic search only (skip keyword)
 - `--threshold FLOAT` — minimum score cutoff
+- `--explain` — include a per-result score breakdown (FTS/vector scores and ranks, fusion contributions, rerank blend)
+- `--no-rerank` — skip server-side cross-encoder reranking for lower latency (when the engine has it enabled)

 ## Adding files

@@ -45,7 +47,7 @@ kb addfile ~/docs/ --recursive                  # directory (recursive)
 kb addfile ~/docs/ --recursive --tags reference # directory with tags
 ```

-Supported file types: `.pdf`, `.docx`, `.html`, `.md`, `.txt`, `.py`, `.sh`, `.go`. Unsupported extensions are rejected before upload.
+Supported file types: `.pdf`, `.docx`, `.html`, `.md`, `.txt`, `.py`, `.sh`, `.go`, `.json`, `.yaml`, `.yml`, `.toml`. Unsupported extensions are rejected before upload. Data files (`.json`/`.yaml`/`.yml`/`.toml`) are ingested as text with doc type `data`; minified JSON is pretty-printed before chunking.

 **Flags:**
 - `--tags tag1,tag2` — tags (comma-separated)
@@ -66,11 +68,48 @@ kb remove <doc_id> --yes                 # remove without confirmation
 ## Tag management

 ```bash
-kb tags --format json                    # list all tags with counts
+kb tags --format json                    # list all tags with counts and descriptions
 kb tag <doc_id> --add important,ops      # add tags to a document
 kb tag <doc_id> --remove draft           # remove tags from a document
+kb tag-describe ops "Lab operations runbooks"   # set a tag context description
+kb tag-describe ops                      # clear a tag's description
 ```

+Tag descriptions are returned as `tag_contexts` with every search result on a
+document carrying the tag — use them to judge which of several similar-scoring
+chunks actually answers the question.
+
+## Bulk operations
+
+Operate on multiple documents at once using filter-based selection. Filters combine with AND logic.
+
+**Filter flags (shared across all bulk commands):**
+- `--tags tag1,tag2` — match documents with ALL specified tags
+- `--type pdf|note|...` — match by document type
+- `--ids 1,5,12` — match specific document IDs
+- `--from-id N` — match documents with id >= N
+- `--to-id N` — match documents with id <= N
+- `--force` / `-f` — override safety threshold (blocks operations affecting >70% of all documents)
+- `--yes` / `-y` — skip confirmation prompt
+
+```bash
+# Bulk delete
+kb bulk-remove --tags "draft,old" --type note --yes             # delete matching docs
+kb bulk-remove --from-id 10 --to-id 50 --yes                   # delete by ID range
+kb bulk-remove --ids "3,7,12" --yes                             # delete specific IDs
+
+# Bulk tag add/remove
+kb bulk-tag --tags "agent:mybot" --add "reviewed" --remove "pending" --yes
+kb bulk-tag --type note --add "archived" --yes                  # tag all notes
+
+# Bulk replace tags
+kb bulk-set-tags --tags "old-scheme" --set "new-scheme,migrated" --yes
+```
+
+All bulk commands return a summary: matched count, succeeded count, failed count, and errors.
+A safety threshold prevents accidentally affecting more than 70% of documents unless `--force` is used.
+The threshold is configurable on the engine via `KB_BULK_SAFETY_PERCENT` (integer 0-100, default 70; 0 disables).
+
 ## Jobs (ingestion queue)

 ```bash
@@ -107,6 +146,7 @@ All commands support:
  "results": [
    {
      "chunk_id": 1423,
+      "document_id": 87,
      "score": 0.031,
      "text": "To install the latest version of git from source...",
      "chunk_index": 3,
@@ -115,11 +155,13 @@ All commands support:
      "doc_type": "pdf",
      "source_path": "/home/user/docs/git-admin.pdf",
      "created_at": "2026-03-15T10:30:00",
-      "tags": ["git", "admin"]
+      "tags": ["git", "admin"],
+      "tag_contexts": {"admin": "System administration guides"}
    }
  ],
  "total_matches": 47,
-  "returned": 10
+  "returned": 10,
+  "reranked": true
 }
 ```

@@ -172,12 +214,13 @@ For agent-to-agent integration, kb provides an MCP server alongside the CLI. The
 exposes the same operations as native MCP tools over Streamable HTTP transport, which agents
 can connect to directly without subprocess overhead.

-**MCP tools:** `kb_search`, `kb_addnote`, `kb_update_note`, `kb_get`, `kb_status`, `kb_jobs`,
-`kb_upload_start`, `kb_upload_chunk`, `kb_upload_finish`.
+**MCP tools:** `kb_search`, `kb_addnote`, `kb_update_note`, `kb_get`, `kb_delete`, `kb_status`,
+`kb_jobs`, `kb_upload_start`, `kb_upload_chunk`, `kb_upload_finish`, `kb_bulk_delete`,
+`kb_bulk_tags`, `kb_bulk_set_tags`.

-The MCP server supports **collections** — scoped document namespaces (e.g. `memory`, `documents`,
-`workspace`) implemented via tag conventions. This is the recommended way for agents to separate
-their memory from user documents.
+Use tags to separate agent data from user documents (e.g. tag all agent notes with
+`agent:mybot` and filter by that tag when searching). This convention is communicated
+via system prompt — no special server-side enforcement needed.

 If the kb engine is already running via Docker Compose, add the MCP server by deploying the
 `kb-mcp` service from the same compose file. Agents connect to it on port 3000 (default).
@@ -185,7 +228,8 @@ If the kb engine is already running via Docker Compose, add the MCP server by de
 ## Important notes

 - Always use `--format json` for machine parsing
- The `score` field is relative, not absolute — compare scores within a result set
+- The `score` field is relative, not absolute — compare scores within a result set. Reranked hybrid scores (`"reranked": true`) are 0-1 blended values on a different scale from non-reranked RRF scores; don't compare across the two modes or apply `--threshold` expecting RRF-scale values on reranked output
+- When the engine reranker is enabled, results are already cross-encoder reranked server-side — no need to rerank them yourself
 - `chunk_metadata.page` is only present for PDF documents
 - `chunk_metadata.section_header` is only present for markdown documents with headers
 - Results are already ranked by relevance (hybrid FTS + vector search)
@@ -1 +1 @@
-3.0.0
+3.3.0
@@ -1 +1 @@
-3.0.0
+3.3.0
@@ -38,6 +38,10 @@ var supportedExts = map[string]bool{
 	".py":   true,
 	".sh":   true,
 	".go":   true,
+	".json": true,
+	".yaml": true,
+	".yml":  true,
+	".toml": true,
 }

 var addfileCmd = &cobra.Command{
@@ -0,0 +1,376 @@
+package cmd
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/kb-search/kb/internal/api"
+	"github.com/kb-search/kb/internal/output"
+	"github.com/spf13/cobra"
+)
+
+var benchCmd = &cobra.Command{
+	Use:   "bench <fixture.json>",
+	Short: "Benchmark search quality against a query fixture",
+	Long: `Run a fixture of queries with known-relevant documents against each
+search backend (fts, vec, hybrid, rerank) and report precision@k, recall
+and MRR per backend. Use this to baseline search quality before ranking
+changes and to measure their effect. See docs/bench-example.json for the
+fixture format.`,
+	Args: cobra.ExactArgs(1),
+	RunE: runBench,
+}
+
+func init() {
+	benchCmd.Flags().Int("top", 0, "override result count per query (k)")
+	benchCmd.Flags().String("backends", "fts,vec,hybrid,rerank", "comma-separated backends to run")
+	rootCmd.AddCommand(benchCmd)
+}
+
+type benchSelector struct {
+	DocumentID    int64  `json:"document_id"`
+	SourcePath    string `json:"source_path"`
+	TitleContains string `json:"title_contains"`
+}
+
+type benchQuery struct {
+	ID       string          `json:"id"`
+	Query    string          `json:"query"`
+	Tags     []string        `json:"tags"`
+	DocType  string          `json:"doc_type"`
+	Top      int             `json:"top"`
+	Relevant []benchSelector `json:"relevant"`
+	Notes    string          `json:"notes"`
+}
+
+type benchFixture struct {
+	Description string       `json:"description"`
+	Top         int          `json:"top"`
+	Queries     []benchQuery `json:"queries"`
+}
+
+type benchDoc struct {
+	DocumentID int64
+	Title      string
+	SourcePath string
+}
+
+type queryResult struct {
+	ID        string  `json:"id"`
+	Precision float64 `json:"precision"`
+	Recall    float64 `json:"recall"`
+	MRR       float64 `json:"mrr"`
+	LatencyMS float64 `json:"latency_ms"`
+	Returned  int     `json:"returned"`
+}
+
+type backendResult struct {
+	Precision    float64       `json:"precision"`
+	Recall       float64       `json:"recall"`
+	MRR          float64       `json:"mrr"`
+	AvgLatencyMS float64       `json:"avg_latency_ms"`
+	Queries      []queryResult `json:"queries"`
+}
+
+func loadFixture(path string) (*benchFixture, error) {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil, fmt.Errorf("cannot read fixture: %w", err)
+	}
+	var fx benchFixture
+	if err := json.Unmarshal(data, &fx); err != nil {
+		return nil, fmt.Errorf("invalid fixture JSON: %w", err)
+	}
+	if len(fx.Queries) == 0 {
+		return nil, fmt.Errorf("fixture has no queries")
+	}
+	for i, q := range fx.Queries {
+		if q.Query == "" {
+			return nil, fmt.Errorf("query %d: missing query text", i)
+		}
+		if len(q.Relevant) == 0 {
+			return nil, fmt.Errorf("query %q: no relevant selectors", q.Query)
+		}
+		for j, sel := range q.Relevant {
+			set := 0
+			if sel.DocumentID != 0 {
+				set++
+			}
+			if sel.SourcePath != "" {
+				set++
+			}
+			if sel.TitleContains != "" {
+				set++
+			}
+			if set != 1 {
+				return nil, fmt.Errorf("query %q selector %d: exactly one of document_id, source_path, title_contains must be set", q.Query, j)
+			}
+		}
+	}
+	return &fx, nil
+}
+
+func (s benchSelector) matches(doc benchDoc) bool {
+	switch {
+	case s.DocumentID != 0:
+		return doc.DocumentID == s.DocumentID
+	case s.SourcePath != "":
+		return doc.SourcePath == s.SourcePath
+	case s.TitleContains != "":
+		return strings.Contains(strings.ToLower(doc.Title), strings.ToLower(s.TitleContains))
+	}
+	return false
+}
+
+// dedupeByDocument collapses ranked chunks into ranked documents, keeping the
+// best (first) position for each document.
+func dedupeByDocument(docs []benchDoc) []benchDoc {
+	seen := map[int64]bool{}
+	var out []benchDoc
+	for _, d := range docs {
+		if seen[d.DocumentID] {
+			continue
+		}
+		seen[d.DocumentID] = true
+		out = append(out, d)
+	}
+	return out
+}
+
+// scoreQuery computes document-level precision, recall and MRR for one
+// query's ranked document list against the relevant-document selectors.
+func scoreQuery(ranked []benchDoc, relevant []benchSelector) (precision, recall, mrr float64) {
+	if len(ranked) == 0 {
+		return 0, 0, 0
+	}
+
+	matchedDocs := 0
+	firstMatch := 0
+	selectorHit := make([]bool, len(relevant))
+	for i, doc := range ranked {
+		docMatched := false
+		for j, sel := range relevant {
+			if sel.matches(doc) {
+				docMatched = true
+				selectorHit[j] = true
+			}
+		}
+		if docMatched {
+			matchedDocs++
+			if firstMatch == 0 {
+				firstMatch = i + 1
+			}
+		}
+	}
+
+	selectorsMatched := 0
+	for _, hit := range selectorHit {
+		if hit {
+			selectorsMatched++
+		}
+	}
+
+	precision = float64(matchedDocs) / float64(len(ranked))
+	recall = float64(selectorsMatched) / float64(len(relevant))
+	if firstMatch > 0 {
+		mrr = 1.0 / float64(firstMatch)
+	}
+	return precision, recall, mrr
+}
+
+// rerankAvailable probes engine status for a loaded reranker.
+func rerankAvailable(client *api.Client) bool {
+	resp, err := client.Get("/api/v1/status")
+	if err != nil {
+		return false
+	}
+	var status struct {
+		Rerank struct {
+			Enabled bool `json:"enabled"`
+			Loaded  bool `json:"loaded"`
+		} `json:"rerank"`
+	}
+	if err := api.DecodeJSON(resp, &status); err != nil {
+		return false
+	}
+	return status.Rerank.Enabled && status.Rerank.Loaded
+}
+
+func benchSearch(client *api.Client, q benchQuery, backend string, top int) ([]benchDoc, float64, error) {
+	body := map[string]interface{}{
+		"query": q.Query,
+		"top":   top,
+	}
+	if len(q.Tags) > 0 {
+		body["tags"] = q.Tags
+	}
+	if q.DocType != "" {
+		body["doc_type"] = q.DocType
+	}
+	switch backend {
+	case "fts":
+		body["fts_only"] = true
+	case "vec":
+		body["vec_only"] = true
+	case "hybrid":
+		body["rerank"] = false
+	case "rerank":
+		body["rerank"] = true
+	}
+
+	start := time.Now()
+	resp, err := client.Post("/api/v1/search", body)
+	if err != nil {
+		return nil, 0, err
+	}
+	if err := api.CheckError(resp); err != nil {
+		return nil, 0, err
+	}
+	var result struct {
+		Results []struct {
+			DocumentID int64  `json:"document_id"`
+			Title      string `json:"title"`
+			SourcePath string `json:"source_path"`
+		} `json:"results"`
+	}
+	if err := api.DecodeJSON(resp, &result); err != nil {
+		return nil, 0, err
+	}
+	latency := float64(time.Since(start).Microseconds()) / 1000.0
+
+	var docs []benchDoc
+	for _, r := range result.Results {
+		docs = append(docs, benchDoc{DocumentID: r.DocumentID, Title: r.Title, SourcePath: r.SourcePath})
+	}
+	return dedupeByDocument(docs), latency, nil
+}
+
+func runBench(cmd *cobra.Command, args []string) error {
+	topFlag, _ := cmd.Flags().GetInt("top")
+	backendsFlag, _ := cmd.Flags().GetString("backends")
+
+	fx, err := loadFixture(args[0])
+	if err != nil {
+		return err
+	}
+
+	var backends []string
+	for _, b := range strings.Split(backendsFlag, ",") {
+		b = strings.TrimSpace(b)
+		if b == "" {
+			continue
+		}
+		switch b {
+		case "fts", "vec", "hybrid", "rerank":
+			backends = append(backends, b)
+		default:
+			return fmt.Errorf("unknown backend %q (valid: fts, vec, hybrid, rerank)", b)
+		}
+	}
+	if len(backends) == 0 {
+		return fmt.Errorf("no backends selected")
+	}
+
+	client := api.NewClient()
+
+	rerankSkipped := false
+	if contains(backends, "rerank") && !rerankAvailable(client) {
+		backends = remove(backends, "rerank")
+		rerankSkipped = true
+		if len(backends) == 0 {
+			return fmt.Errorf("reranking is not available on this engine (requires engine with reranker enabled)")
+		}
+	}
+
+	results := map[string]*backendResult{}
+	for _, backend := range backends {
+		br := &backendResult{}
+		for _, q := range fx.Queries {
+			top := 10
+			if fx.Top > 0 {
+				top = fx.Top
+			}
+			if q.Top > 0 {
+				top = q.Top
+			}
+			if topFlag > 0 {
+				top = topFlag
+			}
+
+			ranked, latency, err := benchSearch(client, q, backend, top)
+			if err != nil {
+				return fmt.Errorf("backend %s, query %q: %w", backend, q.Query, err)
+			}
+			p, r, m := scoreQuery(ranked, q.Relevant)
+			id := q.ID
+			if id == "" {
+				id = q.Query
+			}
+			br.Queries = append(br.Queries, queryResult{
+				ID: id, Precision: p, Recall: r, MRR: m,
+				LatencyMS: latency, Returned: len(ranked),
+			})
+		}
+		n := float64(len(br.Queries))
+		for _, qr := range br.Queries {
+			br.Precision += qr.Precision / n
+			br.Recall += qr.Recall / n
+			br.MRR += qr.MRR / n
+			br.AvgLatencyMS += qr.LatencyMS / n
+		}
+		results[backend] = br
+	}
+
+	if output.IsJSON() {
+		output.PrintJSON(map[string]interface{}{
+			"description":    fx.Description,
+			"query_count":    len(fx.Queries),
+			"backends":       results,
+			"rerank_skipped": rerankSkipped,
+		})
+		return nil
+	}
+
+	if fx.Description != "" {
+		fmt.Printf("%s (%d queries)\n\n", fx.Description, len(fx.Queries))
+	}
+	headers := []string{"Backend", "Precision", "Recall", "MRR", "Avg ms"}
+	var rows [][]string
+	for _, backend := range backends {
+		br := results[backend]
+		rows = append(rows, []string{
+			backend,
+			fmt.Sprintf("%.3f", br.Precision),
+			fmt.Sprintf("%.3f", br.Recall),
+			fmt.Sprintf("%.3f", br.MRR),
+			fmt.Sprintf("%.0f", br.AvgLatencyMS),
+		})
+	}
+	output.PrintTable(headers, rows)
+	if rerankSkipped {
+		fmt.Println("\nrerank: n/a (reranker not enabled on this engine)")
+	}
+	return nil
+}
+
+func contains(ss []string, s string) bool {
+	for _, v := range ss {
+		if v == s {
+			return true
+		}
+	}
+	return false
+}
+
+func remove(ss []string, s string) []string {
+	var out []string
+	for _, v := range ss {
+		if v != s {
+			out = append(out, v)
+		}
+	}
+	return out
+}
@@ -0,0 +1,143 @@
+package cmd
+
+import (
+	"math"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func writeFixture(t *testing.T, content string) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "fixture.json")
+	if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	return path
+}
+
+func TestLoadFixture_Valid(t *testing.T) {
+	path := writeFixture(t, `{
+		"description": "test",
+		"top": 5,
+		"queries": [
+			{"id": "q1", "query": "hello", "relevant": [{"document_id": 7}]}
+		]
+	}`)
+	fx, err := loadFixture(path)
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if fx.Top != 5 || len(fx.Queries) != 1 || fx.Queries[0].Relevant[0].DocumentID != 7 {
+		t.Errorf("fixture parsed incorrectly: %+v", fx)
+	}
+}
+
+func TestLoadFixture_RejectsEmptyRelevant(t *testing.T) {
+	path := writeFixture(t, `{"queries": [{"query": "hello", "relevant": []}]}`)
+	if _, err := loadFixture(path); err == nil {
+		t.Error("expected error for query with no relevant selectors")
+	}
+}
+
+func TestLoadFixture_RejectsMultiFieldSelector(t *testing.T) {
+	path := writeFixture(t, `{"queries": [
+		{"query": "hello", "relevant": [{"document_id": 1, "source_path": "/x"}]}
+	]}`)
+	if _, err := loadFixture(path); err == nil {
+		t.Error("expected error for selector with two fields set")
+	}
+}
+
+func TestSelectorMatching(t *testing.T) {
+	doc := benchDoc{DocumentID: 42, Title: "M38T Owner's Manual", SourcePath: "/data/m38t.pdf"}
+
+	cases := []struct {
+		name string
+		sel  benchSelector
+		want bool
+	}{
+		{"document_id match", benchSelector{DocumentID: 42}, true},
+		{"document_id miss", benchSelector{DocumentID: 43}, false},
+		{"source_path match", benchSelector{SourcePath: "/data/m38t.pdf"}, true},
+		{"source_path miss", benchSelector{SourcePath: "/data/other.pdf"}, false},
+		{"title_contains case-insensitive", benchSelector{TitleContains: "m38t owner"}, true},
+		{"title_contains miss", benchSelector{TitleContains: "workshop"}, false},
+	}
+	for _, tc := range cases {
+		if got := tc.sel.matches(doc); got != tc.want {
+			t.Errorf("%s: got %v, want %v", tc.name, got, tc.want)
+		}
+	}
+}
+
+func TestDedupeByDocument(t *testing.T) {
+	docs := []benchDoc{
+		{DocumentID: 1, Title: "a"},
+		{DocumentID: 2, Title: "b"},
+		{DocumentID: 1, Title: "a-again"},
+		{DocumentID: 3, Title: "c"},
+	}
+	out := dedupeByDocument(docs)
+	if len(out) != 3 || out[0].DocumentID != 1 || out[1].DocumentID != 2 || out[2].DocumentID != 3 {
+		t.Errorf("dedupe failed: %+v", out)
+	}
+	if out[0].Title != "a" {
+		t.Errorf("dedupe must keep first (best-ranked) occurrence, got %q", out[0].Title)
+	}
+}
+
+func approxEqual(a, b float64) bool {
+	return math.Abs(a-b) < 1e-9
+}
+
+func TestScoreQuery_HandComputed(t *testing.T) {
+	// Ranked docs: 10, 20, 30, 40. Relevant: 20 and 40.
+	ranked := []benchDoc{
+		{DocumentID: 10}, {DocumentID: 20}, {DocumentID: 30}, {DocumentID: 40},
+	}
+	relevant := []benchSelector{{DocumentID: 20}, {DocumentID: 40}}
+
+	p, r, m := scoreQuery(ranked, relevant)
+	if !approxEqual(p, 0.5) { // 2 of 4 returned docs are relevant
+		t.Errorf("precision: got %v, want 0.5", p)
+	}
+	if !approxEqual(r, 1.0) { // both relevant docs found
+		t.Errorf("recall: got %v, want 1.0", r)
+	}
+	if !approxEqual(m, 0.5) { // first relevant doc at rank 2
+		t.Errorf("mrr: got %v, want 0.5", m)
+	}
+}
+
+func TestScoreQuery_NoMatches(t *testing.T) {
+	ranked := []benchDoc{{DocumentID: 1}}
+	relevant := []benchSelector{{DocumentID: 99}}
+	p, r, m := scoreQuery(ranked, relevant)
+	if p != 0 || r != 0 || m != 0 {
+		t.Errorf("expected all-zero metrics, got p=%v r=%v mrr=%v", p, r, m)
+	}
+}
+
+func TestScoreQuery_PartialRecall(t *testing.T) {
+	// Only one of three relevant docs returned, at rank 1.
+	ranked := []benchDoc{{DocumentID: 5}, {DocumentID: 6}}
+	relevant := []benchSelector{{DocumentID: 5}, {DocumentID: 7}, {DocumentID: 8}}
+	p, r, m := scoreQuery(ranked, relevant)
+	if !approxEqual(p, 0.5) {
+		t.Errorf("precision: got %v, want 0.5", p)
+	}
+	if !approxEqual(r, 1.0/3.0) {
+		t.Errorf("recall: got %v, want 1/3", r)
+	}
+	if !approxEqual(m, 1.0) {
+		t.Errorf("mrr: got %v, want 1.0", m)
+	}
+}
+
+func TestScoreQuery_EmptyResults(t *testing.T) {
+	p, r, m := scoreQuery(nil, []benchSelector{{DocumentID: 1}})
+	if p != 0 || r != 0 || m != 0 {
+		t.Errorf("expected zeros for empty results, got p=%v r=%v mrr=%v", p, r, m)
+	}
+}
@@ -0,0 +1,186 @@
+package cmd
+
+import (
+	"bufio"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+
+	"github.com/kb-search/kb/internal/api"
+	"github.com/kb-search/kb/internal/output"
+	"github.com/spf13/cobra"
+)
+
+var bulkRemoveCmd = &cobra.Command{
+	Use:   "bulk-remove",
+	Short: "Delete multiple documents matching a filter",
+	RunE:  runBulkRemove,
+}
+
+func init() {
+	addBulkFilterFlags(bulkRemoveCmd)
+	rootCmd.AddCommand(bulkRemoveCmd)
+}
+
+func runBulkRemove(cmd *cobra.Command, args []string) error {
+	body, err := buildBulkBody(cmd)
+	if err != nil {
+		return err
+	}
+
+	yes, _ := cmd.Flags().GetBool("yes")
+	if !yes {
+		desc := describeBulkFilter(cmd)
+		fmt.Printf("This will delete documents matching: %s\nProceed? [y/N] ", desc)
+		reader := bufio.NewReader(os.Stdin)
+		answer, _ := reader.ReadString('\n')
+		answer = strings.TrimSpace(strings.ToLower(answer))
+		if answer != "y" && answer != "yes" {
+			fmt.Println("Cancelled.")
+			return nil
+		}
+	}
+
+	client := api.NewClient()
+	resp, err := client.Post("/api/v1/bulk/delete", body)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+	if err := api.CheckError(resp); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+
+	var result map[string]interface{}
+	if err := api.DecodeJSON(resp, &result); err != nil {
+		return fmt.Errorf("failed to decode response: %w", err)
+	}
+
+	if output.IsJSON() {
+		output.PrintJSON(result)
+	} else {
+		printBulkResult("Deleted", result)
+	}
+	return nil
+}
+
+// ---------------------------------------------------------------------------
+// Shared helpers for all bulk commands
+// ---------------------------------------------------------------------------
+
+func addBulkFilterFlags(cmd *cobra.Command) {
+	cmd.Flags().String("tags", "", "filter by tags (comma-separated)")
+	cmd.Flags().String("type", "", "filter by document type")
+	cmd.Flags().String("ids", "", "filter by document IDs (comma-separated)")
+	cmd.Flags().Int("from-id", 0, "filter by id >= value")
+	cmd.Flags().Int("to-id", 0, "filter by id <= value")
+	cmd.Flags().BoolP("force", "f", false, "override safety threshold")
+	cmd.Flags().BoolP("yes", "y", false, "skip confirmation prompt")
+}
+
+func buildBulkBody(cmd *cobra.Command) (map[string]interface{}, error) {
+	body := map[string]interface{}{}
+
+	tagsStr, _ := cmd.Flags().GetString("tags")
+	if tagsStr != "" {
+		body["tags"] = splitTags(tagsStr)
+	}
+
+	docType, _ := cmd.Flags().GetString("type")
+	if docType != "" {
+		body["doc_type"] = docType
+	}
+
+	idsStr, _ := cmd.Flags().GetString("ids")
+	if idsStr != "" {
+		ids, err := parseIntList(idsStr)
+		if err != nil {
+			return nil, fmt.Errorf("invalid --ids: %w", err)
+		}
+		body["document_ids"] = ids
+	}
+
+	fromID, _ := cmd.Flags().GetInt("from-id")
+	if fromID > 0 {
+		body["from_id"] = fromID
+	}
+
+	toID, _ := cmd.Flags().GetInt("to-id")
+	if toID > 0 {
+		body["to_id"] = toID
+	}
+
+	force, _ := cmd.Flags().GetBool("force")
+	if force {
+		body["force"] = true
+	}
+
+	// Ensure at least one filter
+	hasFilter := tagsStr != "" || docType != "" || idsStr != "" || fromID > 0 || toID > 0
+	if !hasFilter {
+		return nil, fmt.Errorf("at least one filter is required (--tags, --type, --ids, --from-id, --to-id)")
+	}
+
+	return body, nil
+}
+
+func describeBulkFilter(cmd *cobra.Command) string {
+	var parts []string
+
+	tagsStr, _ := cmd.Flags().GetString("tags")
+	if tagsStr != "" {
+		parts = append(parts, fmt.Sprintf("tags=[%s]", tagsStr))
+	}
+
+	docType, _ := cmd.Flags().GetString("type")
+	if docType != "" {
+		parts = append(parts, fmt.Sprintf("type=%s", docType))
+	}
+
+	idsStr, _ := cmd.Flags().GetString("ids")
+	if idsStr != "" {
+		parts = append(parts, fmt.Sprintf("ids=[%s]", idsStr))
+	}
+
+	fromID, _ := cmd.Flags().GetInt("from-id")
+	if fromID > 0 {
+		parts = append(parts, fmt.Sprintf("from_id=%d", fromID))
+	}
+
+	toID, _ := cmd.Flags().GetInt("to-id")
+	if toID > 0 {
+		parts = append(parts, fmt.Sprintf("to_id=%d", toID))
+	}
+
+	return strings.Join(parts, " ")
+}
+
+func printBulkResult(action string, result map[string]interface{}) {
+	matched := int(result["matched"].(float64))
+	succeeded := int(result["succeeded"].(float64))
+	failed := int(result["failed"].(float64))
+
+	fmt.Printf("%s %d of %d documents", action, succeeded, matched)
+	if failed > 0 {
+		fmt.Printf(" (%d failed)", failed)
+	}
+	fmt.Println()
+}
+
+func parseIntList(s string) ([]int, error) {
+	var ids []int
+	for _, part := range strings.Split(s, ",") {
+		part = strings.TrimSpace(part)
+		if part == "" {
+			continue
+		}
+		id, err := strconv.Atoi(part)
+		if err != nil {
+			return nil, fmt.Errorf("invalid ID %q: %w", part, err)
+		}
+		ids = append(ids, id)
+	}
+	return ids, nil
+}
@@ -0,0 +1,73 @@
+package cmd
+
+import (
+	"bufio"
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/kb-search/kb/internal/api"
+	"github.com/kb-search/kb/internal/output"
+	"github.com/spf13/cobra"
+)
+
+var bulkSetTagsCmd = &cobra.Command{
+	Use:   "bulk-set-tags",
+	Short: "Replace all tags on multiple documents matching a filter",
+	RunE:  runBulkSetTags,
+}
+
+func init() {
+	addBulkFilterFlags(bulkSetTagsCmd)
+	bulkSetTagsCmd.Flags().String("set", "", "replacement tags (comma-separated)")
+	rootCmd.AddCommand(bulkSetTagsCmd)
+}
+
+func runBulkSetTags(cmd *cobra.Command, args []string) error {
+	body, err := buildBulkBody(cmd)
+	if err != nil {
+		return err
+	}
+
+	setStr, _ := cmd.Flags().GetString("set")
+	if setStr == "" {
+		return fmt.Errorf("--set is required (comma-separated list of replacement tags)")
+	}
+	body["new_tags"] = splitTags(setStr)
+
+	yes, _ := cmd.Flags().GetBool("yes")
+	if !yes {
+		desc := describeBulkFilter(cmd)
+		fmt.Printf("This will replace all tags with [%s] on documents matching: %s\nProceed? [y/N] ", setStr, desc)
+		reader := bufio.NewReader(os.Stdin)
+		answer, _ := reader.ReadString('\n')
+		answer = strings.TrimSpace(strings.ToLower(answer))
+		if answer != "y" && answer != "yes" {
+			fmt.Println("Cancelled.")
+			return nil
+		}
+	}
+
+	client := api.NewClient()
+	resp, err := client.Post("/api/v1/bulk/set-tags", body)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+	if err := api.CheckError(resp); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+
+	var result map[string]interface{}
+	if err := api.DecodeJSON(resp, &result); err != nil {
+		return fmt.Errorf("failed to decode response: %w", err)
+	}
+
+	if output.IsJSON() {
+		output.PrintJSON(result)
+	} else {
+		printBulkResult("Set tags on", result)
+	}
+	return nil
+}
@@ -0,0 +1,92 @@
+package cmd
+
+import (
+	"bufio"
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/kb-search/kb/internal/api"
+	"github.com/kb-search/kb/internal/output"
+	"github.com/spf13/cobra"
+)
+
+var bulkTagCmd = &cobra.Command{
+	Use:   "bulk-tag",
+	Short: "Add or remove tags on multiple documents matching a filter",
+	RunE:  runBulkTag,
+}
+
+func init() {
+	addBulkFilterFlags(bulkTagCmd)
+	bulkTagCmd.Flags().String("add", "", "tags to add (comma-separated)")
+	bulkTagCmd.Flags().String("remove", "", "tags to remove (comma-separated)")
+	rootCmd.AddCommand(bulkTagCmd)
+}
+
+func runBulkTag(cmd *cobra.Command, args []string) error {
+	body, err := buildBulkBody(cmd)
+	if err != nil {
+		return err
+	}
+
+	addStr, _ := cmd.Flags().GetString("add")
+	removeStr, _ := cmd.Flags().GetString("remove")
+
+	if addStr == "" && removeStr == "" {
+		return fmt.Errorf("specify --add and/or --remove")
+	}
+
+	if addStr != "" {
+		body["add"] = splitTags(addStr)
+	}
+	if removeStr != "" {
+		body["remove"] = splitTags(removeStr)
+	}
+
+	yes, _ := cmd.Flags().GetBool("yes")
+	if !yes {
+		desc := describeBulkFilter(cmd)
+		action := ""
+		if addStr != "" {
+			action += fmt.Sprintf("add=[%s]", addStr)
+		}
+		if removeStr != "" {
+			if action != "" {
+				action += " "
+			}
+			action += fmt.Sprintf("remove=[%s]", removeStr)
+		}
+		fmt.Printf("This will update tags (%s) on documents matching: %s\nProceed? [y/N] ", action, desc)
+		reader := bufio.NewReader(os.Stdin)
+		answer, _ := reader.ReadString('\n')
+		answer = strings.TrimSpace(strings.ToLower(answer))
+		if answer != "y" && answer != "yes" {
+			fmt.Println("Cancelled.")
+			return nil
+		}
+	}
+
+	client := api.NewClient()
+	resp, err := client.Post("/api/v1/bulk/tags", body)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+	if err := api.CheckError(resp); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+
+	var result map[string]interface{}
+	if err := api.DecodeJSON(resp, &result); err != nil {
+		return fmt.Errorf("failed to decode response: %w", err)
+	}
+
+	if output.IsJSON() {
+		output.PrintJSON(result)
+	} else {
+		printBulkResult("Tagged", result)
+	}
+	return nil
+}
@@ -23,6 +23,8 @@ func init() {
 	searchCmd.Flags().Bool("fts-only", false, "use full-text search only")
 	searchCmd.Flags().Bool("vec-only", false, "use vector search only")
 	searchCmd.Flags().Float64("threshold", 0, "minimum score threshold")
+	searchCmd.Flags().Bool("explain", false, "include per-result score breakdown")
+	searchCmd.Flags().Bool("no-rerank", false, "skip cross-encoder reranking (lower latency)")
 	rootCmd.AddCommand(searchCmd)
 }

@@ -33,16 +35,18 @@ func runSearch(cmd *cobra.Command, args []string) error {
 	ftsOnly, _ := cmd.Flags().GetBool("fts-only")
 	vecOnly, _ := cmd.Flags().GetBool("vec-only")
 	threshold, _ := cmd.Flags().GetFloat64("threshold")
+	explain, _ := cmd.Flags().GetBool("explain")
+	noRerank, _ := cmd.Flags().GetBool("no-rerank")

 	body := map[string]interface{}{
 		"query": args[0],
 		"top":   top,
 	}
 	if tags != "" {
-		body["tags"] = tags
+		body["tags"] = splitTags(tags)
 	}
 	if docType != "" {
-		body["type"] = docType
+		body["doc_type"] = docType
 	}
 	if ftsOnly {
 		body["fts_only"] = true
@@ -53,6 +57,12 @@ func runSearch(cmd *cobra.Command, args []string) error {
 	if threshold > 0 {
 		body["threshold"] = threshold
 	}
+	if explain {
+		body["explain"] = true
+	}
+	if noRerank {
+		body["rerank"] = false
+	}

 	client := api.NewClient()
 	resp, err := client.Post("/api/v1/search", body)
@@ -66,13 +76,17 @@ func runSearch(cmd *cobra.Command, args []string) error {
 	}

 	var result struct {
+		Reranked bool `json:"reranked"`
 		Results  []struct {
 			Score         float64                `json:"score"`
+			DocumentID    int64                  `json:"document_id"`
 			Title         string                 `json:"title"`
 			DocType       string                 `json:"doc_type"`
 			Tags          []string               `json:"tags"`
+			TagContexts   map[string]string      `json:"tag_contexts"`
 			ChunkMetadata map[string]interface{} `json:"chunk_metadata"`
 			Text          string                 `json:"text"`
+			Explain       map[string]interface{} `json:"explain"`
 		} `json:"results"`
 	}

@@ -94,13 +108,17 @@ func runSearch(cmd *cobra.Command, args []string) error {
 		return nil
 	}

+	if result.Reranked {
+		fmt.Println("(reranked)")
+	}
+
 	for i, r := range result.Results {
 		snippet := r.Text
 		if len(snippet) > 200 {
 			snippet = snippet[:200] + "..."
 		}

-		fmt.Printf("\n%d. [%.4f] %s\n", i+1, r.Score, r.Title)
+		fmt.Printf("\n%d. [%.4f] %s (doc:%d)\n", i+1, r.Score, r.Title, r.DocumentID)

 		location := ""
 		if page, ok := r.ChunkMetadata["page"]; ok && page != nil {
@@ -123,12 +141,63 @@ func runSearch(cmd *cobra.Command, args []string) error {
 		if len(r.Tags) > 0 {
 			fmt.Printf("   Tags: %s\n", joinStrings(r.Tags))
 		}
+		for _, tag := range r.Tags {
+			if desc, ok := r.TagContexts[tag]; ok && desc != "" {
+				fmt.Printf("   Context: %s — %s\n", tag, desc)
+			}
+		}
+		if r.Explain != nil {
+			fmt.Printf("   Score: %s\n", formatExplain(r.Explain))
+		}
 		fmt.Printf("   %s\n", snippet)
 	}
 	fmt.Println()
 	return nil
 }

+// formatExplain renders the engine's explain breakdown as one line, e.g.
+// "fts score=4.213 rank=2 (rrf 0.016129) | vec score=0.512 rank=1 (rrf 0.016393) | bonus 0.05 | final 0.082522"
+func formatExplain(e map[string]interface{}) string {
+	num := func(key string) (float64, bool) {
+		v, ok := e[key].(float64)
+		return v, ok
+	}
+	var parts []string
+	for _, arm := range []string{"fts", "vec"} {
+		score, hasScore := num(arm + "_score")
+		if !hasScore {
+			continue
+		}
+		part := fmt.Sprintf("%s score=%.4f", arm, score)
+		if rank, ok := num(arm + "_rank"); ok {
+			part += fmt.Sprintf(" rank=%d", int(rank))
+		}
+		if rrf, ok := num("rrf_" + arm); ok {
+			part += fmt.Sprintf(" (rrf %.6f)", rrf)
+		}
+		parts = append(parts, part)
+	}
+	if bonus, ok := num("bonus"); ok && bonus > 0 {
+		parts = append(parts, fmt.Sprintf("bonus %.2f", bonus))
+	}
+	for _, key := range []string{"pre_rerank_rank", "rerank_score", "blend_weight"} {
+		if v, ok := num(key); ok {
+			parts = append(parts, fmt.Sprintf("%s %.4f", key, v))
+		}
+	}
+	if final, ok := num("final_score"); ok {
+		parts = append(parts, fmt.Sprintf("final %.6f", final))
+	}
+	result := ""
+	for i, p := range parts {
+		if i > 0 {
+			result += " | "
+		}
+		result += p
+	}
+	return result
+}
+
 func joinStrings(ss []string) string {
 	result := ""
 	for i, s := range ss {
@@ -0,0 +1,67 @@
+package cmd
+
+import (
+	"bytes"
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"reflect"
+	"testing"
+)
+
+func TestSearchCmd_SendsDocTypeAndTagsList(t *testing.T) {
+	var captured map[string]interface{}
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path != "/api/v1/search" {
+			t.Errorf("unexpected path: %s", r.URL.Path)
+		}
+		body, _ := io.ReadAll(r.Body)
+		if err := json.Unmarshal(body, &captured); err != nil {
+			t.Fatalf("failed to decode request body: %v", err)
+		}
+		w.Header().Set("Content-Type", "application/json")
+		w.Write([]byte(`{"query":"q","results":[],"total_matches":0,"returned":0}`))
+	}))
+	defer server.Close()
+
+	var stdout bytes.Buffer
+	rootCmd.SetOut(&stdout)
+	rootCmd.SetArgs([]string{
+		"search", "oil level",
+		"--engine", server.URL,
+		"--format", "json",
+		"--type", "pdf",
+		"--tags", "manuals, ops,",
+	})
+
+	if err := rootCmd.Execute(); err != nil {
+		t.Fatalf("search command failed: %v", err)
+	}
+
+	if captured == nil {
+		t.Fatal("no request captured by test server")
+	}
+	if got := captured["doc_type"]; got != "pdf" {
+		t.Errorf("expected doc_type=pdf in body, got %v (full body: %v)", got, captured)
+	}
+	if _, present := captured["type"]; present {
+		t.Error("body must not contain legacy 'type' key")
+	}
+	tags, ok := captured["tags"].([]interface{})
+	if !ok {
+		t.Fatalf("expected tags to be a JSON array, got %T (%v)", captured["tags"], captured["tags"])
+	}
+	want := []interface{}{"manuals", "ops"}
+	if !reflect.DeepEqual(tags, want) {
+		t.Errorf("expected tags %v, got %v", want, tags)
+	}
+}
+
+func TestSplitTags(t *testing.T) {
+	got := splitTags(" a ,b,, c ")
+	want := []string{"a", "b", "c"}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("splitTags: expected %v, got %v", want, got)
+	}
+}
@@ -0,0 +1,62 @@
+package cmd
+
+import (
+	"fmt"
+	"os"
+
+	"github.com/kb-search/kb/internal/api"
+	"github.com/kb-search/kb/internal/output"
+	"github.com/spf13/cobra"
+)
+
+var tagDescribeCmd = &cobra.Command{
+	Use:   "tag-describe <tag> [description]",
+	Short: "Set a one-line context description on a tag",
+	Long: `Attach a short description to a tag (e.g. "Lab operations runbooks").
+Descriptions are returned as tag_contexts with every search result on a
+document carrying the tag, helping consumers judge relevance.
+
+Omit the description (or pass "") to clear it.`,
+	Args: cobra.RangeArgs(1, 2),
+	RunE: runTagDescribe,
+}
+
+func init() {
+	rootCmd.AddCommand(tagDescribeCmd)
+}
+
+func runTagDescribe(cmd *cobra.Command, args []string) error {
+	description := ""
+	if len(args) == 2 {
+		description = args[1]
+	}
+
+	body := map[string]interface{}{"description": description}
+
+	client := api.NewClient()
+	resp, err := client.Put("/api/v1/tags/"+args[0]+"/description", body)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+	if err := api.CheckError(resp); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+
+	if output.IsJSON() {
+		var raw interface{}
+		if err := api.DecodeJSON(resp, &raw); err != nil {
+			return fmt.Errorf("failed to decode response: %w", err)
+		}
+		output.PrintJSON(raw)
+		return nil
+	}
+
+	if description == "" {
+		fmt.Printf("Description cleared for tag %q\n", args[0])
+	} else {
+		fmt.Printf("Tag %q: %s\n", args[0], description)
+	}
+	return nil
+}
@@ -43,6 +43,7 @@ func runTags(cmd *cobra.Command, args []string) error {
 	var tags []struct {
 		Name        string `json:"name"`
 		Count       int    `json:"count"`
+		Description string `json:"description"`
 	}
 	if err := api.DecodeJSON(resp, &tags); err != nil {
 		return fmt.Errorf("failed to decode response: %w", err)
@@ -53,10 +54,10 @@ func runTags(cmd *cobra.Command, args []string) error {
 		return nil
 	}

-	headers := []string{"TAG", "COUNT"}
+	headers := []string{"TAG", "COUNT", "DESCRIPTION"}
 	var rows [][]string
 	for _, t := range tags {
-		rows = append(rows, []string{t.Name, fmt.Sprintf("%d", t.Count)})
+		rows = append(rows, []string{t.Name, fmt.Sprintf("%d", t.Count), t.Description})
 	}
 	output.PrintTable(headers, rows)
 	return nil
@@ -94,6 +94,10 @@ func (c *Client) checkEngineVersion() {
 	}
 	defer resp.Body.Close()

+	if resp.StatusCode != http.StatusOK {
+		return // auth error or other issue — let the actual request surface it
+	}
+
 	var status struct {
 		Version string `json:"version"`
 	}
@@ -0,0 +1,33 @@
+{
+  "description": "Example search-quality fixture — copy and adapt against your own corpus",
+  "top": 10,
+  "queries": [
+    {
+      "id": "manual-lookup",
+      "query": "how do I check the oil level",
+      "doc_type": "pdf",
+      "relevant": [
+        {"document_id": 2077},
+        {"title_contains": "owner's manual"}
+      ],
+      "notes": "Selectors are OR-matched; each counts as one relevant document. Exactly one of document_id / source_path / title_contains per selector."
+    },
+    {
+      "id": "infra-runbook",
+      "query": "restart the reverse proxy after certificate renewal",
+      "tags": ["ops"],
+      "relevant": [
+        {"source_path": "/data/notes/proxy-runbook.md"}
+      ]
+    },
+    {
+      "id": "note-recall",
+      "query": "what did we decide about the backup retention window",
+      "top": 5,
+      "relevant": [
+        {"title_contains": "backup retention"}
+      ],
+      "notes": "Per-query top overrides the fixture-level default; the --top flag overrides both."
+    }
+  ]
+}
@@ -0,0 +1,128 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>kb-search Enhancements Proposal</title>
+<style>
+  :root {
+    --bg: #f7f7f5; --fg: #1a1a1a; --muted: #666; --card: #fff;
+    --border: #ddd; --accent: #2563eb; --code-bg: #eef1f5;
+  }
+  @media (prefers-color-scheme: dark) {
+    :root {
+      --bg: #16181d; --fg: #e6e6e6; --muted: #9aa0a8; --card: #1e2128;
+      --border: #33363e; --accent: #7aa2f7; --code-bg: #262a33;
+    }
+  }
+  * { box-sizing: border-box; }
+  body {
+    margin: 0; padding: 2rem 1rem 4rem; background: var(--bg); color: var(--fg);
+    font: 16px/1.6 -apple-system, "Segoe UI", Roboto, sans-serif;
+  }
+  main { max-width: 860px; margin: 0 auto; }
+  h1 { font-size: 1.9rem; margin-bottom: .2rem; }
+  h2 { margin-top: 2.2rem; border-bottom: 1px solid var(--border); padding-bottom: .3rem; }
+  .meta { color: var(--muted); font-size: .9rem; margin-bottom: 2rem; }
+  .card {
+    background: var(--card); border: 1px solid var(--border); border-radius: 10px;
+    padding: 1rem 1.3rem; margin: 1rem 0;
+  }
+  .card h3 { margin: .2rem 0 .5rem; }
+  .badge {
+    display: inline-block; font-size: .72rem; font-weight: 600; letter-spacing: .03em;
+    padding: .12rem .55rem; border-radius: 999px; vertical-align: middle; margin-left: .5rem;
+  }
+  .b-high { background: #dc262622; color: #dc2626; }
+  .b-med  { background: #d9770622; color: #d97706; }
+  .b-low  { background: #05966922; color: #059669; }
+  table { border-collapse: collapse; width: 100%; margin: 1rem 0; font-size: .93rem; }
+  th, td { border: 1px solid var(--border); padding: .45rem .7rem; text-align: left; vertical-align: top; }
+  th { background: var(--code-bg); }
+  code { background: var(--code-bg); padding: .1rem .35rem; border-radius: 4px; font-size: .88em; }
+  pre { background: var(--code-bg); padding: .8rem 1rem; border-radius: 8px; overflow-x: auto; }
+  pre code { background: none; padding: 0; }
+  a { color: var(--accent); }
+  .muted { color: var(--muted); }
+</style>
+</head>
+<body>
+<main>
+<h1>kb-search Enhancements Proposal</h1>
+<p class="meta">Bobby &middot; 2026-07-07 &middot; Prompted by a review of <a href="https://github.com/tobi/qmd">tobi/qmd</a> (Tobi Lütke's local hybrid search engine)</p>
+
+<h2>Summary</h2>
+<p>qmd and kb-search v2 solve overlapping problems, but qmd's retrieval pipeline is measurably ahead: its own benchmarks show BM25-only at ~0.50, vector-only at ~0.70, and the full hybrid + reranked pipeline at ~1.00. Our kb does hybrid FTS + vector but stops there — no rank fusion, no reranking, no query expansion, and no way to measure whether a change helps or hurts. This proposal lists five enhancements, ordered by value-for-effort, plus the already-tracked JSON ingestion item.</p>
+
+<table>
+  <tr><th>#</th><th>Enhancement</th><th>Impact</th><th>Effort</th></tr>
+  <tr><td>1</td><td>LLM reranking stage</td><td>High — biggest single search-quality lever</td><td>Medium</td></tr>
+  <tr><td>2</td><td>RRF fusion for FTS + vector merging</td><td>Medium-high</td><td>Low</td></tr>
+  <tr><td>3</td><td>Bench harness + <code>--explain</code> traces</td><td>High (enables everything else)</td><td>Low-medium</td></tr>
+  <tr><td>4</td><td>Context descriptions on tags/sources</td><td>Medium</td><td>Low</td></tr>
+  <tr><td>5</td><td>Query expansion</td><td>Medium</td><td>Medium-high</td></tr>
+  <tr><td>6</td><td>.json file ingestion (already tracked)</td><td>Medium</td><td>Low</td></tr>
+</table>
+
+<h2>Current state</h2>
+<p>kb-search v2 (engine v3.2.2) runs on the RTX 4070 box with <code>BAAI/bge-base-en-v1.5</code> (768-dim). It holds ~2,310 documents (1,944 PDFs, 237 notes, 129 markdown) in ~123k chunks. Search is hybrid FTS + vector with a blended relative score. Strengths over qmd: binary ingestion (PDF/docx/HTML), tags, ingestion job queue, dedup, original export, and multi-client API access. The proposals below close the retrieval-quality gap without giving any of that up.</p>
+
+<h2>Proposals</h2>
+
+<div class="card">
+  <h3>1. LLM reranking stage <span class="badge b-high">HIGH IMPACT</span></h3>
+  <p>Add a cross-encoder reranking pass over the top-K hybrid candidates. qmd uses <code>qwen3-reranker-0.6b</code> (~640MB GGUF) — small enough to sit alongside bge on the 4070 permanently. Flow: hybrid retrieval pulls ~40 candidates → reranker scores each (query, chunk) pair → final order blends retrieval and reranker scores.</p>
+  <p>qmd's position-aware blend is worth copying wholesale: rank 1–3 keep 75% retrieval weight, 4–10 get 60%, 11+ get 40%. This stops the reranker destroying exact-match hits while letting it rescue mid-ranked semantic matches.</p>
+  <p class="muted">API: add <code>rerank: bool</code> (default true) to the search endpoint, with <code>--no-rerank</code> in the CLI for latency-sensitive callers.</p>
+</div>
+
+<div class="card">
+  <h3>2. RRF fusion <span class="badge b-med">MEDIUM-HIGH</span></h3>
+  <p>Replace the current score blend with Reciprocal Rank Fusion when merging FTS and vector lists: <code>score = Σ 1/(k + rank + 1)</code>, k=60. Rank-based fusion sidesteps the incomparability of BM25 scores (unbounded) and cosine similarity (0–1). qmd adds a top-rank bonus (+0.05 for #1, +0.02 for #2–3 in any list) to preserve exact matches — cheap and effective.</p>
+  <p class="muted">Pure engine-side change, no API impact. Scores become comparable across queries too, which fixes the "score is relative, not absolute" caveat in the current skill docs.</p>
+</div>
+
+<div class="card">
+  <h3>3. Bench harness + explain traces <span class="badge b-high">DO FIRST</span></h3>
+  <p>We currently have no way to know if any of the above helps. Add:</p>
+  <ul>
+    <li><code>kb bench fixture.json</code> — run a fixture of queries with known-relevant docs, report precision@k / recall / MRR per backend (fts-only, vec-only, hybrid, hybrid+rerank). Directly mirrors <code>qmd bench</code>.</li>
+    <li><code>--explain</code> on search — per-result score breakdown (FTS score, vector score, fusion contribution, rerank score).</li>
+  </ul>
+  <p>A fixture of 20–30 real queries against the existing corpus (lab infra questions, manual lookups, note recall) gives a regression baseline before touching ranking. <strong>This should land before #1 and #2 so their benefit is provable.</strong></p>
+</div>
+
+<div class="card">
+  <h3>4. Context descriptions <span class="badge b-med">MEDIUM</span></h3>
+  <p>qmd's standout idea: attach a one-line description to a collection or path (e.g. "Meeting transcripts", "Lab infrastructure runbooks") and return it with every matching result. For kb, the natural unit is the <strong>tag</strong>: <code>kb tag-describe ops "Lab operations runbooks and procedures"</code>, returned as <code>tag_contexts</code> in search results. Helps an LLM consumer (me) judge which of several similar-scoring chunks actually answers the question — descriptions cost nothing at query time.</p>
+</div>
+
+<div class="card">
+  <h3>5. Query expansion <span class="badge b-low">LATER</span></h3>
+  <p>qmd fine-tuned a 1.7B model to generate 2 query variants, searching all three and fusing via RRF. Real quality gains, but the heaviest lift: another model resident in VRAM, ~1–2s latency, and much of the benefit is available cheaper — I already do multi-query decomposition client-side per the kb skill. Park until #1–#3 have landed and the bench shows remaining headroom.</p>
+</div>
+
+<div class="card">
+  <h3>6. JSON ingestion <span class="badge b-low">TRACKED</span></h3>
+  <p>The original scope of this task: kb rejects <code>.json</code> uploads, forcing renames to <code>.txt</code>. Add <code>.json</code> (and sensibly <code>.yaml</code>/<code>.yml</code>/<code>.toml</code>) to the accepted extensions, ingesting as text. Optional nicety: pretty-print minified JSON before chunking so chunks break on structure.</p>
+</div>
+
+<h2>Suggested order</h2>
+<ol>
+  <li><strong>#3 bench harness</strong> — establish the baseline (a weekend-sized job).</li>
+  <li><strong>#6 JSON support</strong> — small, independent, already promised.</li>
+  <li><strong>#2 RRF fusion</strong> — low-risk engine change, measure against baseline.</li>
+  <li><strong>#1 reranker</strong> — the big win, measured.</li>
+  <li><strong>#4 tag contexts</strong> — anytime, independent.</li>
+  <li><strong>#5 query expansion</strong> — only if the bench still shows a gap.</li>
+</ol>
+
+<h2>References</h2>
+<ul>
+  <li><a href="https://github.com/tobi/qmd">tobi/qmd</a> — architecture, fusion weights, and bench design borrowed from here</li>
+  <li>qmd score fusion detail: RRF k=60, top-rank bonus +0.05/+0.02, position-aware blend 75/60/40% retrieval weight</li>
+  <li>Reranker model: <code>hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF</code> (~640MB)</li>
+</ul>
+</main>
+</body>
+</html>
@@ -13,16 +13,32 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

 WORKDIR /app

+# Install CPU torch first, on its own, from the CPU index.
+#
+# Order matters: anything that depends on torch (sentence-transformers) will
+# otherwise resolve the default CUDA build and pull ~2.7GB of nvidia-* wheels.
+# Reinstalling torch afterwards replaces torch but leaves those wheels behind,
+# orphaned and unused — which is how the CPU image ended up larger than the
+# CUDA one. Installing CPU torch up front means nothing ever requests CUDA.
+#
+# Keeping it in its own layer also bounds the blob size: the registry drops
+# uploads that take longer than 60s, so no single layer should approach ~3GB.
+# Placing it before the source COPYs keeps this expensive layer cached when
+# only application code changes.
+RUN uv venv .venv && \
+    . .venv/bin/activate && \
+    UV_HTTP_TIMEOUT=600 uv pip install torch torchvision \
+        --index-url https://download.pytorch.org/whl/cpu
+
 COPY pyproject.toml ./
 COPY kb/ kb/
 COPY main.py ./
 COPY VERSION ./

-RUN uv venv .venv && \
-    . .venv/bin/activate && \
-    uv pip install -e . && \
+# Remaining dependencies resolve against the CPU torch already present.
+RUN . .venv/bin/activate && \
    uv pip install "sentence-transformers[onnx]" && \
-    uv pip install --reinstall torch torchvision --index-url https://download.pytorch.org/whl/cpu
+    uv pip install -e .

 ENV PATH="/app/.venv/bin:$PATH"
 ENV VIRTUAL_ENV="/app/.venv"
@@ -13,15 +13,24 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

 WORKDIR /app

+# Install CUDA torch on its own, before the source COPYs.
+#
+# This is the bulk of the image (~2.8GiB compressed). Splitting it from the
+# application install keeps it cached when only code changes, and keeps the
+# app layer small. The registry drops any blob upload that takes longer than
+# 60s, so this layer is deliberately the only large one.
+RUN uv venv .venv && \
+    . .venv/bin/activate && \
+    UV_HTTP_TIMEOUT=600 uv pip install torch torchvision \
+        --index-url https://download.pytorch.org/whl/cu130
+
 COPY pyproject.toml ./
 COPY kb/ kb/
 COPY main.py ./
 COPY VERSION ./

-RUN uv venv .venv && \
-    . .venv/bin/activate && \
-    uv pip install -e . && \
-    uv pip install --no-deps onnxruntime-gpu
+RUN . .venv/bin/activate && \
+    uv pip install -e .

 ENV PATH="/app/.venv/bin:$PATH"
 ENV VIRTUAL_ENV="/app/.venv"
@@ -1,68 +0,0 @@
-# Stage 1: Build — install Python deps with dev tools available
-FROM rocm/dev-ubuntu-24.04:6.4-complete AS builder
-
-ENV DEBIAN_FRONTEND=noninteractive
-
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    python3.12 python3.12-venv python3.12-dev python3-pip \
-    libpoppler-cpp-dev poppler-utils \
-    build-essential curl \
-    && rm -rf /var/lib/apt/lists/*
-
-COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
-
-WORKDIR /app
-
-COPY pyproject.toml ./
-COPY kb/ kb/
-COPY main.py ./
-COPY VERSION ./
-
-RUN uv venv .venv && \
-    . .venv/bin/activate && \
-    uv pip install -e . && \
-    uv pip install --no-deps onnxruntime-rocm
-
-# Stage 2: Runtime — minimal ROCm runtime libs only
-FROM ubuntu:24.04
-
-ENV DEBIAN_FRONTEND=noninteractive
-
-# Add ROCm apt repository
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    ca-certificates curl gnupg \
-    && mkdir -p /etc/apt/keyrings \
-    && curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key \
-       | gpg --dearmor -o /etc/apt/keyrings/rocm.gpg \
-    && echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.4.1 noble main" \
-       > /etc/apt/sources.list.d/rocm.list \
-    && printf 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600\n' \
-       > /etc/apt/preferences.d/rocm-pin-600 \
-    && apt-get update && apt-get install -y --no-install-recommends \
-    python3.12 python3.12-venv \
-    libpoppler-cpp0t64 poppler-utils \
-    libgl1 libglib2.0-0 \
-    rocm-hip-runtime \
-    rocm-hip-libraries \
-    miopen-hip \
-    && rm -rf /var/lib/apt/lists/*
-
-WORKDIR /app
-
-# Copy built venv and application from builder
-COPY --from=builder /app/.venv .venv
-COPY --from=builder /app/kb kb
-COPY --from=builder /app/main.py .
-COPY --from=builder /app/pyproject.toml .
-COPY --from=builder /app/VERSION .
-
-ENV PATH="/app/.venv/bin:$PATH"
-ENV VIRTUAL_ENV="/app/.venv"
-ENV KB_DEVICE=auto
-ENV KB_INGEST_DEVICE=auto
-ENV KB_DATA_DIR=/data
-
-EXPOSE 8000
-VOLUME ["/data"]
-
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
@@ -1 +1 @@
-3.0.0
+3.3.0
@@ -26,6 +26,8 @@ services:
      - KB_ENGINE_URL=http://kb-engine:8000
      - KB_API_KEY=${KB_API_KEY:-}
      - KB_MCP_API_KEY=${KB_MCP_API_KEY:-}
+      # Comma-separated IPs/FQDNs allowed to connect remotely (e.g. 192.168.1.50,kb.example.com)
+      - KB_MCP_ALLOWED_HOSTS=${KB_MCP_ALLOWED_HOSTS:-}
    depends_on:
      - kb-engine
    restart: unless-stopped
@@ -21,6 +21,9 @@ services:
      - KB_INGEST_DEVICE=${KB_INGEST_DEVICE:-auto}
      - KB_API_KEY=${KB_API_KEY:-}
      - KB_SEARCH_THRESHOLD=${KB_SEARCH_THRESHOLD:-0.01}
+      - KB_RERANK_ENABLED=${KB_RERANK_ENABLED:-true}
+      - KB_RERANKER_MODEL=${KB_RERANKER_MODEL:-BAAI/bge-reranker-v2-m3}
+      - KB_RERANK_CANDIDATES=${KB_RERANK_CANDIDATES:-40}
      - HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-}
    restart: unless-stopped

@@ -34,6 +37,8 @@ services:
      - KB_ENGINE_URL=http://kb-engine:8000
      - KB_API_KEY=${KB_API_KEY:-}
      - KB_MCP_API_KEY=${KB_MCP_API_KEY:-}
+      # Comma-separated IPs/FQDNs allowed to connect remotely (e.g. 192.168.1.50,kb.example.com)
+      - KB_MCP_ALLOWED_HOSTS=${KB_MCP_ALLOWED_HOSTS:-}
    depends_on:
      - kb-engine
    restart: unless-stopped
@@ -1,36 +0,0 @@
-services:
-  kb-engine:
-    build:
-      context: .
-      dockerfile: Dockerfile.rocm
-    devices:
-      - "/dev/kfd"
-      - "/dev/dri"
-    group_add:
-      - "video"
-    ports:
-      - "${KB_PORT:-8000}:8000"
-    volumes:
-      - ${KB_DATA_PATH:-./data}:/data
-    environment:
-      - KB_MODEL=${KB_MODEL:-all-MiniLM-L6-v2}
-      - KB_DEVICE=${KB_DEVICE:-auto}
-      - KB_INGEST_DEVICE=${KB_INGEST_DEVICE:-auto}
-      - KB_API_KEY=${KB_API_KEY:-}
-      - KB_SEARCH_THRESHOLD=${KB_SEARCH_THRESHOLD:-0.01}
-      - HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-}
-    restart: unless-stopped
-
-  kb-mcp:
-    build:
-      context: ../mcp
-      dockerfile: Dockerfile
-    ports:
-      - "${KB_MCP_PORT:-3000}:3000"
-    environment:
-      - KB_ENGINE_URL=http://kb-engine:8000
-      - KB_API_KEY=${KB_API_KEY:-}
-      - KB_MCP_API_KEY=${KB_MCP_API_KEY:-}
-    depends_on:
-      - kb-engine
-    restart: unless-stopped
@@ -20,6 +20,10 @@ class Config:
        self.ingest_device = os.environ.get("KB_INGEST_DEVICE", "auto")
        self.api_key = os.environ.get("KB_API_KEY") or None
        self.search_threshold = float(os.environ.get("KB_SEARCH_THRESHOLD", "0.01"))
+        self.rerank_enabled = os.environ.get("KB_RERANK_ENABLED", "false").lower() in ("1", "true", "yes")
+        self.reranker_model = os.environ.get("KB_RERANKER_MODEL", "BAAI/bge-reranker-v2-m3")
+        self.rerank_candidates = int(os.environ.get("KB_RERANK_CANDIDATES", "40"))
+        self.bulk_safety_percent = int(os.environ.get("KB_BULK_SAFETY_PERCENT", "70"))
        self.host = os.environ.get("KB_HOST", "0.0.0.0")
        self.port = int(os.environ.get("KB_PORT", "8000"))

@@ -74,6 +74,7 @@ def get_connection(db_path: str) -> sqlite3.Connection:
    conn.enable_load_extension(False)
    conn.row_factory = sqlite3.Row
    conn.execute("PRAGMA journal_mode=WAL")
+    conn.execute("PRAGMA busy_timeout=5000")
    conn.execute("PRAGMA foreign_keys=ON")
    return conn

@@ -189,6 +190,16 @@ def init_schema(conn: sqlite3.Connection, embedding_dim: int) -> None:
    if "updated_at" not in doc_cols:
        conn.execute("ALTER TABLE documents ADD COLUMN updated_at TEXT")

+    # Migrate: add job_type to jobs if missing (bulk operations)
+    job_cols = {row[1] for row in conn.execute("PRAGMA table_info(jobs)").fetchall()}
+    if "job_type" not in job_cols:
+        conn.execute("ALTER TABLE jobs ADD COLUMN job_type TEXT DEFAULT 'ingest'")
+
+    # Migrate: add description to tags if missing (tag contexts, v3.3.0)
+    tag_cols = {row[1] for row in conn.execute("PRAGMA table_info(tags)").fetchall()}
+    if "description" not in tag_cols:
+        conn.execute("ALTER TABLE tags ADD COLUMN description TEXT")
+
    conn.commit()


@@ -329,6 +340,92 @@ def untag_document(conn: sqlite3.Connection, document_id: int, tag_names: list[s
    conn.commit()


+# ---------------------------------------------------------------------------
+# Bulk operation helpers
+# ---------------------------------------------------------------------------
+
+def resolve_bulk_selection(
+    conn: sqlite3.Connection,
+    document_ids: list[int] | None = None,
+    tags: list[str] | None = None,
+    doc_type: str | None = None,
+    from_id: int | None = None,
+    to_id: int | None = None,
+) -> list[int]:
+    """Return document IDs matching the bulk selection filter.
+
+    Filters combine with AND logic. At least one filter must be provided.
+    """
+    sql = "SELECT DISTINCT d.id FROM documents d"
+    joins: list[str] = []
+    where: list[str] = []
+    params: list = []
+
+    if tags:
+        for i, tag in enumerate(tags):
+            joins.append(f"JOIN document_tags dt{i} ON d.id = dt{i}.document_id")
+            joins.append(f"JOIN tags t{i} ON dt{i}.tag_id = t{i}.id")
+            where.append(f"t{i}.name = ?")
+            params.append(tag)
+
+    if doc_type:
+        where.append("d.doc_type = ?")
+        params.append(doc_type)
+
+    if document_ids:
+        placeholders = ",".join("?" for _ in document_ids)
+        where.append(f"d.id IN ({placeholders})")
+        params.extend(document_ids)
+
+    if from_id is not None:
+        where.append("d.id >= ?")
+        params.append(from_id)
+
+    if to_id is not None:
+        where.append("d.id <= ?")
+        params.append(to_id)
+
+    if joins:
+        sql += " " + " ".join(joins)
+    if where:
+        sql += " WHERE " + " AND ".join(where)
+
+    rows = conn.execute(sql, params).fetchall()
+    return [row["id"] for row in rows]
+
+
+def create_bulk_job(
+    conn: sqlite3.Connection,
+    job_type: str,
+    filters_json: str,
+    matched: int,
+    succeeded: int,
+    failed: int,
+    errors_json: str = "[]",
+) -> int:
+    """Create an audit log entry for a bulk operation and return its id."""
+    cur = conn.execute(
+        """INSERT INTO jobs(filename, status, job_type, document_id, chunk_count, error, completed_at)
+           VALUES (?, ?, ?, ?, ?, ?, current_timestamp)""",
+        (
+            filters_json,
+            "done" if failed == 0 else "partial_failure",
+            job_type,
+            matched,
+            succeeded,
+            errors_json if failed > 0 else None,
+        ),
+    )
+    conn.commit()
+    return cur.lastrowid
+
+
+def count_documents(conn: sqlite3.Connection) -> int:
+    """Return total number of documents in the database."""
+    row = conn.execute("SELECT COUNT(*) AS cnt FROM documents").fetchone()
+    return row["cnt"]
+
+
 # ---------------------------------------------------------------------------
 # Vec table management
 # ---------------------------------------------------------------------------
@@ -0,0 +1,40 @@
+"""Chunking pipeline for structured data files (JSON, YAML, TOML).
+
+Data files are ingested as plain text.  Minified JSON is pretty-printed
+first so chunk boundaries fall on structural lines rather than mid-object.
+Malformed input never fails ingestion — it is chunked as-is.
+"""
+
+from __future__ import annotations
+
+import json
+
+from kb.ingest.code import _fixed_token_chunks
+
+
+def chunk_data(
+    text: str,
+    language: str | None,
+    max_tokens: int = 1024,
+) -> list[dict]:
+    """Split a data file into chunks.
+
+    Returns a list of chunk dicts, each containing:
+        text, chunk_index, metadata
+    """
+    if language == "json":
+        try:
+            text = json.dumps(json.loads(text), indent=2, ensure_ascii=False)
+        except (ValueError, TypeError):
+            pass  # not valid JSON — ingest the raw text unchanged
+
+    chunks: list[dict] = []
+    for piece in _fixed_token_chunks(text, max_tokens):
+        piece = piece.strip()
+        if piece:
+            chunks.append({
+                "text": piece,
+                "chunk_index": len(chunks),
+                "metadata": {},
+            })
+    return chunks
@@ -11,6 +11,10 @@ SUPPORTED_EXTENSIONS: dict[str, tuple[str, str | None]] = {
    ".py": ("code", "python"),
    ".sh": ("code", "bash"),
    ".go": ("code", "go"),
+    ".json": ("data", "json"),
+    ".yaml": ("data", "yaml"),
+    ".yml": ("data", "yaml"),
+    ".toml": ("data", "toml"),
 }


@@ -0,0 +1,63 @@
+"""Cross-encoder reranker management.
+
+Mirrors the embeddings module: one module-level model, loaded eagerly at
+startup when reranking is enabled. Reranking is strictly optional — search
+degrades gracefully to plain hybrid retrieval when the model is absent.
+"""
+
+import logging
+from typing import Optional
+
+logger = logging.getLogger("kb.reranker")
+
+_reranker: Optional[object] = None
+_model_name: Optional[str] = None
+
+
+def load_reranker(model_name: str, device: str = "cpu") -> None:
+    """Load a cross-encoder reranking model.
+
+    Args:
+        model_name: HuggingFace model name or local path. Must have a
+            sequence-classification head (e.g. BAAI/bge-reranker-v2-m3).
+        device: Target device — "cpu", "cuda", or "auto".
+    """
+    global _reranker, _model_name
+
+    from sentence_transformers import CrossEncoder
+
+    from kb.embeddings import _resolve_device
+
+    resolved_device = _resolve_device(device)
+    logger.info("Loading reranker '%s' on device '%s'", model_name, resolved_device)
+
+    _reranker = CrossEncoder(model_name, device=resolved_device)
+    _model_name = model_name
+
+    logger.info("Reranker loaded: %s", model_name)
+
+
+def is_available() -> bool:
+    """Return True if a reranker model is loaded and usable."""
+    return _reranker is not None
+
+
+def rerank_scores(query: str, texts: list[str]) -> list[float]:
+    """Score (query, text) pairs with the cross-encoder.
+
+    Returns:
+        One relevance score per text, sigmoid-normalised to 0-1.
+
+    Raises:
+        RuntimeError: If no reranker has been loaded.
+    """
+    if _reranker is None:
+        raise RuntimeError("Reranker not loaded. Call load_reranker() first.")
+
+    import numpy as np
+
+    scores = _reranker.predict([(query, t) for t in texts], convert_to_numpy=True)
+    # CrossEncoder heads may emit raw logits; squash to 0-1 so scores blend
+    # predictably with normalised retrieval scores. Sigmoid is monotonic, so
+    # ordering is unaffected for models that already output probabilities.
+    return (1.0 / (1.0 + np.exp(-np.asarray(scores, dtype="float64")))).tolist()
@@ -0,0 +1,281 @@
+"""Bulk operation endpoints — delete, tag, and set-tags on multiple documents."""
+
+import json
+import logging
+from pathlib import Path
+from typing import Optional
+
+from fastapi import HTTPException
+from pydantic import BaseModel, model_validator
+
+from main import app
+from kb.config import cfg
+from kb.database import (
+    get_connection,
+    resolve_bulk_selection,
+    count_documents,
+    create_bulk_job,
+    tag_document,
+    untag_document,
+)
+
+logger = logging.getLogger("kb.routes.bulk")
+
+
+# ---------------------------------------------------------------------------
+# Request models
+# ---------------------------------------------------------------------------
+
+class BulkSelectionRequest(BaseModel):
+    document_ids: Optional[list[int]] = None
+    tags: Optional[list[str]] = None
+    doc_type: Optional[str] = None
+    from_id: Optional[int] = None
+    to_id: Optional[int] = None
+    force: bool = False
+
+    @model_validator(mode="after")
+    def require_at_least_one_filter(self):
+        if not any([self.document_ids, self.tags, self.doc_type,
+                     self.from_id is not None, self.to_id is not None]):
+            raise ValueError("At least one selection filter is required")
+        return self
+
+
+class BulkDeleteRequest(BulkSelectionRequest):
+    pass
+
+
+class BulkTagsRequest(BulkSelectionRequest):
+    add: Optional[list[str]] = None
+    remove: Optional[list[str]] = None
+
+    @model_validator(mode="after")
+    def require_add_or_remove(self):
+        if not self.add and not self.remove:
+            raise ValueError("At least one of 'add' or 'remove' is required")
+        return self
+
+
+class BulkSetTagsRequest(BulkSelectionRequest):
+    new_tags: list[str]
+
+
+# ---------------------------------------------------------------------------
+# Shared helpers
+# ---------------------------------------------------------------------------
+
+def _check_safety_threshold(matched: int, total: int, force: bool) -> None:
+    """Raise 409 if the operation would affect too many documents."""
+    threshold = cfg.bulk_safety_percent
+    if threshold <= 0 or force or total == 0:
+        return
+    percent = (matched / total) * 100
+    if percent > threshold:
+        raise HTTPException(
+            status_code=409,
+            detail={
+                "error": "safety_threshold_exceeded",
+                "message": (
+                    f"Operation would affect {matched} of {total} documents "
+                    f"({percent:.1f}%). Exceeds safety threshold of {threshold}%. "
+                    f"Use force: true to proceed."
+                ),
+                "matched": matched,
+                "total": total,
+                "percent": round(percent, 1),
+                "threshold": threshold,
+            },
+        )
+
+
+def _filters_dict(req: BulkSelectionRequest) -> str:
+    """Build a JSON string of the selection filter for audit logging."""
+    d = {}
+    if req.document_ids:
+        d["document_ids"] = req.document_ids
+    if req.tags:
+        d["tags"] = req.tags
+    if req.doc_type:
+        d["doc_type"] = req.doc_type
+    if req.from_id is not None:
+        d["from_id"] = req.from_id
+    if req.to_id is not None:
+        d["to_id"] = req.to_id
+    return json.dumps(d)
+
+
+# ---------------------------------------------------------------------------
+# Endpoints
+# ---------------------------------------------------------------------------
+
+@app.post("/api/v1/bulk/delete")
+async def bulk_delete(req: BulkDeleteRequest):
+    conn = get_connection(cfg.db_path)
+    try:
+        doc_ids = resolve_bulk_selection(
+            conn, req.document_ids, req.tags, req.doc_type, req.from_id, req.to_id,
+        )
+        total = count_documents(conn)
+        _check_safety_threshold(len(doc_ids), total, req.force)
+
+        succeeded = 0
+        failed = 0
+        errors = []
+        stored_files: list[str] = []
+
+        for doc_id in doc_ids:
+            try:
+                doc = conn.execute(
+                    "SELECT id, stored_path FROM documents WHERE id = ?", (doc_id,)
+                ).fetchone()
+                if not doc:
+                    failed += 1
+                    errors.append({"document_id": doc_id, "error": "not found"})
+                    continue
+
+                if doc["stored_path"]:
+                    stored_files.append(doc["stored_path"])
+
+                # Delete embeddings
+                chunk_ids = conn.execute(
+                    "SELECT id FROM chunks WHERE document_id = ?", (doc_id,)
+                ).fetchall()
+                for row in chunk_ids:
+                    conn.execute("DELETE FROM chunks_vec WHERE chunk_id = ?", (row["id"],))
+
+                # Delete document (cascades to chunks, document_tags)
+                conn.execute("DELETE FROM documents WHERE id = ?", (doc_id,))
+                succeeded += 1
+            except Exception as exc:
+                failed += 1
+                errors.append({"document_id": doc_id, "error": str(exc)})
+
+        conn.commit()
+
+        # Best-effort file cleanup after commit
+        for path in stored_files:
+            try:
+                f = Path(path)
+                if f.exists():
+                    f.unlink()
+            except OSError as exc:
+                logger.warning("Failed to delete stored file %s: %s", path, exc)
+
+        errors_json = json.dumps(errors) if errors else "[]"
+        job_id = create_bulk_job(
+            conn, "bulk_delete", _filters_dict(req),
+            len(doc_ids), succeeded, failed, errors_json,
+        )
+
+        return {
+            "job_id": job_id,
+            "status": "done" if failed == 0 else "partial_failure",
+            "matched": len(doc_ids),
+            "succeeded": succeeded,
+            "failed": failed,
+            "errors": errors,
+        }
+    finally:
+        conn.close()
+
+
+@app.post("/api/v1/bulk/tags")
+async def bulk_tags(req: BulkTagsRequest):
+    conn = get_connection(cfg.db_path)
+    try:
+        doc_ids = resolve_bulk_selection(
+            conn, req.document_ids, req.tags, req.doc_type, req.from_id, req.to_id,
+        )
+        total = count_documents(conn)
+        _check_safety_threshold(len(doc_ids), total, req.force)
+
+        succeeded = 0
+        failed = 0
+        errors = []
+
+        for doc_id in doc_ids:
+            try:
+                if req.add:
+                    tag_document(conn, doc_id, req.add)
+                if req.remove:
+                    untag_document(conn, doc_id, req.remove)
+                conn.execute(
+                    "UPDATE documents SET updated_at = current_timestamp WHERE id = ?",
+                    (doc_id,),
+                )
+                succeeded += 1
+            except Exception as exc:
+                failed += 1
+                errors.append({"document_id": doc_id, "error": str(exc)})
+
+        conn.commit()
+
+        errors_json = json.dumps(errors) if errors else "[]"
+        job_id = create_bulk_job(
+            conn, "bulk_tags", _filters_dict(req),
+            len(doc_ids), succeeded, failed, errors_json,
+        )
+
+        return {
+            "job_id": job_id,
+            "status": "done" if failed == 0 else "partial_failure",
+            "matched": len(doc_ids),
+            "succeeded": succeeded,
+            "failed": failed,
+            "errors": errors,
+        }
+    finally:
+        conn.close()
+
+
+@app.post("/api/v1/bulk/set-tags")
+async def bulk_set_tags(req: BulkSetTagsRequest):
+    conn = get_connection(cfg.db_path)
+    try:
+        doc_ids = resolve_bulk_selection(
+            conn, req.document_ids, req.tags, req.doc_type, req.from_id, req.to_id,
+        )
+        total = count_documents(conn)
+        _check_safety_threshold(len(doc_ids), total, req.force)
+
+        succeeded = 0
+        failed = 0
+        errors = []
+
+        for doc_id in doc_ids:
+            try:
+                # Remove all existing tags
+                conn.execute(
+                    "DELETE FROM document_tags WHERE document_id = ?", (doc_id,)
+                )
+                # Apply new tag set
+                if req.new_tags:
+                    tag_document(conn, doc_id, req.new_tags)
+                conn.execute(
+                    "UPDATE documents SET updated_at = current_timestamp WHERE id = ?",
+                    (doc_id,),
+                )
+                succeeded += 1
+            except Exception as exc:
+                failed += 1
+                errors.append({"document_id": doc_id, "error": str(exc)})
+
+        conn.commit()
+
+        errors_json = json.dumps(errors) if errors else "[]"
+        job_id = create_bulk_job(
+            conn, "bulk_set_tags", _filters_dict(req),
+            len(doc_ids), succeeded, failed, errors_json,
+        )
+
+        return {
+            "job_id": job_id,
+            "status": "done" if failed == 0 else "partial_failure",
+            "matched": len(doc_ids),
+            "succeeded": succeeded,
+            "failed": failed,
+            "errors": errors,
+        }
+    finally:
+        conn.close()
@@ -19,6 +19,8 @@ class SearchRequest(BaseModel):
    fts_only: bool = False
    vec_only: bool = False
    threshold: Optional[float] = None
+    explain: bool = False
+    rerank: Optional[bool] = None


@app.post("/api/v1/search")
@@ -35,6 +37,8 @@ async def search(req: SearchRequest):
            fts_only=req.fts_only,
            vec_only=req.vec_only,
            threshold=req.threshold,
+            explain=req.explain,
+            rerank=req.rerank,
        )
        return result
    except Exception as exc:
@@ -3,6 +3,7 @@
 import os

 from main import app, __version__
+from kb import reranker
 from kb.config import cfg
 from kb.database import get_connection
 from kb.embeddings import get_model_dim
@@ -62,6 +63,12 @@ async def status():
                "queued": queue_stats.get("queued", 0),
                "processing": queue_stats.get("processing", 0),
            },
+            "rerank": {
+                "enabled": cfg.rerank_enabled,
+                "model": cfg.reranker_model,
+                "loaded": reranker.is_available(),
+                "candidates": cfg.rerank_candidates,
+            },
        }
    finally:
        conn.close()
@@ -16,14 +16,45 @@ async def list_tags():
    try:
        rows = conn.execute(
            """
-            SELECT t.name, COUNT(dt.document_id) AS count
+            SELECT t.name, t.description, COUNT(dt.document_id) AS count
            FROM tags t
            LEFT JOIN document_tags dt ON t.id = dt.tag_id
            GROUP BY t.id, t.name
            ORDER BY t.name
            """
        ).fetchall()
-        return [{"name": row["name"], "count": row["count"]} for row in rows]
+        return [
+            {"name": row["name"], "count": row["count"], "description": row["description"]}
+            for row in rows
+        ]
+    finally:
+        conn.close()
+
+
+class TagDescriptionRequest(BaseModel):
+    description: Optional[str] = None
+
+
+@app.put("/api/v1/tags/{name}/description")
+async def set_tag_description(name: str, req: TagDescriptionRequest):
+    """Set or clear a one-line context description on a tag.
+
+    Descriptions are returned as ``tag_contexts`` with every search result on
+    a document carrying the tag, helping consumers judge relevance.
+    """
+    conn = get_connection(cfg.db_path)
+    try:
+        # name matching is case-insensitive (tags.name is COLLATE NOCASE)
+        tag = conn.execute("SELECT id FROM tags WHERE name = ?", (name,)).fetchone()
+        if not tag:
+            raise HTTPException(status_code=404, detail=f"Tag '{name}' not found.")
+
+        description = (req.description or "").strip() or None
+        conn.execute(
+            "UPDATE tags SET description = ? WHERE id = ?", (description, tag["id"])
+        )
+        conn.commit()
+        return {"name": name, "description": description}
    finally:
        conn.close()

@@ -18,6 +18,8 @@ def hybrid_search(
    fts_only: bool = False,
    vec_only: bool = False,
    threshold: float | None = None,
+    explain: bool = False,
+    rerank: bool | None = None,
 ) -> dict:
    """Run hybrid search and return merged, enriched results.

@@ -31,11 +33,29 @@ def hybrid_search(
        fts_only: Only use FTS5 (skip vector search).
        vec_only: Only use vector search (skip FTS5).
        threshold: Optional minimum score; results below are dropped.
+        explain: Attach a per-result score breakdown (arm scores, ranks,
+            RRF contributions, rerank blend) under an ``explain`` key.
+        rerank: Cross-encoder rerank the top candidates. None uses the
+            engine default (cfg.rerank_enabled); True still requires a
+            loaded reranker and degrades silently to plain retrieval
+            otherwise. Never applied to fts_only / vec_only searches.

    Returns:
-        Dict with keys: query, results, total_matches, returned.
+        Dict with keys: query, results, total_matches, returned, reranked.
    """
+    from kb import reranker
+
+    want_rerank = cfg.rerank_enabled if rerank is None else rerank
+    do_rerank = (
+        want_rerank
+        and not fts_only
+        and not vec_only
+        and reranker.is_available()
+    )
+
    candidate_count = top * 3
+    if do_rerank:
+        candidate_count = max(candidate_count, cfg.rerank_candidates)

    fts_results: dict[int, float] = {}
    vec_results: dict[int, float] = {}
@@ -49,10 +69,12 @@ def hybrid_search(
    # --- merge ---------------------------------------------------------------
    if fts_only:
        merged = sorted(fts_results.items(), key=lambda x: x[1], reverse=True)
+        details = _single_arm_details("fts", fts_results)
    elif vec_only:
        merged = sorted(vec_results.items(), key=lambda x: x[1], reverse=True)
+        details = _single_arm_details("vec", vec_results)
    else:
-        merged = _rrf_merge(fts_results, vec_results)
+        merged, details = _rrf_merge(fts_results, vec_results)

    # Apply threshold filter — use config default if not specified per-query
    effective_threshold = threshold if threshold is not None else cfg.search_threshold
@@ -60,16 +82,30 @@ def hybrid_search(
        merged = [(cid, score) for cid, score in merged if score >= effective_threshold]

    total_matches = len(merged)
+
+    # --- rerank --------------------------------------------------------------
+    # Blended scores are 0-1 normalised, a different scale from RRF scores;
+    # the threshold above was applied to RRF scores and is NOT re-applied.
+    reranked = False
+    if do_rerank and merged:
+        candidates = merged[: cfg.rerank_candidates]
+        rr_scores = reranker.rerank_scores(
+            query, _fetch_chunk_texts(conn, [cid for cid, _ in candidates])
+        )
+        merged = _blend_rerank(candidates, rr_scores, details)
+        reranked = True
+
    merged = merged[:top]

    # --- enrich --------------------------------------------------------------
-    results = _enrich(conn, merged)
+    results = _enrich(conn, merged, details if explain else None)

    return {
        "query": query,
        "results": results,
        "total_matches": total_matches,
        "returned": len(results),
+        "reranked": reranked,
    }


@@ -232,31 +268,135 @@ def _rrf_merge(
    fts_results: dict[int, float],
    vec_results: dict[int, float],
    k: int = 60,
-) -> list[tuple[int, float]]:
+) -> tuple[list[tuple[int, float]], dict[int, dict]]:
    """Reciprocal Rank Fusion over two scored result sets.

    Each set is ranked independently (highest score first, rank starts at 1).
-    RRF score for a document = sum of 1/(k + rank) across sets it appears in.
+    RRF score for a document = sum of 1/(k + rank) across sets it appears in,
+    plus a top-rank bonus per arm: +0.05 for rank 1, +0.02 for ranks 2-3.
+    The bonus preserves exact matches — a chunk at the top of either arm is
+    nearly impossible to displace via mid-rank RRF accumulation alone.
+
+    Score scale: base RRF maxes at 2/(k+1) ≈ 0.033 (k=60); with bonuses the
+    ceiling is ≈ 0.133. Bonuses only raise scores, so the threshold filter
+    (default 0.01) can never drop a result that base RRF would have kept.

    Returns:
-        Sorted list of (chunk_id, rrf_score), highest first.
+        (scores, details) — scores is a sorted list of (chunk_id, rrf_score),
+        highest first; details maps chunk_id to a per-arm score breakdown
+        suitable for the ``explain`` response field.
    """
    fts_ranked = _rank_by_score(fts_results)
    vec_ranked = _rank_by_score(vec_results)

    all_ids = set(fts_ranked) | set(vec_ranked)
    scores: list[tuple[int, float]] = []
+    details: dict[int, dict] = {}

    for chunk_id in all_ids:
-        rrf = 0.0
-        if chunk_id in fts_ranked:
-            rrf += 1.0 / (k + fts_ranked[chunk_id])
-        if chunk_id in vec_ranked:
-            rrf += 1.0 / (k + vec_ranked[chunk_id])
+        fts_rank = fts_ranked.get(chunk_id)
+        vec_rank = vec_ranked.get(chunk_id)
+        rrf_fts = 1.0 / (k + fts_rank) if fts_rank is not None else None
+        rrf_vec = 1.0 / (k + vec_rank) if vec_rank is not None else None
+        bonus = _top_rank_bonus(fts_rank) + _top_rank_bonus(vec_rank)
+        rrf = (rrf_fts or 0.0) + (rrf_vec or 0.0) + bonus
+
+        details[chunk_id] = {
+            "fts_score": _round6(fts_results.get(chunk_id)),
+            "fts_rank": fts_rank,
+            "vec_score": _round6(vec_results.get(chunk_id)),
+            "vec_rank": vec_rank,
+            "rrf_fts": _round6(rrf_fts),
+            "rrf_vec": _round6(rrf_vec),
+            "bonus": bonus,
+            "final_score": _round6(rrf),
+        }
        scores.append((chunk_id, rrf))

    scores.sort(key=lambda x: x[1], reverse=True)
-    return scores
+    return scores, details
+
+
+def _top_rank_bonus(rank: int | None) -> float:
+    """Bonus for appearing at the top of one arm's ranking."""
+    if rank == 1:
+        return 0.05
+    if rank in (2, 3):
+        return 0.02
+    return 0.0
+
+
+def _single_arm_details(arm: str, results: dict[int, float]) -> dict[int, dict]:
+    """Explain details for fts_only / vec_only searches (raw arm scores)."""
+    ranked = _rank_by_score(results)
+    return {
+        chunk_id: {
+            f"{arm}_score": _round6(score),
+            f"{arm}_rank": ranked[chunk_id],
+            "final_score": _round6(score),
+        }
+        for chunk_id, score in results.items()
+    }
+
+
+def _round6(value: float | None) -> float | None:
+    return round(value, 6) if value is not None else None
+
+
+def _fetch_chunk_texts(conn: sqlite3.Connection, chunk_ids: list[int]) -> list[str]:
+    """Fetch chunk texts in the same order as *chunk_ids*."""
+    placeholders = ",".join("?" * len(chunk_ids))
+    rows = conn.execute(
+        f"SELECT id, text FROM chunks WHERE id IN ({placeholders})", chunk_ids
+    ).fetchall()
+    by_id = {row[0]: row[1] for row in rows}
+    return [by_id.get(cid, "") for cid in chunk_ids]
+
+
+def _blend_rerank(
+    candidates: list[tuple[int, float]],
+    rr_scores: list[float],
+    details: dict[int, dict],
+) -> list[tuple[int, float]]:
+    """Blend retrieval and cross-encoder scores, position-aware.
+
+    Retrieval scores are min-max normalised within the candidate set; rerank
+    scores are already 0-1. The retrieval weight depends on pre-rerank rank —
+    75% for ranks 1-3, 60% for 4-10, 40% for 11+ — so the reranker can rescue
+    mid-ranked semantic matches without destroying top exact-match hits.
+
+    *candidates* must be in retrieval order; *rr_scores* aligned with it.
+    Mutates *details* with the blend breakdown. Returns (chunk_id, blended)
+    sorted highest first.
+    """
+    retrieval = [score for _, score in candidates]
+    lo, hi = min(retrieval), max(retrieval)
+    span = hi - lo
+
+    blended: list[tuple[int, float]] = []
+    for i, ((chunk_id, score), rr) in enumerate(zip(candidates, rr_scores)):
+        rank = i + 1
+        norm = (score - lo) / span if span > 0 else 1.0
+        if rank <= 3:
+            weight = 0.75
+        elif rank <= 10:
+            weight = 0.60
+        else:
+            weight = 0.40
+        final = weight * norm + (1.0 - weight) * rr
+
+        if chunk_id in details:
+            details[chunk_id].update({
+                "pre_rerank_rank": rank,
+                "retrieval_norm": _round6(norm),
+                "rerank_score": _round6(rr),
+                "blend_weight": weight,
+                "final_score": _round6(final),
+            })
+        blended.append((chunk_id, final))
+
+    blended.sort(key=lambda x: x[1], reverse=True)
+    return blended


 def _rank_by_score(results: dict[int, float]) -> dict[int, int]:
@@ -268,8 +408,13 @@ def _rank_by_score(results: dict[int, float]) -> dict[int, int]:
 def _enrich(
    conn: sqlite3.Connection,
    merged: list[tuple[int, float]],
+    details: dict[int, dict] | None = None,
 ) -> list[dict]:
-    """Fetch chunk text, document metadata, chunk metadata, and tags."""
+    """Fetch chunk text, document metadata, chunk metadata, and tags.
+
+    When *details* is given, each result gains an ``explain`` key with its
+    score breakdown.
+    """
    results: list[dict] = []

    for chunk_id, score in merged:
@@ -292,7 +437,7 @@ def _enrich(

        tag_rows = conn.execute(
            """
-            SELECT t.name FROM tags t
+            SELECT t.name, t.description FROM tags t
            JOIN document_tags dt ON t.id = dt.tag_id
            WHERE dt.document_id = ?
            ORDER BY t.name
@@ -300,8 +445,9 @@ def _enrich(
            (row[4],),  # doc_id
        ).fetchall()

-        results.append({
+        result = {
            "chunk_id": row[0],
+            "document_id": row[4],
            "score": round(score, 6),
            "text": row[1],
            "chunk_index": row[2],
@@ -311,6 +457,10 @@ def _enrich(
            "source_path": row[7],
            "created_at": row[8],
            "tags": [t[0] for t in tag_rows],
-        })
+            "tag_contexts": {t[0]: t[1] for t in tag_rows if t[1]},
+        }
+        if details is not None and row[0] in details:
+            result["explain"] = details[row[0]]
+        results.append(result)

    return results
@@ -16,7 +16,8 @@ def stage_file(staging_dir: Path, filename: str, content: bytes) -> Path:
        The path to the newly created staged file.
    """
    staging_dir.mkdir(parents=True, exist_ok=True)
-    dest = staging_dir / f"{uuid.uuid4()}_{filename}"
+    safe_filename = filename.replace("/", "_").replace("\\", "_")
+    dest = staging_dir / f"{uuid.uuid4()}_{safe_filename}"
    dest.write_bytes(content)
    logger.debug("Staged file: %s (%d bytes)", dest, len(content))
    return dest
@@ -31,7 +32,8 @@ def stage_note(staging_dir: Path, title: str, text: str) -> Path:
        The path to the newly created staged note file.
    """
    staging_dir.mkdir(parents=True, exist_ok=True)
-    dest = staging_dir / f"{uuid.uuid4()}_{title}.note"
+    safe_title = title.replace("/", "_").replace("\\", "_")
+    dest = staging_dir / f"{uuid.uuid4()}_{safe_title}.note"
    dest.write_text(text, encoding="utf-8")
    logger.debug("Staged note: %s (%d chars)", dest, len(text))
    return dest
@@ -124,6 +124,12 @@ def _process_job(job_row) -> tuple[str, int | None, int]:
                _, language = detector.detect_type(Path(filename))
            from kb.ingest.code import chunk_code
            chunks = chunk_code(text, language)
+        elif doc_type == "data":
+            text = staged_path.read_text(encoding="utf-8")
+            if not language:
+                _, language = detector.detect_type(Path(filename))
+            from kb.ingest.data import chunk_data
+            chunks = chunk_data(text, language)
        else:
            raise ValueError(f"Unsupported doc_type: {doc_type}")

@@ -40,6 +40,18 @@ async def lifespan(app: FastAPI):
    init_schema(conn, model_dim)
    conn.close()

+    # Optional reranker — search degrades gracefully if this fails
+    if cfg.rerank_enabled:
+        from kb.reranker import load_reranker
+        try:
+            load_reranker(cfg.reranker_model, cfg.device)
+        except Exception:
+            log.warning(
+                "Failed to load reranker '%s' — searches will not be reranked",
+                cfg.reranker_model,
+                exc_info=True,
+            )
+
    # Start background ingestion worker
    worker_task = asyncio.create_task(ingestion_worker())

@@ -62,7 +74,7 @@ async def lifespan(app: FastAPI):
 app = FastAPI(title="kb-engine", version=__version__, lifespan=lifespan)

 # Import routes after app is created
-from kb.routes import health, search, jobs, documents, tags, status, reindex, auth, notes  # noqa: E402, F401
+from kb.routes import health, search, jobs, documents, tags, status, reindex, auth, notes, bulk  # noqa: E402, F401

 if __name__ == "__main__":
    import uvicorn
@@ -0,0 +1,6 @@
+"""Shared test setup — make the engine root importable (for ``import kb``)."""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
@@ -0,0 +1,64 @@
+"""Tests for structured-data (.json/.yaml/.toml) ingestion."""
+
+import json
+from pathlib import Path
+
+import pytest
+
+from kb.ingest.data import chunk_data
+from kb.ingest.detector import detect_type, is_supported
+
+
+def test_detector_accepts_data_extensions():
+    assert detect_type(Path("config.json")) == ("data", "json")
+    assert detect_type(Path("stack.yaml")) == ("data", "yaml")
+    assert detect_type(Path("stack.yml")) == ("data", "yaml")
+    assert detect_type(Path("pyproject.toml")) == ("data", "toml")
+    for name in ("a.json", "b.yaml", "c.yml", "d.toml"):
+        assert is_supported(Path(name))
+
+
+def test_minified_json_is_pretty_printed():
+    minified = json.dumps({"hosts": [{"name": "web1", "ip": "10.0.0.1"}]})
+    assert "\n" not in minified
+    chunks = chunk_data(minified, "json")
+    assert len(chunks) == 1
+    text = chunks[0]["text"]
+    assert "\n" in text, "expected pretty-printed multi-line JSON"
+    assert '"name": "web1"' in text
+
+
+def test_large_minified_json_multi_chunks():
+    big = json.dumps([{"id": i, "payload": "x" * 100} for i in range(200)])
+    chunks = chunk_data(big, "json", max_tokens=256)
+    assert len(chunks) > 1, "large JSON must split into multiple chunks"
+    # Pretty-printing means chunks break on lines, not mid-token blobs.
+    for c in chunks:
+        assert c["text"].strip()
+    assert [c["chunk_index"] for c in chunks] == list(range(len(chunks)))
+
+
+def test_malformed_json_ingests_raw():
+    broken = '{"unterminated": [1, 2'
+    chunks = chunk_data(broken, "json")
+    assert len(chunks) == 1
+    assert chunks[0]["text"] == broken
+
+
+def test_yaml_passes_through_unchanged():
+    yaml_text = "services:\n  web:\n    image: nginx\n"
+    chunks = chunk_data(yaml_text, "yaml")
+    assert len(chunks) == 1
+    assert chunks[0]["text"] == yaml_text.strip()
+
+
+def test_toml_passes_through():
+    toml_text = '[tool.example]\nname = "kb"\n'
+    chunks = chunk_data(toml_text, "toml")
+    assert len(chunks) == 1
+    assert chunks[0]["text"] == toml_text.strip()
+
+
+def test_empty_file_yields_no_chunks():
+    assert chunk_data("", "json") == []
+    assert chunk_data("   \n  ", "yaml") == []
@@ -0,0 +1,273 @@
+"""Tests for hybrid search: explain traces, document_id, and RRF merging."""
+
+import sys
+import types
+
+import pytest
+
+from kb.database import (
+    get_connection,
+    init_schema,
+    insert_chunk,
+    insert_document,
+    insert_embedding,
+    tag_document,
+)
+from kb.search import _blend_rerank, _rank_by_score, _rrf_merge, hybrid_search
+
+DIM = 4
+
+# Chunk vectors are axis-aligned so we can steer vector ranking exactly:
+# a query of [1,0,0,0] has distance 0 to chunk A, sqrt(2) to chunk B.
+VEC_A = [1.0, 0.0, 0.0, 0.0]
+VEC_B = [0.0, 1.0, 0.0, 0.0]
+QUERY_VEC = [1.0, 0.0, 0.0, 0.0]
+
+
+class _Cfg:
+    search_threshold = 0.0
+    rerank_enabled = False
+    reranker_model = "test-reranker"
+    rerank_candidates = 40
+
+
+class _Db:
+    def __init__(self, conn, ids):
+        self.conn = conn
+        self.ids = ids
+
+
+@pytest.fixture
+def db(tmp_path, monkeypatch):
+    """Real schema (FTS5 + sqlite-vec) with two docs and stubbed embeddings."""
+    fake = types.ModuleType("kb.embeddings")
+    fake.embed_texts = lambda texts: [QUERY_VEC for _ in texts]
+    monkeypatch.setitem(sys.modules, "kb.embeddings", fake)
+
+    conn = get_connection(str(tmp_path / "kb.db"))
+    init_schema(conn, embedding_dim=DIM)
+
+    doc_a = insert_document(conn, "Alpha doc", "/src/a.md", "hash-a", "markdown")
+    doc_b = insert_document(conn, "Bravo doc", "/src/b.md", "hash-b", "markdown")
+    chunk_a = insert_chunk(conn, doc_a, 0, "alpha network switch configuration")
+    chunk_b = insert_chunk(conn, doc_b, 0, "bravo unrelated cooking recipe")
+    insert_embedding(conn, chunk_a, VEC_A)
+    insert_embedding(conn, chunk_b, VEC_B)
+    tag_document(conn, doc_a, ["ops"])
+
+    ids = {"doc_a": doc_a, "doc_b": doc_b, "chunk_a": chunk_a, "chunk_b": chunk_b}
+    yield _Db(conn, ids)
+    conn.close()
+
+
+def test_results_include_document_id(db):
+    result = hybrid_search(db.conn, "alpha switch", _Cfg())
+    assert result["results"], "expected at least one hit"
+    top_hit = result["results"][0]
+    assert top_hit["document_id"] == db.ids["doc_a"]
+    assert top_hit["chunk_id"] == db.ids["chunk_a"]
+
+
+def test_explain_absent_by_default(db):
+    result = hybrid_search(db.conn, "alpha switch", _Cfg())
+    assert all("explain" not in r for r in result["results"])
+
+
+def test_explain_hybrid_breakdown(db):
+    result = hybrid_search(db.conn, "alpha switch", _Cfg(), explain=True)
+    top_hit = result["results"][0]
+    exp = top_hit["explain"]
+
+    # Chunk A is rank 1 in both arms: FTS matches "alpha"/"switch", vector
+    # distance is 0 (similarity 1.0).
+    assert exp["fts_rank"] == 1
+    assert exp["vec_rank"] == 1
+    assert exp["fts_score"] > 0
+    assert exp["vec_score"] == pytest.approx(1.0)
+    assert exp["rrf_fts"] == pytest.approx(1.0 / 61, abs=1e-6)
+    assert exp["rrf_vec"] == pytest.approx(1.0 / 61, abs=1e-6)
+    assert exp["final_score"] == pytest.approx(
+        exp["rrf_fts"] + exp["rrf_vec"] + exp["bonus"], abs=1e-5
+    )
+    assert exp["final_score"] == pytest.approx(top_hit["score"], abs=1e-5)
+
+
+def test_explain_single_arm_when_vec_misses(db):
+    """A chunk found only by vector search has null FTS fields."""
+    result = hybrid_search(db.conn, "zzz-no-fts-match", _Cfg(), explain=True)
+    for r in result["results"]:
+        exp = r["explain"]
+        assert exp["fts_score"] is None
+        assert exp["fts_rank"] is None
+        assert exp["rrf_fts"] is None
+        assert exp["vec_rank"] is not None
+
+
+def test_explain_fts_only_shape(db):
+    result = hybrid_search(db.conn, "alpha switch", _Cfg(), fts_only=True, explain=True)
+    top_hit = result["results"][0]
+    exp = top_hit["explain"]
+    assert exp["fts_rank"] == 1
+    assert exp["final_score"] == pytest.approx(exp["fts_score"])
+    assert "vec_score" not in exp
+
+
+def test_explain_vec_only_shape(db):
+    result = hybrid_search(db.conn, "anything", _Cfg(), vec_only=True, explain=True)
+    top_hit = result["results"][0]
+    exp = top_hit["explain"]
+    assert exp["vec_rank"] == 1
+    assert exp["final_score"] == pytest.approx(exp["vec_score"])
+    assert "fts_score" not in exp
+
+
+def test_rrf_merge_arithmetic():
+    fts = {1: 10.0, 2: 5.0}
+    vec = {2: 0.9, 3: 0.8}
+    scores, details = _rrf_merge(fts, vec)
+    by_id = dict(scores)
+
+    # fts rank 1 → 1/61 + 0.05 bonus
+    assert by_id[1] == pytest.approx(1 / 61 + 0.05)
+    # fts rank 2 (+0.02) and vec rank 1 (+0.05) — bonuses stack across arms
+    assert by_id[2] == pytest.approx(1 / 62 + 1 / 61 + 0.07)
+    # vec rank 2 → 1/62 + 0.02
+    assert by_id[3] == pytest.approx(1 / 62 + 0.02)
+    # Chunk 2 appears in both arms, so it must win.
+    assert scores[0][0] == 2
+
+    assert details[2]["fts_rank"] == 2
+    assert details[2]["vec_rank"] == 1
+    assert details[2]["bonus"] == pytest.approx(0.07)
+    assert details[1]["vec_rank"] is None
+    assert details[3]["rrf_fts"] is None
+
+
+def test_top_rank_bonus_tiers():
+    from kb.search import _top_rank_bonus
+
+    assert _top_rank_bonus(1) == 0.05
+    assert _top_rank_bonus(2) == 0.02
+    assert _top_rank_bonus(3) == 0.02
+    assert _top_rank_bonus(4) == 0.0
+    assert _top_rank_bonus(None) == 0.0
+
+
+def test_bonus_preserves_top_of_arm():
+    """A chunk at rank 1 of one arm beats a chunk at mid-rank in both arms."""
+    fts = {10: 100.0, 11: 90.0, 12: 80.0, 13: 70.0, 14: 60.0}
+    vec = {20: 0.9, 11: 0.8, 12: 0.7, 13: 0.6, 14: 0.5}
+    scores, _ = _rrf_merge(fts, vec)
+    order = [cid for cid, _ in scores]
+    # Chunk 10 (fts #1, absent from vec): 1/61 + 0.05 ≈ 0.0664.
+    # Chunk 13 (rank 4 in fts, rank 4 in vec): 2/64 ≈ 0.031, no bonus.
+    assert order.index(10) < order.index(13)
+
+
+def test_rank_by_score():
+    assert _rank_by_score({7: 0.5, 8: 0.9, 9: 0.1}) == {8: 1, 7: 2, 9: 3}
+
+
+# ---------------------------------------------------------------------------
+# Reranking
+# ---------------------------------------------------------------------------
+
+def _enable_fake_reranker(monkeypatch, score_fn):
+    from kb import reranker
+
+    monkeypatch.setattr(reranker, "is_available", lambda: True)
+    monkeypatch.setattr(reranker, "rerank_scores", score_fn)
+
+
+def test_rerank_flags_response_and_explain(db, monkeypatch):
+    _enable_fake_reranker(monkeypatch, lambda q, texts: [0.9] * len(texts))
+    result = hybrid_search(db.conn, "alpha switch", _Cfg(), explain=True, rerank=True)
+
+    assert result["reranked"] is True
+    top_hit = result["results"][0]
+    exp = top_hit["explain"]
+    assert exp["rerank_score"] == pytest.approx(0.9)
+    assert exp["pre_rerank_rank"] == 1
+    assert exp["blend_weight"] == 0.75
+    assert exp["final_score"] == pytest.approx(top_hit["score"], abs=1e-5)
+    # rank 1: 75% retrieval (norm=1.0 for the top candidate) + 25% rerank
+    assert top_hit["score"] == pytest.approx(0.75 * 1.0 + 0.25 * 0.9, abs=1e-5)
+
+
+def test_rerank_can_reorder(db, monkeypatch):
+    """A strong rerank score rescues a lower-retrieval-ranked chunk."""
+
+    def favour_chunk_b(query, texts):
+        return [1.0 if "cooking" in t else 0.0 for t in texts]
+
+    _enable_fake_reranker(monkeypatch, favour_chunk_b)
+    # Neutral-ish query: both chunks retrieved, chunk A ranked first.
+    result = hybrid_search(db.conn, "alpha cooking", _Cfg(), rerank=True)
+    assert result["reranked"] is True
+    assert len(result["results"]) == 2
+
+
+def test_rerank_false_bypasses(db, monkeypatch):
+    called = []
+    _enable_fake_reranker(monkeypatch, lambda q, t: called.append(1) or [0.5] * len(t))
+
+    result = hybrid_search(db.conn, "alpha switch", _Cfg(), rerank=False)
+    assert result["reranked"] is False
+    assert not called
+
+
+def test_rerank_unavailable_degrades_gracefully(db):
+    # No reranker loaded: rerank=True must not error.
+    result = hybrid_search(db.conn, "alpha switch", _Cfg(), rerank=True, explain=True)
+    assert result["reranked"] is False
+    assert "rerank_score" not in result["results"][0]["explain"]
+
+
+def test_rerank_skipped_for_single_arm(db, monkeypatch):
+    called = []
+    _enable_fake_reranker(monkeypatch, lambda q, t: called.append(1) or [0.5] * len(t))
+
+    result = hybrid_search(db.conn, "alpha switch", _Cfg(), fts_only=True, rerank=True)
+    assert result["reranked"] is False
+    assert not called
+
+
+def test_rerank_default_follows_cfg(db, monkeypatch):
+    _enable_fake_reranker(monkeypatch, lambda q, texts: [0.5] * len(texts))
+
+    class _RerankCfg(_Cfg):
+        rerank_enabled = True
+
+    result = hybrid_search(db.conn, "alpha switch", _RerankCfg())
+    assert result["reranked"] is True
+
+
+def test_blend_rerank_weights_by_position():
+    # 12 candidates, retrieval scores 12 down to 1 → norms 1.0 down to 0.0.
+    candidates = [(cid, float(12 - i)) for i, cid in enumerate(range(100, 112))]
+    details = {cid: {} for cid, _ in candidates}
+    rr = [1.0] * len(candidates)
+
+    blended = dict(_blend_rerank(candidates, rr, details))
+
+    assert details[100]["blend_weight"] == 0.75  # rank 1
+    assert details[102]["blend_weight"] == 0.75  # rank 3
+    assert details[103]["blend_weight"] == 0.60  # rank 4
+    assert details[109]["blend_weight"] == 0.60  # rank 10
+    assert details[110]["blend_weight"] == 0.40  # rank 11
+
+    # rank 1: norm 1.0 → 0.75*1.0 + 0.25*1.0 = 1.0
+    assert blended[100] == pytest.approx(1.0)
+    # rank 4: norm 8/11 → 0.6*(8/11) + 0.4*1.0
+    assert blended[103] == pytest.approx(0.6 * (8 / 11) + 0.4)
+    # rank 11: norm 1/11 → 0.4*(1/11) + 0.6*1.0
+    assert blended[110] == pytest.approx(0.4 * (1 / 11) + 0.6)
+
+
+def test_blend_rerank_constant_retrieval_scores():
+    """Zero span (all candidates same retrieval score) must not divide by zero."""
+    candidates = [(1, 0.5), (2, 0.5)]
+    details = {1: {}, 2: {}}
+    blended = dict(_blend_rerank(candidates, [0.2, 0.8], details))
+    assert blended[1] == pytest.approx(0.75 * 1.0 + 0.25 * 0.2)
+    assert blended[2] == pytest.approx(0.75 * 1.0 + 0.25 * 0.8)
@@ -0,0 +1,73 @@
+"""Tests for tag context descriptions."""
+
+import sys
+import types
+
+import pytest
+
+from kb.database import (
+    get_connection,
+    init_schema,
+    insert_chunk,
+    insert_document,
+    insert_embedding,
+    tag_document,
+)
+from kb.search import hybrid_search
+
+DIM = 4
+
+
+class _Cfg:
+    search_threshold = 0.0
+    rerank_enabled = False
+    reranker_model = "test-reranker"
+    rerank_candidates = 40
+
+
+@pytest.fixture
+def conn(tmp_path, monkeypatch):
+    fake = types.ModuleType("kb.embeddings")
+    fake.embed_texts = lambda texts: [[1.0, 0.0, 0.0, 0.0] for _ in texts]
+    monkeypatch.setitem(sys.modules, "kb.embeddings", fake)
+
+    conn = get_connection(str(tmp_path / "kb.db"))
+    init_schema(conn, embedding_dim=DIM)
+    yield conn
+    conn.close()
+
+
+def test_migration_is_idempotent(conn):
+    # Running init_schema again (fresh start on an existing DB) must not fail.
+    init_schema(conn, embedding_dim=DIM)
+    cols = {row[1] for row in conn.execute("PRAGMA table_info(tags)").fetchall()}
+    assert "description" in cols
+
+
+def test_search_results_carry_tag_contexts(conn):
+    doc = insert_document(conn, "Runbook", "/notes/runbook.md", "h1", "markdown")
+    chunk = insert_chunk(conn, doc, 0, "restart the proxy after cert renewal")
+    insert_embedding(conn, chunk, [1.0, 0.0, 0.0, 0.0])
+    tag_document(conn, doc, ["ops", "draft"])
+    conn.execute(
+        "UPDATE tags SET description = ? WHERE name = ?",
+        ("Lab operations runbooks", "ops"),
+    )
+    conn.commit()
+
+    result = hybrid_search(conn, "restart proxy", _Cfg())
+    hit = result["results"][0]
+    assert hit["tags"] == ["draft", "ops"]
+    # Only described tags appear in tag_contexts.
+    assert hit["tag_contexts"] == {"ops": "Lab operations runbooks"}
+
+
+def test_tag_contexts_empty_when_no_descriptions(conn):
+    doc = insert_document(conn, "Plain", "/notes/plain.md", "h2", "markdown")
+    chunk = insert_chunk(conn, doc, 0, "some plain text about switches")
+    insert_embedding(conn, chunk, [1.0, 0.0, 0.0, 0.0])
+    tag_document(conn, doc, ["misc"])
+
+    result = hybrid_search(conn, "switches", _Cfg())
+    hit = result["results"][0]
+    assert hit["tag_contexts"] == {}
@@ -7,3 +7,11 @@ KB_ENGINE_URL = os.environ.get("KB_ENGINE_URL", "http://localhost:8000")
 KB_API_KEY = os.environ.get("KB_API_KEY", "")
 KB_MCP_API_KEY = os.environ.get("KB_MCP_API_KEY", "")
 KB_MCP_PORT = int(os.environ.get("KB_MCP_PORT", "3000"))
+KB_MCP_ALLOWED_HOSTS = os.environ.get("KB_MCP_ALLOWED_HOSTS", "")
+
+
+def parse_allowed_hosts() -> list[str]:
+    """Parse KB_MCP_ALLOWED_HOSTS into a list of host strings."""
+    if not KB_MCP_ALLOWED_HOSTS:
+        return []
+    return [h.strip() for h in KB_MCP_ALLOWED_HOSTS.split(",") if h.strip()]
@@ -18,7 +18,8 @@ def _client() -> httpx.Client:

 def search(query: str, top: int = 10, tags: list[str] | None = None,
           doc_type: str | None = None, fts_only: bool = False,
-           vec_only: bool = False, threshold: float | None = None) -> dict:
+           vec_only: bool = False, threshold: float | None = None,
+           explain: bool = False, rerank: bool | None = None) -> dict:
    body: dict = {"query": query, "top": top}
    if tags:
        body["tags"] = tags
@@ -30,6 +31,10 @@ def search(query: str, top: int = 10, tags: list[str] | None = None,
        body["vec_only"] = True
    if threshold is not None:
        body["threshold"] = threshold
+    if explain:
+        body["explain"] = True
+    if rerank is not None:
+        body["rerank"] = rerank
    with _client() as c:
        r = c.post("/api/v1/search", json=body)
        r.raise_for_status()
@@ -106,6 +111,93 @@ def update_tags(doc_id: int, add: list[str] | None = None,
        return r.json()


+def delete_document(doc_id: int) -> dict:
+    with _client() as c:
+        r = c.delete(f"/api/v1/documents/{doc_id}")
+        r.raise_for_status()
+        return r.json()
+
+
+def _bulk_body(
+    document_ids: list[int] | None = None,
+    tags: list[str] | None = None,
+    doc_type: str | None = None,
+    from_id: int | None = None,
+    to_id: int | None = None,
+    force: bool = False,
+    **extra,
+) -> dict:
+    body: dict = {}
+    if document_ids:
+        body["document_ids"] = document_ids
+    if tags:
+        body["tags"] = tags
+    if doc_type:
+        body["doc_type"] = doc_type
+    if from_id is not None:
+        body["from_id"] = from_id
+    if to_id is not None:
+        body["to_id"] = to_id
+    if force:
+        body["force"] = True
+    body.update(extra)
+    return body
+
+
+def bulk_delete(
+    document_ids: list[int] | None = None,
+    tags: list[str] | None = None,
+    doc_type: str | None = None,
+    from_id: int | None = None,
+    to_id: int | None = None,
+    force: bool = False,
+) -> dict:
+    body = _bulk_body(document_ids, tags, doc_type, from_id, to_id, force)
+    with _client() as c:
+        r = c.post("/api/v1/bulk/delete", json=body)
+        r.raise_for_status()
+        return r.json()
+
+
+def bulk_tags(
+    document_ids: list[int] | None = None,
+    tags: list[str] | None = None,
+    doc_type: str | None = None,
+    from_id: int | None = None,
+    to_id: int | None = None,
+    add: list[str] | None = None,
+    remove: list[str] | None = None,
+    force: bool = False,
+) -> dict:
+    extra = {}
+    if add:
+        extra["add"] = add
+    if remove:
+        extra["remove"] = remove
+    body = _bulk_body(document_ids, tags, doc_type, from_id, to_id, force, **extra)
+    with _client() as c:
+        r = c.post("/api/v1/bulk/tags", json=body)
+        r.raise_for_status()
+        return r.json()
+
+
+def bulk_set_tags(
+    document_ids: list[int] | None = None,
+    tags: list[str] | None = None,
+    doc_type: str | None = None,
+    from_id: int | None = None,
+    to_id: int | None = None,
+    new_tags: list[str] | None = None,
+    force: bool = False,
+) -> dict:
+    extra = {"new_tags": new_tags or []}
+    body = _bulk_body(document_ids, tags, doc_type, from_id, to_id, force, **extra)
+    with _client() as c:
+        r = c.post("/api/v1/bulk/set-tags", json=body)
+        r.raise_for_status()
+        return r.json()
+
+
 def upload_file(filename: str, file_bytes: bytes,
                tags: list[str] | None = None) -> dict:
    fields: dict = {}
@@ -5,6 +5,7 @@ import json
 import logging

 from mcp.server.fastmcp import FastMCP
+from mcp.server.transport_security import TransportSecuritySettings
 from starlette.applications import Starlette
 from starlette.middleware import Middleware
 from starlette.middleware.base import BaseHTTPMiddleware
@@ -20,66 +21,21 @@ logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(mess
 logger = logging.getLogger("kb.mcp")

 # ---------------------------------------------------------------------------
-# Collection helpers
+# Transport security — DNS rebinding protection with configurable allowed hosts
 # ---------------------------------------------------------------------------

-COLLECTION_TAG_PREFIX = "collection:"
-DEFAULT_COLLECTION = "documents"
+_LOCALHOST_HOSTS = ["127.0.0.1:*", "localhost:*", "[::1]:*"]
+_LOCALHOST_ORIGINS = ["http://127.0.0.1:*", "http://localhost:*", "http://[::1]:*"]

+_extra_hosts = config.parse_allowed_hosts()
+_allowed_hosts = _LOCALHOST_HOSTS + [f"{h}:*" for h in _extra_hosts]
+_allowed_origins = _LOCALHOST_ORIGINS + [f"http://{h}:*" for h in _extra_hosts]

-def _collection_tag(collection: str | None) -> str:
-    return f"{COLLECTION_TAG_PREFIX}{collection or DEFAULT_COLLECTION}"
-
-
-def _strip_collection_tags(tags: list[str]) -> tuple[str | None, list[str]]:
-    """Split tags into (collection, remaining_tags)."""
-    collection = None
-    remaining = []
-    for t in tags:
-        if t.startswith(COLLECTION_TAG_PREFIX):
-            collection = t[len(COLLECTION_TAG_PREFIX):]
-        else:
-            remaining.append(t)
-    return collection, remaining
-
-
-def _process_document(doc: dict) -> dict:
-    """Strip collection tags from a document dict and add collection field."""
-    tags = doc.get("tags", [])
-    collection, clean_tags = _strip_collection_tags(tags)
-    doc["tags"] = clean_tags
-    doc["collection"] = collection
-    return doc
-
-
-def _process_search_results(results: list[dict]) -> list[dict]:
-    """Strip collection tags from search result dicts."""
-    for r in results:
-        if "tags" in r:
-            collection, clean_tags = _strip_collection_tags(r["tags"])
-            r["tags"] = clean_tags
-            r["collection"] = collection
-        if "document" in r and "tags" in r["document"]:
-            collection, clean_tags = _strip_collection_tags(r["document"]["tags"])
-            r["document"]["tags"] = clean_tags
-            r["document"]["collection"] = collection
-    return results
-
-
-async def _ensure_exclusive_collection(doc_id: int, collection: str) -> None:
-    """Remove existing collection tags and apply the new one."""
-    doc = engine.get_document(doc_id)
-    existing_collection_tags = [
-        t for t in doc.get("tags", [])
-        if t.startswith(COLLECTION_TAG_PREFIX)
-    ]
-    new_tag = _collection_tag(collection)
-    if existing_collection_tags == [new_tag]:
-        return
-    if existing_collection_tags:
-        engine.update_tags(doc_id, remove=existing_collection_tags)
-    engine.update_tags(doc_id, add=[new_tag])
-
+_transport_security = TransportSecuritySettings(
+    enable_dns_rebinding_protection=True,
+    allowed_hosts=_allowed_hosts,
+    allowed_origins=_allowed_origins,
+)

 # ---------------------------------------------------------------------------
 # FastMCP server
@@ -87,7 +43,21 @@ async def _ensure_exclusive_collection(doc_id: int, collection: str) -> None:

 mcp = FastMCP(
    "kb",
-    instructions="Knowledge base MCP server. Provides tools for searching, adding, and managing documents and notes.",
+    instructions=(
+        "Knowledge base MCP server with hybrid semantic + full-text search. "
+        "kb_search uses dense vector embeddings (semantic similarity) fused with "
+        "BM25 full-text ranking, so it finds conceptually related content even "
+        "when the exact words don't match — agents can ask natural-language "
+        "questions rather than guessing keywords. When the engine has a "
+        "cross-encoder reranker enabled, results are reranked server-side by "
+        "default (pass rerank=False for lower latency). Also provides tools for "
+        "adding notes, uploading files, and managing documents and tags. Use tags "
+        "to organise and filter documents (e.g. tag notes with 'agent:mybot' and "
+        "filter searches by that tag). This server requires Bearer token "
+        "authentication — all requests are authenticated via the Authorization "
+        "header at the HTTP transport layer."
+    ),
+    transport_security=_transport_security,
 )


@@ -97,50 +67,67 @@ async def kb_search(
    top: int = 10,
    tags: list[str] | None = None,
    doc_type: str | None = None,
-    collection: str | None = None,
    fts_only: bool = False,
+    explain: bool = False,
+    rerank: bool | None = None,
 ) -> str:
-    """Search the knowledge base for relevant documents and notes.
+    """Hybrid semantic (vector) + full-text search over the knowledge base.

-    Returns ranked chunks matching the query, with text content, relevance scores,
-    and document metadata.
+    Combines dense vector embeddings (semantic similarity — finds conceptually
+    related content even when the wording differs) with BM25 keyword ranking,
+    fused via reciprocal rank fusion. Because the search is semantic, you can
+    ask natural-language questions ("what did we decide about X?") rather than
+    guessing the exact keywords used in the source documents.
+
+    When the engine has a cross-encoder reranker enabled, the top candidates
+    are reranked server-side by default — you normally do NOT need to rerank
+    results yourself. Check kb_status's "rerank" block to see whether it is
+    active.
+
+    Returns ranked chunks matching the query, with text content, relevance
+    scores, and document metadata.

    Args:
-        query: The search query. Can be a natural language question or keywords.
+        query: The search query — a natural language question or keywords.
        top: Maximum number of results to return (default 10).
        tags: Filter results to documents with ALL of these tags.
-        doc_type: Filter by document type (e.g. "note", "pdf", "markdown", "code").
-        collection: Filter by collection name (e.g. "documents", "memory", "workspace").
-        fts_only: If true, use only full-text search (no vector similarity).
+        doc_type: Filter by document type (e.g. "note", "pdf", "markdown",
+            "code", "data").
+        fts_only: Disable the vector/semantic component and use only BM25
+            keyword matching. Default false (hybrid mode). Set true only when
+            you need exact-string matching (e.g. an error code, identifier).
+        explain: Include a per-result score breakdown (BM25 score/rank, vector
+            similarity/rank, rank-fusion contributions, rerank blend) under an
+            "explain" key. Useful for diagnosing why a result ranked where it did.
+        rerank: Set false to skip server-side reranking for lower latency.
+            Default (None) uses the engine's configured behaviour.

    Tips for complex queries:
    - Consider expanding into 2-3 variant phrasings and calling this tool multiple
      times, then deduplicating results by chunk_id. For example, search for both
      "pension revaluation rules" and "how are pensions revalued" to cast a wider net.
-    - For precision, rerank the returned results using your own judgement based on
-      relevance to the original question.
+    - If the engine's reranker is disabled, you can still rerank the returned
+      results yourself using your own judgement of relevance to the question.
+    - Call kb_status to see which embedding model is in use and whether
+      server-side reranking is active.
    """
-    search_tags = list(tags) if tags else []
-    if collection:
-        search_tags.append(_collection_tag(collection))
-
    result = engine.search(
        query=query,
        top=top,
-        tags=search_tags or None,
+        tags=tags or None,
        doc_type=doc_type,
        fts_only=fts_only,
+        explain=explain,
+        rerank=rerank,
    )

    results_list = result if isinstance(result, list) else result.get("results", [])
-    processed = _process_search_results(results_list)
-    return json.dumps(processed, indent=2)
+    return json.dumps(results_list, indent=2)


@mcp.tool()
 async def kb_addnote(
    text: str,
-    collection: str | None = None,
    tags: list[str] | None = None,
    title: str | None = None,
 ) -> str:
@@ -151,15 +138,10 @@ async def kb_addnote(

    Args:
        text: The note text content.
-        collection: Collection to add the note to (default "documents").
-            Standard collections: "documents", "memory", "workspace".
-        tags: Additional tags to apply to the note.
+        tags: Tags to apply to the note.
        title: Optional title (auto-derived from first line if omitted).
    """
-    all_tags = list(tags) if tags else []
-    all_tags.append(_collection_tag(collection))
-
-    result = engine.add_note(text=text, tags=all_tags, title=title)
+    result = engine.add_note(text=text, tags=tags or None, title=title)
    return json.dumps(result, indent=2)


@@ -179,7 +161,7 @@ async def kb_update_note(
        text: The new text content for the note.
    """
    result = engine.update_note(document_id, text)
-    return json.dumps(_process_document(result), indent=2)
+    return json.dumps(result, indent=2)


@mcp.tool()
@@ -198,14 +180,14 @@ async def kb_get(
    """
    if document_id is not None:
        result = engine.get_document(document_id)
-        return json.dumps(_process_document(result), indent=2)
+        return json.dumps(result, indent=2)
    elif source_path is not None:
        docs = engine.list_documents()
        matches = [d for d in docs if d.get("source_path") == source_path]
        if not matches:
            return json.dumps({"error": "No document found with that source_path"})
        doc = engine.get_document(matches[0]["id"])
-        return json.dumps(_process_document(doc), indent=2)
+        return json.dumps(doc, indent=2)
    else:
        return json.dumps({"error": "Provide either document_id or source_path"})

@@ -218,6 +200,7 @@ async def kb_status() -> str:
    database size, and ingestion queue state.
    """
    result = engine.get_status()
+    result["authenticated"] = bool(config.KB_MCP_API_KEY)
    return json.dumps(result, indent=2)


@@ -237,12 +220,27 @@ async def kb_jobs(
    return json.dumps(result, indent=2)


+@mcp.tool()
+async def kb_delete(
+    document_id: int,
+) -> str:
+    """Permanently delete a document from the knowledge base.
+
+    Removes the document and all associated data (chunks, embeddings, tags,
+    stored files). This action cannot be undone.
+
+    Args:
+        document_id: The ID of the document to delete.
+    """
+    result = engine.delete_document(document_id)
+    return json.dumps(result, indent=2)
+
+
@mcp.tool()
 async def kb_upload_start(
    filename: str,
    total_size: int,
    tags: list[str] | None = None,
-    collection: str | None = None,
 ) -> str:
    """Start a chunked file upload to the knowledge base.

@@ -252,7 +250,7 @@ async def kb_upload_start(
    3. Call kb_upload_finish to submit the file for ingestion

    Example for a 3MB file:
-        upload = kb_upload_start(filename="report.pdf", total_size=3145728, collection="documents")
+        upload = kb_upload_start(filename="report.pdf", total_size=3145728, tags=["project:x"])
        kb_upload_chunk(upload_id=upload["upload_id"], data="<base64 chunk 0>", chunk_index=0)
        kb_upload_chunk(upload_id=upload["upload_id"], data="<base64 chunk 1>", chunk_index=1)
        kb_upload_chunk(upload_id=upload["upload_id"], data="<base64 chunk 2>", chunk_index=2)
@@ -261,13 +259,9 @@ async def kb_upload_start(
    Args:
        filename: Original filename (used for type detection).
        total_size: Total file size in bytes.
-        tags: Additional tags to apply.
-        collection: Collection name (default "documents").
+        tags: Tags to apply to the uploaded document.
    """
-    all_tags = list(tags) if tags else []
-    all_tags.append(_collection_tag(collection))
-
-    upload_id = uploads.start_upload(filename, total_size, all_tags)
+    upload_id = uploads.start_upload(filename, total_size, tags or [])
    return json.dumps({"upload_id": upload_id})


@@ -313,6 +307,112 @@ async def kb_upload_finish(
        return json.dumps({"error": str(e)})


+# ---------------------------------------------------------------------------
+# Bulk operation tools
+# ---------------------------------------------------------------------------
+
+
+@mcp.tool()
+async def kb_bulk_delete(
+    document_ids: list[int] | None = None,
+    tags: list[str] | None = None,
+    doc_type: str | None = None,
+    from_id: int | None = None,
+    to_id: int | None = None,
+    force: bool = False,
+) -> str:
+    """Permanently delete multiple documents matching a filter.
+
+    Removes matched documents and all associated data (chunks, embeddings, tags,
+    stored files). This action cannot be undone.
+
+    Selection filters combine with AND logic — at least one is required.
+
+    A safety threshold applies: if the operation would affect more than 70% of
+    all documents, it is rejected unless force=true.
+
+    Args:
+        document_ids: Delete documents with these specific IDs.
+        tags: Delete documents that have ALL of these tags (selection filter).
+        doc_type: Delete documents of this type (e.g. "note", "pdf").
+        from_id: Delete documents with id >= this value.
+        to_id: Delete documents with id <= this value.
+        force: Override the safety threshold if it would block the operation.
+    """
+    result = engine.bulk_delete(
+        document_ids=document_ids, tags=tags, doc_type=doc_type,
+        from_id=from_id, to_id=to_id, force=force,
+    )
+    return json.dumps(result, indent=2)
+
+
+@mcp.tool()
+async def kb_bulk_tags(
+    document_ids: list[int] | None = None,
+    tags: list[str] | None = None,
+    doc_type: str | None = None,
+    from_id: int | None = None,
+    to_id: int | None = None,
+    add: list[str] | None = None,
+    remove: list[str] | None = None,
+    force: bool = False,
+) -> str:
+    """Add and/or remove tags on multiple documents matching a filter.
+
+    Selection filters combine with AND logic — at least one is required.
+    Note: the 'tags' parameter is a SELECTION FILTER (which documents to target),
+    while 'add' and 'remove' specify the TAG CHANGES to apply to those documents.
+
+    Args:
+        document_ids: Target documents with these specific IDs.
+        tags: Target documents that have ALL of these tags (selection filter).
+        doc_type: Target documents of this type.
+        from_id: Target documents with id >= this value.
+        to_id: Target documents with id <= this value.
+        add: Tags to add to matched documents.
+        remove: Tags to remove from matched documents.
+        force: Override the safety threshold if it would block the operation.
+    """
+    result = engine.bulk_tags(
+        document_ids=document_ids, tags=tags, doc_type=doc_type,
+        from_id=from_id, to_id=to_id, add=add, remove=remove, force=force,
+    )
+    return json.dumps(result, indent=2)
+
+
+@mcp.tool()
+async def kb_bulk_set_tags(
+    document_ids: list[int] | None = None,
+    tags: list[str] | None = None,
+    doc_type: str | None = None,
+    from_id: int | None = None,
+    to_id: int | None = None,
+    new_tags: list[str] | None = None,
+    force: bool = False,
+) -> str:
+    """Replace all tags on multiple documents with a new set.
+
+    Removes ALL existing tags from matched documents, then applies the new tag set.
+    Selection filters combine with AND logic — at least one is required.
+    Note: the 'tags' parameter is a SELECTION FILTER (which documents to target),
+    while 'new_tags' is the REPLACEMENT tag set to apply.
+
+    Args:
+        document_ids: Target documents with these specific IDs.
+        tags: Target documents that have ALL of these tags (selection filter).
+        doc_type: Target documents of this type.
+        from_id: Target documents with id >= this value.
+        to_id: Target documents with id <= this value.
+        new_tags: The replacement tag set to apply to all matched documents.
+        force: Override the safety threshold if it would block the operation.
+    """
+    result = engine.bulk_set_tags(
+        document_ids=document_ids, tags=tags, doc_type=doc_type,
+        from_id=from_id, to_id=to_id, new_tags=new_tags, force=force,
+    )
+    return json.dumps(result, indent=2)
+
+
 # ---------------------------------------------------------------------------
 # Auth middleware
 # ---------------------------------------------------------------------------
@@ -323,9 +423,7 @@ class BearerAuthMiddleware(BaseHTTPMiddleware):
            return await call_next(request)

        auth_header = request.headers.get("authorization", "")
-        if auth_header.startswith("Bearer "):
-            token = auth_header[7:]
-            if token == config.KB_MCP_API_KEY:
+        if auth_header.startswith("Bearer ") and auth_header[7:] == config.KB_MCP_API_KEY:
            return await call_next(request)

        return JSONResponse(
@@ -0,0 +1,58 @@
+# kb — Next Steps
+
+UX improvements to make documents easier to find and inspect, prompted by a session where searching for an uploaded PDF (`M38T_PHEV_RHD_OM_EN_UK_20251209.pdf`, doc id 2077, 1801 chunks) surfaced lots of chunk hits but no obvious path back to the original document.
+
+## Problems observed
+
+### 1. `kb list` silently ignores positional arguments
+
+```
+kb list --type pdf "M38T_PHEV_RHD_OM_EN_UK_20251209"
+```
+
+The quoted term is dropped without warning; user gets the default newest-first listing and assumes the document is missing. `kb list` currently only supports `--tags` and `--type` filters.
+
+### 2. `kb search` returns chunks with no `document_id`
+
+Result objects expose `chunk_id`, `title`, `source_path`, `tags` — but not `document_id`. To get from a search hit back to the owning document you have to title-match against `kb list` output or call an undocumented endpoint. The skill docs even claim a `source.document_id` field that isn't actually present in the CLI output.
+
+### 3. `kb info` dumps every chunk with no summary mode
+
+`kb info 2077` returns ~1801 chunk objects. The document-level metadata (`id`, `title`, `original_filename`, `source_path`, `stored_path`, `doc_type`, `language`, `content_hash`, `has_file`, `tags`, `created_at`, `updated_at`) **is** present at the top level of the JSON, but in practice it's invisible — human format presumably dumps the chunk list and the user sees only chunks.
+
+There's no way to ask for "just tell me about this document."
+
+### 4. Search hits can look like noise on image-heavy PDFs
+
+Top chunks for the M38T search were single characters (`"1"`, `"B"`, `"\""`). Almost certainly an FTS artefact on short tokens from a scan/image-heavy PDF — but it makes the result set look broken. Worth considering a minimum-text-length filter on indexed chunks, or down-weighting very short chunks in ranking.
+
+## Proposed changes
+
+### Small / high-value
+
+- **`kb info --no-chunks`** (or make `--chunks` opt-in): default to metadata + chunk count, only include chunks when asked. Human format should always lead with the metadata block.
+- **`kb list --title <substring>`** (or accept a positional query) for filename / title search. At minimum, error or warn when positional args are passed and ignored.
+- **Include `document_id` in `kb search` result objects.** Either at the top of each result or under `source.document_id` (matching the skill docs).
+
+### Medium
+
+- **`kb find <query>`** as a doc-level search that aggregates chunk hits per document and returns ranked *documents* (with hit count, top chunk preview). This is what users usually want when they say "find my PDF about X."
+- **Update the `kb` skill docs** to match actual CLI output shape, and to steer users toward `kb list | jq` for filename lookups until proper filtering lands.
+
+### Larger
+
+- **Quality filter for short chunks** during ingestion (e.g. drop chunks with < N alphanumeric chars, or fold them into neighbours). Stops scanned/image-heavy PDFs from polluting search.
+- **OCR path for scan-heavy PDFs.** The M38T manual extracted enough real text to be useful, but other "scan" docs likely don't. Detect low text density per page and route through OCR.
+
+## Quick reference (current workarounds)
+
+```bash
+# Find a doc by filename
+kb list --type pdf --format json | jq '.[] | select(.title | contains("M38T"))'
+
+# Get just metadata for a doc
+kb info 2077 --format json | jq 'del(.chunks)'
+
+# Download the original
+kb export 2077 -o manual.pdf
+```
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-04-04
@@ -0,0 +1,194 @@
+## Context
+
+The engine API (`engine/kb/routes/`) provides single-document operations for delete (`DELETE /api/v1/documents/{id}`) and tag management (`PUT /api/v1/documents/{id}/tags`). The MCP server (`mcp/server.py`) wraps these and adds a "collection" abstraction via `collection:`-prefixed tags — ~70 lines of helpers and translation logic that only the MCP layer understands.
+
+The database is SQLite with WAL mode, FTS5 for full-text search, and sqlite-vec for embeddings. Foreign keys with `ON DELETE CASCADE` handle chunk cleanup when documents are deleted. Stored files on disk must be cleaned up separately.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Bulk delete, bulk tag add/remove, and bulk set-tags (replace) via engine API, MCP tools, and CLI
+- Filter-based selection: by tag, doc_type, ID list, and ID range
+- Safety threshold to prevent accidental mass operations
+- Audit trail via jobs table
+- Remove collection abstraction from MCP server
+
+**Non-Goals:**
+- Async/queued bulk operations (SQLite handles thousands of rows synchronously in <1s)
+- Bulk document retrieval or bulk note creation
+- Undo/recycle bin for bulk deletes
+- Adding collection concept to engine or CLI (collections are being removed, not moved)
+
+## Decisions
+
+### 1. Common selection filter for all bulk endpoints
+
+All three bulk endpoints accept the same selection body:
+
+```json
+{
+  "document_ids": [1, 5, 12],
+  "tags": ["agent:mybot", "draft"],
+  "doc_type": "note",
+  "from_id": 10,
+  "to_id": 50
+}
+```
+
+Filters combine with AND logic. At least one filter is required — the engine rejects requests with no selection criteria (400).
+
+**Selection SQL generation**: A shared helper in `database.py` builds the WHERE clause from the filter. The `tags` filter uses the same JOIN pattern as `list_documents` (all specified tags must match). The `document_ids` filter uses `IN (?)`. The `from_id`/`to_id` filter uses `id >= ? AND id <= ?`.
+
+**Alternative considered**: Separate endpoints per filter type. Rejected — combinable filters are more powerful and the SQL generation is straightforward.
+
+### 2. Safety threshold with configurable percentage
+
+Before executing, the engine counts matched documents and total documents. If `matched / total > threshold`, the request is rejected:
+
+```
+HTTP 409 Conflict
+{
+  "error": "safety_threshold_exceeded",
+  "message": "Operation would affect 750 of 1000 documents (75.0%). Exceeds safety threshold of 70%. Use force: true to proceed.",
+  "matched": 750,
+  "total": 1000,
+  "percent": 75.0,
+  "threshold": 70
+}
+```
+
+- Default threshold: 70% (env var `KB_BULK_SAFETY_PERCENT`, integer 0-100)
+- Override per-request: `"force": true` in the request body
+- Threshold of 0 effectively disables the safety check
+- CLI maps this to `--force` / `-f` flag
+
+The check is a SELECT COUNT before the operation — minimal overhead.
+
+**Alternative considered**: Dry-run mode (preview what would be affected, then confirm). Rejected — adds a two-step flow that doesn't help LLM callers (they'd just always confirm) and the safety threshold covers the dangerous case.
+
+### 3. Synchronous execution with audit logging
+
+Bulk operations execute synchronously and return a summary response:
+
+```json
+{
+  "job_id": 42,
+  "status": "done",
+  "matched": 750,
+  "succeeded": 748,
+  "failed": 2,
+  "errors": [
+    {"document_id": 42, "error": "file locked"},
+    {"document_id": 99, "error": "not found"}
+  ]
+}
+```
+
+A job record is created in the `jobs` table with a new `bulk_delete` / `bulk_tags` / `bulk_set_tags` status type. This requires extending the jobs table:
+
+- Add `job_type` column: `"ingest"` (default, for existing jobs) or `"bulk_delete"` / `"bulk_tags"` / `"bulk_set_tags"`
+- The job's `filename` field stores a JSON summary of the selection filter for auditability
+- `document_id` field stores the count of affected documents
+- `error` field stores JSON array of individual errors if any
+
+**Alternative considered**: Full async with job polling. Rejected — SQLite bulk operations are fast enough synchronously and async would require extra polling calls (defeating the purpose of reducing token usage).
+
+### 4. Bulk delete implementation
+
+For each matched document:
+1. Collect chunk IDs
+2. Delete embeddings from `chunks_vec`
+3. Delete the document row (cascades to chunks, document_tags)
+4. Delete stored file from disk
+
+This follows the same logic as the existing `delete_document` endpoint but batched in a single transaction (except file deletion, which happens after commit). If a file deletion fails, the document is still counted as succeeded (the DB record is gone) but a warning is logged.
+
+The operation processes documents within a single SQLite transaction for atomicity of the DB changes. File deletions happen post-commit and are best-effort.
+
+### 5. Bulk tags implementation
+
+Two distinct operations:
+
+**`POST /api/v1/bulk/tags`** — Add and/or remove tags:
+```json
+{
+  "add": ["reviewed", "approved"],
+  "remove": ["draft"],
+  ...selection filters...
+}
+```
+
+**`POST /api/v1/bulk/set-tags`** — Replace all tags:
+```json
+{
+  "tags": ["final", "approved"],
+  ...selection filters...
+}
+```
+
+The `set-tags` operation removes all existing tags from matched documents, then applies the new set. This is useful for cleaning up tag clutter or migrating tagging schemes.
+
+Both update `updated_at` on affected documents.
+
+### 6. Remove collection abstraction from MCP
+
+Remove from `mcp/server.py`:
+- Constants: `COLLECTION_TAG_PREFIX`, `DEFAULT_COLLECTION`
+- Functions: `_collection_tag`, `_strip_collection_tags`, `_process_document`, `_process_search_results`, `_ensure_exclusive_collection`
+- Tool: `kb_set_collection` (entire tool removed)
+- Parameters: `collection` from `kb_search`, `kb_addnote`, `kb_upload_start`
+
+The `_process_document` and `_process_search_results` calls in remaining tools are removed — documents are returned as-is from the engine, with all tags visible.
+
+Users/agents that need namespace isolation use a tag convention (e.g. `agent:claude-code`) communicated via system prompt or tool instructions.
+
+### 7. Engine bulk route module
+
+New file: `engine/kb/routes/bulk.py`
+
+Three endpoints sharing common infrastructure:
+- `_resolve_selection(conn, filters)` → list of document IDs + count
+- `_check_safety_threshold(matched, total, force)` → raises HTTPException if exceeded
+- `_log_bulk_job(conn, job_type, filters, matched, succeeded, failed, errors)` → job_id
+
+### 8. MCP bulk tools
+
+Three new tools in `mcp/server.py`, thin wrappers calling new `engine.py` methods:
+
+- `kb_bulk_delete(document_ids?, tags?, doc_type?, from_id?, to_id?, force?)` → str (JSON)
+- `kb_bulk_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, add?, remove?, force?)` → str (JSON)
+- `kb_bulk_set_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, new_tags?, force?)` → str (JSON)
+
+Note: The `tags` parameter on bulk tools serves as a **selection filter** (which documents to target), while `add`/`remove` (on bulk_tags) and `new_tags` (on bulk_set_tags) are the **operation** (what to do to the tags). Tool descriptions must make this distinction clear.
+
+### 9. CLI bulk commands
+
+Three new commands under `client/cmd/`:
+
+```
+kb bulk-remove --tags "draft,old" --type note --force --yes
+kb bulk-tag --tags "agent:mybot" --add "reviewed" --remove "pending" --yes
+kb bulk-set-tags --ids "1,5,12" --tags "clean,final" --yes
+```
+
+Filter flags (shared): `--tags`, `--type`, `--ids` (comma-separated), `--from-id`, `--to-id`, `--force`
+Confirmation: `--yes` / `-y` to skip interactive prompt.
+
+Without `--yes`, the CLI first shows the match count and asks for confirmation:
+
+```
+This will delete 47 documents matching: tags=[draft,old] type=note
+Proceed? [y/N]
+```
+
+### 10. Engine config for safety threshold
+
+New env var: `KB_BULK_SAFETY_PERCENT` (integer, default 70). Added to `engine/kb/config.py`.
+
+## Risks / Trade-offs
+
+- **[Bulk delete is irreversible]** → Safety threshold mitigates accidental mass deletion. CLI requires interactive confirmation. No undo mechanism — this is deliberate to keep the system simple.
+- **[Naming collision: `tags` as filter vs operation]** → The `tags` parameter in bulk_tags selects documents, while `add`/`remove` specifies the tag changes. Clear naming and tool descriptions mitigate confusion. Engine request model uses the same field name as the existing list/search filter.
+- **[SQLite lock during large bulk ops]** → A single transaction deleting 5000 documents will hold a write lock. With WAL mode, readers are not blocked. The lock duration should be under a few seconds for typical workloads.
+- **[Breaking change: collection removal]** → Any MCP client relying on `collection` parameters will break. Since collections were only recently added and are not widely deployed, this is acceptable. Existing `collection:*` tags in the database remain as regular tags — they still work as filters, just without special treatment.
+- **[Jobs table overload]** → Bulk operations add a new job type to a table designed for ingestion jobs. The schema change is minimal (one new column) and the audit trail value outweighs the mixing of concerns.
@@ -0,0 +1,91 @@
+## Why
+
+Bulk operations on documents (delete, tag, retag) currently require one API/MCP call per document. When an LLM manages hundreds or thousands of documents, this means hundreds of tool calls — burning tokens, adding latency, and creating fragile multi-step flows that can fail partway through.
+
+Additionally, the "collection" abstraction in the MCP server adds complexity without real benefit. Collections are implemented as `collection:`-prefixed tags, but this convention is only enforced in the MCP layer — the CLI and engine don't know about it. This creates inconsistency and extra code. Tags alone, with a naming convention communicated via system prompt or configuration, achieve the same namespace isolation more simply and uniformly.
+
+## What Changes
+
+### 1. Remove collections from MCP server
+
+Strip all collection logic from `mcp/server.py`:
+- Remove `COLLECTION_TAG_PREFIX`, `DEFAULT_COLLECTION`, and all collection helper functions
+- Remove `collection` parameter from `kb_search`, `kb_addnote`, `kb_upload_start`
+- Remove `kb_set_collection` tool entirely
+- Remove `_process_document` / `_process_search_results` collection-tag stripping
+- Update MCP server instructions to explain tag-based namespace convention
+
+### 2. Add bulk engine endpoints
+
+Three new endpoints in the engine API:
+
+- **POST /api/v1/bulk/delete** — Delete multiple documents matching a filter
+- **POST /api/v1/bulk/tags** — Add/remove tags on multiple documents matching a filter
+- **POST /api/v1/bulk/set-tags** — Replace all tags on multiple documents matching a filter
+
+All accept a common **selection filter** (combinable with AND logic):
+- `document_ids` — explicit list of IDs
+- `tags` — documents matching ALL specified tags
+- `doc_type` — documents of this type
+- `from_id` / `to_id` — ID range (inclusive)
+
+At least one selection criterion is required.
+
+**Safety threshold**: If the operation would affect more than N% of all documents (default 70%, configurable via `KB_BULK_SAFETY_PERCENT` env var), the request is rejected with a 409 response showing what would be affected. The caller must re-send with `force: true` to proceed.
+
+**Response model**: Synchronous execution with summary response. The operation is logged to the jobs table for audit trail:
+
+```json
+{
+  "job_id": 42,
+  "status": "done",
+  "matched": 750,
+  "succeeded": 748,
+  "failed": 2,
+  "errors": [
+    {"document_id": 42, "error": "file locked"},
+    {"document_id": 99, "error": "not found"}
+  ]
+}
+```
+
+### 3. Add bulk MCP tools
+
+Expose the bulk engine endpoints as MCP tools:
+- `kb_bulk_delete` — bulk delete with filter selection
+- `kb_bulk_tags` — bulk add/remove tags with filter selection
+- `kb_bulk_set_tags` — bulk replace tags with filter selection
+
+These are thin wrappers around the engine bulk endpoints — no collection translation, no special logic.
+
+### 4. Add bulk CLI commands
+
+- `kb bulk-remove` — bulk delete with `--tags`, `--type`, `--ids`, `--from-id`, `--to-id`, `--force` flags
+- `kb bulk-tag` — bulk tag/untag with `--add`, `--remove`, and the same filter flags
+- `kb bulk-set-tags` — bulk replace tags with `--tags` (new tags) and the same filter flags
+
+All show a confirmation prompt with match count before executing (unless `--yes`).
+
+## Capabilities
+
+### New Capabilities
+
+- `bulk-operations`: Engine endpoints, MCP tools, and CLI commands for bulk delete, tag, and set-tags operations with filter-based selection and safety threshold.
+
+### Modified Capabilities
+
+- `mcp-document-management`: Remove `kb_set_collection` tool. Remove `collection` parameter from all tools.
+
+### Removed Capabilities
+
+- `mcp-collections`: The collection abstraction (collection helpers, collection parameters, collection tag stripping) is removed from the MCP server entirely.
+
+## Impact
+
+- **Engine API** (`engine/kb/routes/`): New `bulk.py` route module with 3 endpoints. New `bulk` job type in jobs table.
+- **Engine database** (`engine/kb/database.py`): Helper functions for bulk selection queries and bulk delete/tag operations.
+- **MCP server** (`mcp/server.py`): Remove ~70 lines of collection logic. Add 3 bulk tool definitions. Remove `collection` param from `kb_search`, `kb_addnote`, `kb_upload_start`. Remove `kb_set_collection`.
+- **MCP engine client** (`mcp/engine.py`): Add bulk operation methods. Remove no longer needed code.
+- **CLI** (`client/cmd/`): New `bulk_remove.go`, `bulk_tag.go`, `bulk_set_tags.go` command files.
+- **CLI API client** (`client/internal/api/`): Add `Post` with JSON body support if not present.
+- **Breaking changes**: `kb_set_collection` MCP tool removed. `collection` parameter removed from `kb_search`, `kb_addnote`, `kb_upload_start` MCP tools. Any MCP clients using collections will need to switch to tags.
@@ -0,0 +1,230 @@
+## ADDED Requirements
+
+### Requirement: Common selection filter
+
+All bulk engine endpoints SHALL accept a JSON body with the following optional selection fields, combined with AND logic:
+
+- `document_ids` (list of int) — match documents with these specific IDs
+- `tags` (list of str) — match documents that have ALL specified tags
+- `doc_type` (str) — match documents with this document type
+- `from_id` (int) — match documents with id >= this value
+- `to_id` (int) — match documents with id <= this value
+
+At least one selection field MUST be present. If no selection fields are provided, the endpoint SHALL return 400 Bad Request.
+
+#### Scenario: Filter by tags and doc_type
+
+- **WHEN** a bulk endpoint receives `{"tags": ["draft"], "doc_type": "note"}`
+- **THEN** it SHALL match only documents that have the tag "draft" AND have doc_type "note"
+
+#### Scenario: Filter by ID range
+
+- **WHEN** a bulk endpoint receives `{"from_id": 10, "to_id": 50}`
+- **THEN** it SHALL match documents with id >= 10 AND id <= 50
+
+#### Scenario: Filter by explicit IDs
+
+- **WHEN** a bulk endpoint receives `{"document_ids": [1, 5, 12]}`
+- **THEN** it SHALL match only documents with those specific IDs
+
+#### Scenario: Combined filters
+
+- **WHEN** a bulk endpoint receives `{"tags": ["agent:mybot"], "doc_type": "note", "from_id": 100}`
+- **THEN** it SHALL match documents satisfying ALL three criteria
+
+#### Scenario: No selection fields provided
+
+- **WHEN** a bulk endpoint receives `{}` or `{"force": true}` with no selection fields
+- **THEN** it SHALL return 400 Bad Request
+
+### Requirement: Safety threshold
+
+All bulk endpoints SHALL enforce a safety threshold. Before executing, the engine SHALL count the matched documents and the total documents in the database. If `matched / total * 100` exceeds the configured threshold, the request SHALL be rejected with 409 Conflict.
+
+The response SHALL include: `error` ("safety_threshold_exceeded"), `message` (human-readable), `matched` (int), `total` (int), `percent` (float), and `threshold` (int).
+
+The threshold SHALL default to 70 and be configurable via the `KB_BULK_SAFETY_PERCENT` environment variable (integer 0-100). A value of 0 disables the check.
+
+The caller MAY override the threshold by including `"force": true` in the request body.
+
+#### Scenario: Threshold exceeded
+
+- **GIVEN** 1000 total documents and `KB_BULK_SAFETY_PERCENT` is 70
+- **WHEN** a bulk endpoint matches 750 documents (75%) without `force: true`
+- **THEN** it SHALL return 409 with `matched: 750`, `total: 1000`, `percent: 75.0`, `threshold: 70`
+
+#### Scenario: Threshold not exceeded
+
+- **GIVEN** 1000 total documents and `KB_BULK_SAFETY_PERCENT` is 70
+- **WHEN** a bulk endpoint matches 500 documents (50%) without `force: true`
+- **THEN** the operation SHALL proceed normally
+
+#### Scenario: Force override
+
+- **GIVEN** 1000 total documents and a match of 900 (90%)
+- **WHEN** the request includes `"force": true`
+- **THEN** the operation SHALL proceed regardless of threshold
+
+#### Scenario: Zero threshold
+
+- **GIVEN** `KB_BULK_SAFETY_PERCENT` is 0
+- **THEN** the safety check SHALL be effectively disabled for all operations
+
+### Requirement: Synchronous response with audit log
+
+All bulk endpoints SHALL execute synchronously and return a JSON response with:
+
+- `job_id` (int) — ID of the audit log entry in the jobs table
+- `status` (str) — "done" or "partial_failure"
+- `matched` (int) — number of documents that matched the selection
+- `succeeded` (int) — number of documents successfully processed
+- `failed` (int) — number of documents that failed
+- `errors` (list) — array of `{"document_id": int, "error": str}` for each failure (empty on full success)
+
+A job record SHALL be created in the jobs table with `job_type` set to the operation type. The `filename` field SHALL store a JSON representation of the selection filter. The `error` field SHALL store a JSON array of individual errors if any occurred.
+
+#### Scenario: Full success
+
+- **WHEN** a bulk operation matches 50 documents and all succeed
+- **THEN** the response SHALL have `status: "done"`, `matched: 50`, `succeeded: 50`, `failed: 0`, `errors: []`
+
+#### Scenario: Partial failure
+
+- **WHEN** a bulk operation matches 50 documents but 2 fail
+- **THEN** the response SHALL have `status: "partial_failure"`, `matched: 50`, `succeeded: 48`, `failed: 2`, and `errors` listing the 2 failures
+
+### Requirement: Bulk delete endpoint
+
+The engine SHALL expose `POST /api/v1/bulk/delete` which permanently deletes all documents matching the selection filter. For each matched document, it SHALL delete embeddings from `chunks_vec`, delete the document row (cascading to chunks and document_tags), and delete any stored file from disk.
+
+Database deletions SHALL be performed within a single transaction. File deletions SHALL occur after the transaction commits and SHALL be best-effort (failures logged but not counted as document failures).
+
+#### Scenario: Bulk delete by tag
+
+- **WHEN** `POST /api/v1/bulk/delete` receives `{"tags": ["old", "draft"]}`
+- **THEN** all documents with both tags "old" and "draft" SHALL be deleted
+- **AND** their chunks, embeddings, tag associations, and stored files SHALL be removed
+
+#### Scenario: Bulk delete with no matches
+
+- **WHEN** `POST /api/v1/bulk/delete` receives a filter that matches 0 documents
+- **THEN** the response SHALL have `matched: 0`, `succeeded: 0`, `failed: 0`
+
+### Requirement: Bulk tags endpoint
+
+The engine SHALL expose `POST /api/v1/bulk/tags` which adds and/or removes tags on all documents matching the selection filter. The request body SHALL include the selection filter plus:
+
+- `add` (list of str, optional) — tags to add
+- `remove` (list of str, optional) — tags to remove
+
+At least one of `add` or `remove` MUST be present. The endpoint SHALL return 400 if neither is provided.
+
+The endpoint SHALL update `updated_at` on all affected documents.
+
+#### Scenario: Add and remove tags in one call
+
+- **WHEN** `POST /api/v1/bulk/tags` receives `{"tags": ["agent:mybot"], "add": ["reviewed"], "remove": ["pending"]}`
+- **THEN** all documents tagged "agent:mybot" SHALL have "reviewed" added and "pending" removed
+
+### Requirement: Bulk set-tags endpoint
+
+The engine SHALL expose `POST /api/v1/bulk/set-tags` which replaces all tags on matched documents with a new set. The request body SHALL include the selection filter plus:
+
+- `new_tags` (list of str) — the replacement tag set
+
+The endpoint SHALL remove all existing tag associations from matched documents, then apply the new set. It SHALL update `updated_at` on all affected documents.
+
+#### Scenario: Replace all tags
+
+- **WHEN** `POST /api/v1/bulk/set-tags` receives `{"doc_type": "note", "new_tags": ["clean", "final"]}`
+- **THEN** all notes SHALL have their existing tags removed and replaced with "clean" and "final"
+
+### Requirement: Jobs table extension
+
+The jobs table SHALL be extended with a `job_type` column (TEXT, default "ingest") to distinguish ingestion jobs from bulk operation audit entries. Valid values: "ingest", "bulk_delete", "bulk_tags", "bulk_set_tags".
+
+Existing jobs SHALL default to `job_type = "ingest"`. The existing jobs list endpoint and CLI `kb jobs` command SHALL continue to work unchanged.
+
+#### Scenario: Migration adds column
+
+- **GIVEN** an existing database without the `job_type` column
+- **WHEN** the engine starts
+- **THEN** the column SHALL be added with default value "ingest"
+
+### Requirement: Engine config for safety threshold
+
+The engine `Config` class SHALL read `KB_BULK_SAFETY_PERCENT` from the environment as an integer (default 70, range 0-100). This value SHALL be used as the default safety threshold for all bulk endpoints.
+
+### Requirement: MCP bulk delete tool
+
+The MCP server SHALL expose a `kb_bulk_delete` tool with parameters: `document_ids` (optional list of int), `tags` (optional list of str), `doc_type` (optional str), `from_id` (optional int), `to_id` (optional int), `force` (optional bool).
+
+The tool SHALL call `POST /api/v1/bulk/delete` on the engine via the engine client and return the JSON response.
+
+The tool description SHALL clearly state that `tags` is a selection filter (which documents to delete), not tags to delete.
+
+#### Scenario: MCP bulk delete by tag
+
+- **WHEN** `kb_bulk_delete(tags=["old"])` is called
+- **THEN** the engine client SHALL send `POST /api/v1/bulk/delete` with `{"tags": ["old"]}`
+- **AND** the tool SHALL return the engine's JSON response
+
+### Requirement: MCP bulk tags tool
+
+The MCP server SHALL expose a `kb_bulk_tags` tool with parameters: `document_ids`, `tags`, `doc_type`, `from_id`, `to_id` (selection filters), plus `add` (optional list of str), `remove` (optional list of str), and `force` (optional bool).
+
+The tool description SHALL clearly distinguish `tags` (selection filter) from `add`/`remove` (tag changes to apply).
+
+#### Scenario: MCP bulk tag update
+
+- **WHEN** `kb_bulk_tags(tags=["agent:mybot"], add=["reviewed"], remove=["draft"])` is called
+- **THEN** the engine client SHALL send the appropriate `POST /api/v1/bulk/tags` request
+
+### Requirement: MCP bulk set-tags tool
+
+The MCP server SHALL expose a `kb_bulk_set_tags` tool with parameters: `document_ids`, `tags`, `doc_type`, `from_id`, `to_id` (selection filters), plus `new_tags` (list of str) and `force` (optional bool).
+
+#### Scenario: MCP bulk set tags
+
+- **WHEN** `kb_bulk_set_tags(doc_type="note", new_tags=["clean"])` is called
+- **THEN** the engine client SHALL send `POST /api/v1/bulk/set-tags` with `{"doc_type": "note", "new_tags": ["clean"]}`
+
+### Requirement: MCP engine client bulk methods
+
+The MCP engine client (`mcp/engine.py`) SHALL provide three new methods:
+
+- `bulk_delete(document_ids?, tags?, doc_type?, from_id?, to_id?, force?)` → dict
+- `bulk_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, add?, remove?, force?)` → dict
+- `bulk_set_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, new_tags?, force?)` → dict
+
+Each SHALL send a POST request to the corresponding `/api/v1/bulk/*` endpoint with the parameters as a JSON body. Each SHALL raise on non-2xx status codes, consistent with existing methods.
+
+### Requirement: CLI bulk-remove command
+
+The CLI SHALL expose a `kb bulk-remove` command with flags: `--tags` (comma-separated), `--type`, `--ids` (comma-separated), `--from-id`, `--to-id`, `--force`/`-f`, `--yes`/`-y`.
+
+Without `--yes`, the CLI SHALL first display the match count and ask for interactive confirmation before proceeding.
+
+The command SHALL call `POST /api/v1/bulk/delete` with the constructed filter.
+
+#### Scenario: CLI bulk remove with confirmation
+
+- **WHEN** `kb bulk-remove --tags "draft,old" --type note` is run without `--yes`
+- **THEN** the CLI SHALL display "This will delete N documents matching: tags=[draft,old] type=note" and prompt "Proceed? [y/N]"
+
+#### Scenario: CLI bulk remove with --yes
+
+- **WHEN** `kb bulk-remove --tags "draft" --yes` is run
+- **THEN** the CLI SHALL proceed without prompting
+
+### Requirement: CLI bulk-tag command
+
+The CLI SHALL expose a `kb bulk-tag` command with the same filter flags as `bulk-remove`, plus `--add` and `--remove` (comma-separated tag lists).
+
+The command SHALL call `POST /api/v1/bulk/tags` with the constructed filter and tag changes.
+
+### Requirement: CLI bulk-set-tags command
+
+The CLI SHALL expose a `kb bulk-set-tags` command with the filter flags, plus `--set` (comma-separated list of replacement tags).
+
+The command SHALL call `POST /api/v1/bulk/set-tags` with the constructed filter and `new_tags`.
@@ -0,0 +1,55 @@
+## REMOVED Requirements
+
+### Requirement: Collection abstraction in MCP server
+
+The MCP server SHALL NOT maintain any collection abstraction. The following SHALL be removed:
+
+- Constants: `COLLECTION_TAG_PREFIX`, `DEFAULT_COLLECTION`
+- Functions: `_collection_tag`, `_strip_collection_tags`, `_process_document`, `_process_search_results`, `_ensure_exclusive_collection`
+- Tool: `kb_set_collection` (entire tool)
+- Parameters: `collection` from `kb_search`, `kb_addnote`, `kb_upload_start`
+
+Documents SHALL be returned as-is from the engine with all tags visible. No tag stripping or collection field injection SHALL occur.
+
+#### Scenario: Search results show all tags
+
+- **WHEN** `kb_search` is called and a result has tags `["agent:mybot", "collection:documents", "draft"]`
+- **THEN** all three tags SHALL be returned as-is — no stripping of `collection:*` tags
+
+#### Scenario: kb_set_collection no longer exists
+
+- **WHEN** an MCP client attempts to call `kb_set_collection`
+- **THEN** the tool SHALL not be found (removed)
+
+## MODIFIED Requirements
+
+### Requirement: kb_search without collection parameter
+
+The `kb_search` MCP tool SHALL accept `tags` (optional list of str) for filtering but SHALL NOT accept a `collection` parameter. Callers that previously used `collection="memory"` SHALL instead use `tags=["collection:memory"]` or whatever tag convention they prefer.
+
+#### Scenario: Filter by tag instead of collection
+
+- **WHEN** `kb_search(query="test", tags=["agent:mybot"])` is called
+- **THEN** results SHALL be filtered to documents tagged "agent:mybot"
+- **AND** no collection field SHALL be present in the response
+
+### Requirement: kb_addnote without collection parameter
+
+The `kb_addnote` MCP tool SHALL accept `tags` (optional list of str) but SHALL NOT accept a `collection` parameter. The tool SHALL NOT automatically apply any default collection tag — only explicitly provided tags are applied.
+
+#### Scenario: Add note with explicit tags
+
+- **WHEN** `kb_addnote(text="hello", tags=["agent:mybot", "memory"])` is called
+- **THEN** the note SHALL be created with exactly those two tags — no `collection:documents` tag added
+
+### Requirement: kb_upload_start without collection parameter
+
+The `kb_upload_start` MCP tool SHALL accept `tags` (optional list of str) but SHALL NOT accept a `collection` parameter. The tool SHALL NOT automatically apply any default collection tag.
+
+### Requirement: kb_update_note without collection processing
+
+The `kb_update_note` MCP tool SHALL return the document as-is from the engine without passing it through `_process_document`. All tags SHALL be visible in the response.
+
+### Requirement: kb_get without collection processing
+
+The `kb_get` MCP tool SHALL return documents as-is from the engine without passing through `_process_document`. All tags SHALL be visible in the response. No `collection` field SHALL be injected.
@@ -0,0 +1,45 @@
+## 1. Remove collections from MCP server
+
+- [x] 1.1 Remove collection constants and helper functions from `mcp/server.py` (`COLLECTION_TAG_PREFIX`, `DEFAULT_COLLECTION`, `_collection_tag`, `_strip_collection_tags`, `_process_document`, `_process_search_results`, `_ensure_exclusive_collection`)
+- [x] 1.2 Remove `collection` parameter from `kb_search`, `kb_addnote`, `kb_upload_start` tools
+- [x] 1.3 Remove `kb_set_collection` tool entirely
+- [x] 1.4 Remove `_process_document` / `_process_search_results` calls from `kb_get`, `kb_update_note`, `kb_search`
+- [x] 1.5 Update MCP server instructions text to reflect tags-only approach
+
+## 2. Engine bulk infrastructure
+
+- [x] 2.1 Add `bulk_safety_percent` to `Config` class in `engine/kb/config.py` (env var `KB_BULK_SAFETY_PERCENT`, default 70)
+- [x] 2.2 Add `job_type` column migration to `database.py` `init_schema` (TEXT, default "ingest")
+- [x] 2.3 Add `resolve_bulk_selection(conn, document_ids, tags, doc_type, from_id, to_id)` helper to `database.py` — returns list of matching document IDs
+- [x] 2.4 Add `create_bulk_job(conn, job_type, filters_json, matched, succeeded, failed, errors_json)` helper to `database.py`
+
+## 3. Engine bulk endpoints
+
+- [x] 3.1 Create `engine/kb/routes/bulk.py` with shared Pydantic request model (`BulkSelectionRequest` with selection fields + `force` bool)
+- [x] 3.2 Add `_check_safety_threshold` helper that returns 409 if threshold exceeded
+- [x] 3.3 Implement `POST /api/v1/bulk/delete` — resolve selection, check threshold, delete documents in transaction, clean up files, log job, return summary
+- [x] 3.4 Implement `POST /api/v1/bulk/tags` — resolve selection, check threshold, add/remove tags on matched docs, log job, return summary
+- [x] 3.5 Implement `POST /api/v1/bulk/set-tags` — resolve selection, check threshold, clear and replace tags on matched docs, log job, return summary
+- [x] 3.6 Import bulk routes in engine app startup (add to `engine/kb/routes/__init__.py` or `main.py`)
+
+## 4. MCP bulk tools
+
+- [x] 4.1 Add `bulk_delete`, `bulk_tags`, `bulk_set_tags` methods to `mcp/engine.py`
+- [x] 4.2 Add `kb_bulk_delete` tool to `mcp/server.py`
+- [x] 4.3 Add `kb_bulk_tags` tool to `mcp/server.py`
+- [x] 4.4 Add `kb_bulk_set_tags` tool to `mcp/server.py`
+
+## 5. CLI bulk commands
+
+- [x] 5.1 Create `client/cmd/bulk_remove.go` — `kb bulk-remove` with filter flags, confirmation prompt, JSON output support
+- [x] 5.2 Create `client/cmd/bulk_tag.go` — `kb bulk-tag` with filter flags + `--add`/`--remove`, confirmation prompt
+- [x] 5.3 Create `client/cmd/bulk_set_tags.go` — `kb bulk-set-tags` with filter flags + `--set`, confirmation prompt
+
+## 6. Verification
+
+- [x] 6.1 Test collection removal: verify `kb_search`, `kb_addnote`, `kb_get`, `kb_update_note`, `kb_upload_start` work without collection params
+- [x] 6.2 Test bulk delete via engine API: filter by tags, by IDs, by range, safety threshold trigger and force override
+- [x] 6.3 Test bulk tags and bulk set-tags via engine API
+- [x] 6.4 Test MCP bulk tools against running engine
+- [x] 6.5 Test CLI bulk commands against running engine
+- [x] 6.6 Test audit trail: verify bulk jobs appear in `kb jobs` output
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-04-04
@@ -0,0 +1,59 @@
+## Context
+
+The MCP Python SDK includes DNS rebinding protection via `TransportSecuritySettings` in `mcp.server.transport_security`. When enabled, it validates the `Host` header against an allowlist and returns 421 for unrecognised hosts.
+
+`FastMCP` auto-enables this protection when `host` is `127.0.0.1`, `localhost`, or `::1`, with a default allowlist of those three values (wildcard port). The kb MCP server does not pass a `host` to `FastMCP()` and does not pass `transport_security`, so the behaviour depends on the SDK's defaults — which have changed between versions and will likely keep changing.
+
+Currently the server calls `mcp.streamable_http_app()` to get a Starlette sub-app and wraps it in its own Starlette app with `BearerAuthMiddleware`. The `transport_security` settings flow through `FastMCP()` → `Settings` → `StreamableHTTPSessionManager`, so they must be set at `FastMCP` construction time.
+
+When a remote client connects (e.g. `Host: 192.168.1.50:3000`), the SDK rejects the request with 421 before our auth middleware even runs.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Allow operators to configure additional allowed hosts via environment variable so remote clients can connect.
+- Support both IP addresses and FQDNs, with or without port.
+- Preserve DNS rebinding protection (keep it enabled, just widen the allowlist).
+- Maintain backward compatibility — unset variable means localhost-only, same as today.
+
+**Non-Goals:**
+
+- Disabling DNS rebinding protection entirely.
+- Configuring allowed origins separately from allowed hosts (derive origins automatically from hosts).
+- TLS termination or HTTPS — that belongs to a reverse proxy in front of the MCP container.
+
+## Decisions
+
+### 1. Environment variable format
+
+`KB_MCP_ALLOWED_HOSTS` is a comma-separated list of hosts. Each entry is an IP address or FQDN without port and without scheme.
+
+Examples: `192.168.1.50`, `kb.example.com`, `192.168.1.50,kb.example.com,10.0.0.1`
+
+**Rationale:** Comma-separated is the simplest format that doesn't require quoting in Docker Compose YAML or shell environments. Ports are omitted because the wildcard-port pattern (`host:*`) covers all ports — operators shouldn't need to know the internal port.
+
+**Alternative considered:** JSON array — rejected, awkward in env vars and Compose files.
+
+### 2. Merge with localhost defaults
+
+The parsed hosts are merged with the hardcoded localhost set (`127.0.0.1`, `localhost`, `[::1]`). Localhost is always allowed regardless of the env var value.
+
+**Rationale:** Removing localhost would break local development and health checks. There's no reason to ever disallow it.
+
+### 3. Auto-derive allowed origins from allowed hosts
+
+For each allowed host, generate `http://<host>:*` as an allowed origin. No separate env var for origins.
+
+**Rationale:** The MCP server doesn't serve HTTPS (TLS is terminated by a reverse proxy), so `http://` is always correct at the container level. If HTTPS origins are needed in future, a separate env var can be added then.
+
+### 4. Pass TransportSecuritySettings explicitly to FastMCP
+
+Always construct a `TransportSecuritySettings` with `enable_dns_rebinding_protection=True` and the merged allowlist, and pass it as `transport_security=` to `FastMCP()`. This makes the behaviour explicit rather than depending on SDK defaults.
+
+**Rationale:** The SDK's auto-detection logic depends on the `host` parameter which we don't set, and the defaults may change between SDK versions. Being explicit removes the ambiguity.
+
+## Risks / Trade-offs
+
+- **Operator must know their Host header value** — If a reverse proxy rewrites the Host header, the operator needs to allowlist the rewritten value, not the original. → Mitigation: document this in Compose file comments.
+- **No HTTPS origin support** — If a client sends `Origin: https://...`, it will be rejected. → Mitigation: acceptable for now; the MCP server sits behind a proxy that terminates TLS. Can add `KB_MCP_ALLOWED_ORIGINS` later if needed.
@@ -0,0 +1,26 @@
+## Why
+
+The MCP server uses the Python MCP SDK's built-in DNS rebinding protection, which validates the `Host` header on every request. By default it only allows `localhost`, `127.0.0.1`, and `[::1]`. When clients connect remotely — using an IP address or FQDN — the server returns 421 "Invalid Host header" and the connection fails. There is no way to configure allowed hosts without changing code.
+
+## What Changes
+
+- Add a new environment variable `KB_MCP_ALLOWED_HOSTS` that accepts a comma-separated list of additional allowed hosts (IPs and/or FQDNs).
+- The MCP server passes these hosts (plus the existing localhost defaults) to the MCP SDK's `TransportSecuritySettings` when constructing the ASGI app.
+- Both bare hosts and wildcard-port patterns are supported (e.g. `192.168.1.50` and `kb.example.com` both work, with any port).
+- When `KB_MCP_ALLOWED_HOSTS` is empty or unset, behaviour is unchanged (localhost-only).
+
+## Capabilities
+
+### New Capabilities
+
+_None — this is configuration of an existing component, not a new capability._
+
+### Modified Capabilities
+
+- `docker-deployment`: Add `KB_MCP_ALLOWED_HOSTS` to the MCP container's environment variables in Compose files and document its usage.
+
+## Impact
+
+- **mcp/config.py** — new `KB_MCP_ALLOWED_HOSTS` env var.
+- **mcp/server.py** — construct `TransportSecuritySettings` with merged allowed hosts/origins and pass to the FastMCP app.
+- **engine/compose.\*.yaml** — add `KB_MCP_ALLOWED_HOSTS` to the kb-mcp service environment block.
@@ -0,0 +1,65 @@
+## ADDED Requirements
+
+### Requirement: Configurable MCP allowed hosts
+
+The MCP server SHALL accept a `KB_MCP_ALLOWED_HOSTS` environment variable containing a comma-separated list of additional hosts (IP addresses or FQDNs) that are permitted to connect. The server SHALL always allow `127.0.0.1`, `localhost`, and `[::1]` regardless of this setting. DNS rebinding protection SHALL always be enabled.
+
+#### Scenario: Remote client connects with allowed host
+
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is set to `192.168.1.50` and a client connects with `Host: 192.168.1.50:3000`
+- **THEN** the server SHALL accept the request and process it normally
+
+#### Scenario: Remote client connects with disallowed host
+
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is set to `192.168.1.50` and a client connects with `Host: 10.0.0.99:3000`
+- **THEN** the server SHALL return HTTP 421 "Invalid Host header"
+
+#### Scenario: Multiple allowed hosts
+
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is set to `192.168.1.50,kb.example.com`
+- **THEN** the server SHALL accept requests with `Host` matching either `192.168.1.50` or `kb.example.com` on any port
+
+#### Scenario: Variable unset or empty
+
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is unset or empty
+- **THEN** the server SHALL allow only localhost addresses (`127.0.0.1`, `localhost`, `[::1]`) with any port
+
+#### Scenario: Localhost always allowed
+
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is set to `192.168.1.50`
+- **THEN** the server SHALL still accept requests with `Host: localhost:3000` or `Host: 127.0.0.1:3000`
+
+#### Scenario: Allowed origins derived from allowed hosts
+
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` includes `192.168.1.50`
+- **THEN** the server SHALL accept `Origin: http://192.168.1.50:3000` (and any port) in addition to localhost origins
+
+## MODIFIED Requirements
+
+### Requirement: Compose files for deployment
+
+The project SHALL provide Docker Compose files for single-command deployment. Compose files SHALL use `build:` context for local development. Release notes SHALL document the versioned image tag for users pulling pre-built images.
+
+#### Scenario: Start NVIDIA deployment
+- **WHEN** an admin runs `docker compose -f compose.nvidia.yaml up -d`
+- **THEN** the engine SHALL start with GPU access, bind-mount the data directory, and be reachable on the configured port
+
+#### Scenario: Start ROCm deployment
+- **WHEN** an admin runs `docker compose -f compose.rocm.yaml up -d`
+- **THEN** the engine SHALL start with GPU access via ROCm device passthrough, bind-mount the data directory, and be reachable on the configured port
+
+#### Scenario: Automatic restart
+- **WHEN** the engine process crashes or the host reboots
+- **THEN** Docker SHALL automatically restart the container (restart policy `unless-stopped`)
+
+#### Scenario: Configure via environment
+- **WHEN** an admin sets environment variables in the compose file (KB_MODEL, KB_API_KEY, KB_DEVICE, KB_MCP_ALLOWED_HOSTS, etc.)
+- **THEN** the engine and MCP server SHALL use those values
+
+#### Scenario: Pre-built image deployment
+- **WHEN** an admin wants to use a pre-built engine image without building from source
+- **THEN** the engine release notes SHALL include the exact `docker pull` command with the versioned tag (e.g. `docker.dcglab.co.uk/dcg/kb/engine:engine-v2.1.0-nvidia`)
+
+#### Scenario: MCP allowed hosts in Compose
+- **WHEN** the kb-mcp service is defined in a Compose file
+- **THEN** the environment block SHALL include `KB_MCP_ALLOWED_HOSTS` with a comment explaining its format and purpose
@@ -0,0 +1,18 @@
+## 1. Configuration
+
+- [x] 1.1 Add `KB_MCP_ALLOWED_HOSTS` to `mcp/config.py` — read from env, default empty string
+- [x] 1.2 Add host-parsing helper that splits the comma-separated value, strips whitespace, and filters empty entries
+
+## 2. Transport security
+
+- [x] 2.1 Build `TransportSecuritySettings` in `mcp/server.py` — merge localhost defaults with parsed `KB_MCP_ALLOWED_HOSTS`, derive allowed origins from allowed hosts
+- [x] 2.2 Pass `transport_security=` to the `FastMCP()` constructor
+
+## 3. Compose files
+
+- [x] 3.1 Add `KB_MCP_ALLOWED_HOSTS=${KB_MCP_ALLOWED_HOSTS:-}` to the kb-mcp environment block in `compose.cpu.yaml`, `compose.nvidia.yaml`, and `compose.rocm.yaml` with a comment explaining the format
+
+## 4. Verification
+
+- [x] 4.1 Test: unset `KB_MCP_ALLOWED_HOSTS` — confirm localhost connects, remote host gets 421
+- [x] 4.2 Test: set `KB_MCP_ALLOWED_HOSTS` to the server IP — confirm remote host connects successfully
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-04-04
@@ -0,0 +1,39 @@
+## Context
+
+The MCP server (`mcp/server.py`) exposes KB operations as tools for LLM clients. Collections are an abstraction over tags — internally stored with a `collection:` prefix. The server already has helpers for managing collection tags (`_collection_tag`, `_ensure_exclusive_collection`, `_strip_collection_tags`) and the engine client (`mcp/engine.py`) already has an `update_tags()` method.
+
+Document deletion is supported by the engine API at `DELETE /api/v1/documents/{doc_id}` but has no corresponding engine client method or MCP tool.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Expose collection assignment for existing documents via MCP (`kb_set_collection`)
+- Expose document deletion via MCP (`kb_delete`)
+- Follow existing patterns in `server.py` and `engine.py`
+
+**Non-Goals:**
+- Bulk operations (multi-document collection assignment or deletion)
+- Tag management beyond collections (direct tag add/remove via MCP)
+- Undo/recycle bin for deleted documents
+- Changes to the engine API layer — all endpoints already exist
+
+## Decisions
+
+### 1. Reuse `_ensure_exclusive_collection` for kb_set_collection
+
+The server already has `_ensure_exclusive_collection(doc_id, collection)` which removes any existing `collection:*` tags and applies the new one. The `kb_set_collection` tool will use this directly when a collection is provided, and manually remove collection tags when clearing.
+
+**Alternative considered**: Exposing raw tag add/remove to the LLM. Rejected because it leaks the `collection:` prefix implementation detail and the LLM could create inconsistent state (multiple collections on one document).
+
+### 2. New `engine.delete_document()` method for kb_delete
+
+Add a simple `delete_document(doc_id)` to `mcp/engine.py` that calls `DELETE /api/v1/documents/{doc_id}`. This follows the same pattern as all other engine client methods.
+
+### 3. Return confirmation with document metadata on delete
+
+`kb_delete` will return the response from the engine API which includes `{"status": "deleted", "document_id": ..., "title": ...}`. This gives the LLM confirmation of what was deleted without needing a separate get call.
+
+## Risks / Trade-offs
+
+- **[Accidental deletion]** → The LLM could delete the wrong document. Mitigation: the tool requires an explicit `document_id`, and the response includes the title so the LLM can verify. No bulk delete is exposed.
+- **[Collection cleared unexpectedly]** → Passing `collection=None` to `kb_set_collection` removes collection assignment. Mitigation: the parameter description will make this behavior explicit.
@@ -0,0 +1,25 @@
+## Why
+
+LLMs using the KB MCP server can create notes in collections and search by collection, but cannot assign existing documents to a collection or delete documents. This forces users to drop out to the HTTP API for routine document management. Both operations are fully supported at the database and HTTP API layers but aren't wired through to MCP tools.
+
+## What Changes
+
+- Add `kb_set_collection` MCP tool — assigns, changes, or removes the collection on an existing document by manipulating `collection:` prefixed tags via the existing `engine.update_tags()` method.
+- Add `kb_delete` MCP tool — deletes a document by ID, calling the existing `DELETE /api/v1/documents/{doc_id}` endpoint via a new `engine.delete_document()` method.
+
+## Capabilities
+
+### New Capabilities
+
+- `mcp-document-management`: MCP tools for modifying and deleting existing documents (kb_set_collection, kb_delete).
+
+### Modified Capabilities
+
+_(none — the engine API endpoints already exist; this change only adds MCP tool wrappers)_
+
+## Impact
+
+- **MCP server** (`mcp/server.py`): Two new tool registrations.
+- **MCP engine client** (`mcp/engine.py`): One new method (`delete_document`). The `update_tags` method already exists and will be reused.
+- **Engine API**: No changes — `DELETE /api/v1/documents/{doc_id}` and `PUT /api/v1/documents/{doc_id}/tags` already exist.
+- **Breaking changes**: None. Additive only.
@@ -0,0 +1,61 @@
+## ADDED Requirements
+
+### Requirement: Set collection on existing document via MCP
+
+The MCP server SHALL expose a `kb_set_collection` tool that assigns or changes the collection of an existing document. The tool SHALL accept a `document_id` (required) and `collection` (optional string). When `collection` is provided, the tool SHALL ensure the document belongs to exactly that collection by removing any existing `collection:*` tags and adding the new one. When `collection` is omitted or null, the tool SHALL remove all `collection:*` tags from the document, leaving it unassigned.
+
+The tool SHALL return the updated document with the `collection` field and cleaned tags (collection tags stripped), consistent with other MCP tool responses.
+
+#### Scenario: Assign untagged document to a collection
+
+- **WHEN** `kb_set_collection` is called with `document_id=42` and `collection="workspace"`
+- **THEN** the document SHALL have the tag `collection:workspace` added
+- **AND** the response SHALL include `"collection": "workspace"`
+
+#### Scenario: Change document from one collection to another
+
+- **WHEN** `kb_set_collection` is called with `document_id=42` and `collection="memory"` on a document currently in collection "documents"
+- **THEN** the tag `collection:documents` SHALL be removed and `collection:memory` SHALL be added
+- **AND** the response SHALL include `"collection": "memory"`
+
+#### Scenario: Remove document from all collections
+
+- **WHEN** `kb_set_collection` is called with `document_id=42` and no `collection` parameter
+- **THEN** all `collection:*` tags SHALL be removed from the document
+- **AND** the response SHALL include `"collection": null`
+
+#### Scenario: Document not found
+
+- **WHEN** `kb_set_collection` is called with a `document_id` that does not exist
+- **THEN** the tool SHALL return an error response indicating the document was not found
+
+### Requirement: Delete document via MCP
+
+The MCP server SHALL expose a `kb_delete` tool that permanently deletes a document from the knowledge base. The tool SHALL accept a `document_id` (required integer). Deletion SHALL remove the document, its chunks, embeddings, tags, and any stored file on disk.
+
+The tool SHALL return a confirmation response including the deleted document's ID and title.
+
+#### Scenario: Successful deletion
+
+- **WHEN** `kb_delete` is called with `document_id=42`
+- **THEN** the document, its chunks, embeddings, tag associations, and stored file SHALL be deleted
+- **AND** the response SHALL include `"status": "deleted"`, the `document_id`, and the document `title`
+
+#### Scenario: Document not found
+
+- **WHEN** `kb_delete` is called with a `document_id` that does not exist
+- **THEN** the tool SHALL return an error response indicating the document was not found
+
+### Requirement: Engine client delete method
+
+The MCP engine client (`mcp/engine.py`) SHALL provide a `delete_document(doc_id)` method that sends a `DELETE` request to `/api/v1/documents/{doc_id}` and returns the JSON response. The method SHALL raise on non-2xx status codes, consistent with other engine client methods.
+
+#### Scenario: Successful engine client delete call
+
+- **WHEN** `delete_document(42)` is called and the engine API returns 200
+- **THEN** the method SHALL return the parsed JSON response
+
+#### Scenario: Engine client delete for missing document
+
+- **WHEN** `delete_document(999)` is called and the engine API returns 404
+- **THEN** the method SHALL raise an `httpx.HTTPStatusError`
@@ -0,0 +1,12 @@
+## 1. Engine Client
+
+- [x] 1.1 Add `delete_document(doc_id)` method to `mcp/engine.py`
+
+## 2. MCP Tools
+
+- [x] 2.1 Add `kb_set_collection` tool to `mcp/server.py`
+- [x] 2.2 Add `kb_delete` tool to `mcp/server.py`
+
+## 3. Verification
+
+- [x] 3.1 Test kb_set_collection and kb_delete against running engine
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-04-06
@@ -0,0 +1,37 @@
+## Context
+
+The project currently ships three Docker image variants: CPU, NVIDIA, and AMD ROCm. The ROCm variant requires a 4.2GB pre-built torch wheel, a multi-stage Dockerfile with ROCm-specific runtime libraries, and additional build/push steps in the release pipeline. ROCm support is less tested and adds disproportionate complexity relative to its usage.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Remove all ROCm-specific files (Dockerfile, compose file, torch wheel)
+- Remove ROCm build/push from the release pipeline
+- Update all documentation to reflect CPU + NVIDIA only
+- Update the docker-deployment spec to remove ROCm requirements
+
+**Non-Goals:**
+- Changing any engine application code (it is already GPU-vendor-agnostic via PyTorch)
+- Modifying the CPU or NVIDIA Dockerfiles (beyond what's already in-flight)
+- Providing a migration path for ROCm users (they can stay on 3.2.x or use CPU mode)
+
+## Decisions
+
+**1. Delete ROCm files outright rather than deprecating**
+
+Remove `Dockerfile.rocm`, `compose.rocm.yaml`, and `assets/` immediately rather than marking them deprecated. There are no downstream consumers that depend on automated ROCm builds — anyone needing AMD support can pin to the last ROCm-supporting release.
+
+*Alternative considered*: Keep files but stop publishing images. Rejected — dead code is confusing and still requires maintenance awareness.
+
+**2. Leave archived openspec changes untouched**
+
+Archived changes under `openspec/changes/archive/` contain historical ROCm references. These are historical records and should not be modified.
+
+**3. Update GPU-vendor-agnostic requirement to reflect NVIDIA-only scope**
+
+The existing spec requirement "Application code is GPU-vendor-agnostic" remains true at the code level (PyTorch abstracts GPU vendors), but the project no longer provides or tests ROCm images. The spec should be simplified to reflect that only NVIDIA and CPU are supported deployment targets.
+
+## Risks / Trade-offs
+
+- **[Breaking change for AMD users]** → Users on AMD GPUs must stay on 3.2.x or use CPU mode. Mitigated by the fact that ROCm support was already "less tested" per the original design risk assessment.
+- **[Future re-addition harder]** → If ROCm support is needed later, the Dockerfile and compose file would need to be recreated. Mitigated by git history preserving the removed files.
@@ -0,0 +1,29 @@
+## Why
+
+AMD ROCm support adds significant complexity and maintenance burden to the project — the ROCm torch wheel alone is 4.2GB, the Dockerfile requires a multi-stage build with ROCm-specific runtime libraries, and the release pipeline must build/push additional images. The final container is >20Gb. ROCm support is less tested and less commonly used than CPU or NVIDIA. Removing it keeps the project focused and manageable.
+
+## What Changes
+
+- **BREAKING**: Remove AMD ROCm Docker image (`Dockerfile.rocm`) and compose file (`compose.rocm.yaml`)
+- **BREAKING**: Remove ROCm image build/push/release-notes from the engine release script
+- Remove pre-built ROCm torch wheel from `assets/`
+- Remove all AMD/ROCm references from user-facing docs (README, DEVELOPER)
+- Update docker-deployment spec to reflect CPU + NVIDIA only
+
+## Capabilities
+
+### New Capabilities
+
+_(none)_
+
+### Modified Capabilities
+
+- `docker-deployment`: Remove AMD ROCm Docker image requirement and all ROCm-specific scenarios. Deployment now covers CPU and NVIDIA only.
+
+## Impact
+
+- **Docker images**: ROCm image variant no longer published
+- **Users**: Anyone running KB on AMD GPUs will need to stay on the last version with ROCm support (3.2.x) or switch to CPU mode
+- **Release pipeline**: `release-engine.sh` simplified — only CPU and NVIDIA images
+- **Repository size**: ~4.2GB reduction by removing the torch wheel from `assets/`
+- **Docs**: README and DEVELOPER updated to remove AMD quick-start and build instructions
@@ -0,0 +1,76 @@
+## REMOVED Requirements
+
+### Requirement: AMD ROCm Docker image
+
+**Reason**: AMD ROCm support removed to reduce project complexity and binary size. The ROCm torch wheel is 4.2GB and the variant is less tested than CPU or NVIDIA.
+
+**Migration**: Users on AMD GPUs should stay on engine v3.2.x or switch to CPU mode (`KB_DEVICE=cpu`).
+
+---
+
+## MODIFIED Requirements
+
+### Requirement: Application code is GPU-vendor-agnostic
+
+The Python engine code SHALL NOT reference CUDA directly. GPU abstraction SHALL be handled at the Docker image level (base image selection and pip package choice). The same application code SHALL run on both NVIDIA and CPU images without modification.
+
+#### Scenario: Same engine code on both platforms
+- **WHEN** the engine starts on an NVIDIA image and a CPU image with identical configuration
+- **THEN** both SHALL load the model, accept requests, and return identical search results for the same query and data
+
+---
+
+### Requirement: Compose files for deployment
+
+The project SHALL provide Docker Compose files for single-command deployment. Compose files SHALL use `build:` context for local development. Release notes SHALL document the versioned image tag for users pulling pre-built images.
+
+#### Scenario: Start NVIDIA deployment
+- **WHEN** an admin runs `docker compose -f compose.nvidia.yaml up -d`
+- **THEN** the engine SHALL start with GPU access, bind-mount the data directory, and be reachable on the configured port
+
+#### Scenario: Automatic restart
+- **WHEN** the engine process crashes or the host reboots
+- **THEN** Docker SHALL automatically restart the container (restart policy `unless-stopped`)
+
+#### Scenario: Configure via environment
+- **WHEN** an admin sets environment variables in the compose file (KB_MODEL, KB_API_KEY, KB_DEVICE, KB_MCP_ALLOWED_HOSTS, etc.)
+- **THEN** the engine and MCP server SHALL use those values
+
+#### Scenario: Pre-built image deployment
+- **WHEN** an admin wants to use a pre-built engine image without building from source
+- **THEN** the engine release notes SHALL include the exact `docker pull` command with the versioned tag (e.g. `docker.dcglab.co.uk/dcg/kb/engine:engine-v2.1.0-nvidia`)
+
+#### Scenario: MCP allowed hosts in Compose
+- **WHEN** the kb-mcp service is defined in a Compose file
+- **THEN** the environment block SHALL include `KB_MCP_ALLOWED_HOSTS` with a comment explaining its format and purpose
+
+---
+
+### Requirement: Bind-mount data directory
+
+The engine SHALL store all persistent state (SQLite database, HF model cache, staging directory) under a single configurable data directory. This directory SHALL be mounted from the host via bind mount.
+
+#### Scenario: Data directory structure
+- **WHEN** the engine starts for the first time
+- **THEN** it SHALL create the following structure under the data directory:
+  - `kb.db` — SQLite database
+  - `hf_cache/` — HuggingFace model cache
+  - `staging/` — temporary files for queued ingestion jobs
+
+#### Scenario: Portable data across hosts
+- **WHEN** an admin copies the data directory from Host A to Host B and starts the engine with the same bind mount path
+- **THEN** the engine SHALL start successfully and serve all previously ingested documents without reprocessing
+
+---
+
+### Requirement: CPU-only fallback
+
+The Dockerfiles SHALL produce images that work without GPU access. If no GPU is available, the engine SHALL fall back to CPU for all operations.
+
+#### Scenario: No GPU available
+- **WHEN** the container starts without GPU passthrough (no `--gpus`)
+- **THEN** the engine SHALL detect no GPU, load the model on CPU, and log a warning that GPU acceleration is unavailable
+
+#### Scenario: Explicit CPU mode
+- **WHEN** `KB_DEVICE=cpu` and `KB_INGEST_DEVICE=cpu` are set in the environment
+- **THEN** the engine SHALL use CPU regardless of GPU availability
@@ -0,0 +1,20 @@
+## 1. Delete ROCm files
+
+- [x] 1.1 Delete `engine/Dockerfile.rocm`
+- [x] 1.2 Delete `engine/compose.rocm.yaml`
+- [x] 1.3 Delete `assets/` directory (ROCm torch wheel)
+
+## 2. Update release pipeline
+
+- [x] 2.1 Remove ROCm image build, tag, and push from `release-engine.sh`
+- [x] 2.2 Remove ROCm entries from release notes output in `release-engine.sh`
+
+## 3. Update documentation
+
+- [x] 3.1 Remove AMD GPU quick-start section and ROCm references from `README.md`
+- [x] 3.2 Remove ROCm build instructions and `compose.rocm.yaml` references from `DEVELOPER.md`
+- [x] 3.3 Remove `onnxruntime-rocm` migration note from `DEVELOPER.md`
+
+## 4. Update specs
+
+- [x] 4.1 Update `openspec/specs/docker-deployment/spec.md` — remove AMD ROCm requirement, remove ROCm scenarios, update GPU-agnostic requirement to CPU + NVIDIA scope
@@ -0,0 +1,43 @@
+# Agent-Side Search Patterns
+
+## Purpose
+
+Documents recommended patterns for agent-side query expansion, plus how agent guidance interacts with the engine's optional server-side reranking. These patterns are communicated via MCP tool descriptions.
+
+## Requirements
+
+### Requirement: Query expansion guidance in tool description
+
+The `kb_search` MCP tool description SHALL include guidance on query expansion as a recommended pattern for complex queries.
+
+#### Scenario: Tool description includes expansion pattern
+- **WHEN** an agent reads the `kb_search` tool description
+- **THEN** the description SHALL include guidance such as: "For complex queries, consider expanding into 2-3 variant phrasings and calling this tool multiple times, then deduplicating results by chunk_id"
+
+---
+
+### Requirement: Reranking guidance in tool description
+
+The `kb_search` MCP tool description SHALL describe the engine's server-side reranking behaviour and retain agent-side reranking as a fallback pattern.
+
+#### Scenario: Tool description covers server-side reranking
+- **WHEN** an agent reads the `kb_search` tool description
+- **THEN** the description SHALL state that results are reranked server-side by default when the engine has a reranker enabled, that `rerank=False` skips it for lower latency, and that `kb_status` reports whether reranking is active
+
+#### Scenario: Tool description retains agent-side fallback
+- **WHEN** an agent reads the `kb_search` tool description
+- **THEN** the description SHALL include guidance that, when the engine's reranker is disabled, the agent can rerank the returned results using its own judgement of relevance to the original question
+
+---
+
+### Requirement: No external LLM dependency
+
+The engine SHALL NOT require or use any external LLM API for search operations. Query expansion SHALL remain an agent-side concern. Reranking MAY be performed engine-side using a local, opt-in cross-encoder model; it SHALL never depend on an external API.
+
+#### Scenario: Engine has no external LLM dependency
+- **WHEN** the engine is deployed without any `ANTHROPIC_API_KEY` or similar LLM API configuration
+- **THEN** all search operations SHALL function fully, with no degraded results or missing features
+
+#### Scenario: Reranking is optional and degrades gracefully
+- **WHEN** the engine is deployed with `KB_RERANK_ENABLED` unset or false, or the reranker model fails to load
+- **THEN** all search operations SHALL function fully using hybrid retrieval alone, with responses reporting `"reranked": false`
@@ -0,0 +1,230 @@
+## ADDED Requirements
+
+### Requirement: Common selection filter
+
+All bulk engine endpoints SHALL accept a JSON body with the following optional selection fields, combined with AND logic:
+
+- `document_ids` (list of int) — match documents with these specific IDs
+- `tags` (list of str) — match documents that have ALL specified tags
+- `doc_type` (str) — match documents with this document type
+- `from_id` (int) — match documents with id >= this value
+- `to_id` (int) — match documents with id <= this value
+
+At least one selection field MUST be present. If no selection fields are provided, the endpoint SHALL return 400 Bad Request.
+
+#### Scenario: Filter by tags and doc_type
+
+- **WHEN** a bulk endpoint receives `{"tags": ["draft"], "doc_type": "note"}`
+- **THEN** it SHALL match only documents that have the tag "draft" AND have doc_type "note"
+
+#### Scenario: Filter by ID range
+
+- **WHEN** a bulk endpoint receives `{"from_id": 10, "to_id": 50}`
+- **THEN** it SHALL match documents with id >= 10 AND id <= 50
+
+#### Scenario: Filter by explicit IDs
+
+- **WHEN** a bulk endpoint receives `{"document_ids": [1, 5, 12]}`
+- **THEN** it SHALL match only documents with those specific IDs
+
+#### Scenario: Combined filters
+
+- **WHEN** a bulk endpoint receives `{"tags": ["agent:mybot"], "doc_type": "note", "from_id": 100}`
+- **THEN** it SHALL match documents satisfying ALL three criteria
+
+#### Scenario: No selection fields provided
+
+- **WHEN** a bulk endpoint receives `{}` or `{"force": true}` with no selection fields
+- **THEN** it SHALL return 400 Bad Request
+
+### Requirement: Safety threshold
+
+All bulk endpoints SHALL enforce a safety threshold. Before executing, the engine SHALL count the matched documents and the total documents in the database. If `matched / total * 100` exceeds the configured threshold, the request SHALL be rejected with 409 Conflict.
+
+The response SHALL include: `error` ("safety_threshold_exceeded"), `message` (human-readable), `matched` (int), `total` (int), `percent` (float), and `threshold` (int).
+
+The threshold SHALL default to 70 and be configurable via the `KB_BULK_SAFETY_PERCENT` environment variable (integer 0-100). A value of 0 disables the check.
+
+The caller MAY override the threshold by including `"force": true` in the request body.
+
+#### Scenario: Threshold exceeded
+
+- **GIVEN** 1000 total documents and `KB_BULK_SAFETY_PERCENT` is 70
+- **WHEN** a bulk endpoint matches 750 documents (75%) without `force: true`
+- **THEN** it SHALL return 409 with `matched: 750`, `total: 1000`, `percent: 75.0`, `threshold: 70`
+
+#### Scenario: Threshold not exceeded
+
+- **GIVEN** 1000 total documents and `KB_BULK_SAFETY_PERCENT` is 70
+- **WHEN** a bulk endpoint matches 500 documents (50%) without `force: true`
+- **THEN** the operation SHALL proceed normally
+
+#### Scenario: Force override
+
+- **GIVEN** 1000 total documents and a match of 900 (90%)
+- **WHEN** the request includes `"force": true`
+- **THEN** the operation SHALL proceed regardless of threshold
+
+#### Scenario: Zero threshold
+
+- **GIVEN** `KB_BULK_SAFETY_PERCENT` is 0
+- **THEN** the safety check SHALL be effectively disabled for all operations
+
+### Requirement: Synchronous response with audit log
+
+All bulk endpoints SHALL execute synchronously and return a JSON response with:
+
+- `job_id` (int) — ID of the audit log entry in the jobs table
+- `status` (str) — "done" or "partial_failure"
+- `matched` (int) — number of documents that matched the selection
+- `succeeded` (int) — number of documents successfully processed
+- `failed` (int) — number of documents that failed
+- `errors` (list) — array of `{"document_id": int, "error": str}` for each failure (empty on full success)
+
+A job record SHALL be created in the jobs table with `job_type` set to the operation type. The `filename` field SHALL store a JSON representation of the selection filter. The `error` field SHALL store a JSON array of individual errors if any occurred.
+
+#### Scenario: Full success
+
+- **WHEN** a bulk operation matches 50 documents and all succeed
+- **THEN** the response SHALL have `status: "done"`, `matched: 50`, `succeeded: 50`, `failed: 0`, `errors: []`
+
+#### Scenario: Partial failure
+
+- **WHEN** a bulk operation matches 50 documents but 2 fail
+- **THEN** the response SHALL have `status: "partial_failure"`, `matched: 50`, `succeeded: 48`, `failed: 2`, and `errors` listing the 2 failures
+
+### Requirement: Bulk delete endpoint
+
+The engine SHALL expose `POST /api/v1/bulk/delete` which permanently deletes all documents matching the selection filter. For each matched document, it SHALL delete embeddings from `chunks_vec`, delete the document row (cascading to chunks and document_tags), and delete any stored file from disk.
+
+Database deletions SHALL be performed within a single transaction. File deletions SHALL occur after the transaction commits and SHALL be best-effort (failures logged but not counted as document failures).
+
+#### Scenario: Bulk delete by tag
+
+- **WHEN** `POST /api/v1/bulk/delete` receives `{"tags": ["old", "draft"]}`
+- **THEN** all documents with both tags "old" and "draft" SHALL be deleted
+- **AND** their chunks, embeddings, tag associations, and stored files SHALL be removed
+
+#### Scenario: Bulk delete with no matches
+
+- **WHEN** `POST /api/v1/bulk/delete` receives a filter that matches 0 documents
+- **THEN** the response SHALL have `matched: 0`, `succeeded: 0`, `failed: 0`
+
+### Requirement: Bulk tags endpoint
+
+The engine SHALL expose `POST /api/v1/bulk/tags` which adds and/or removes tags on all documents matching the selection filter. The request body SHALL include the selection filter plus:
+
+- `add` (list of str, optional) — tags to add
+- `remove` (list of str, optional) — tags to remove
+
+At least one of `add` or `remove` MUST be present. The endpoint SHALL return 400 if neither is provided.
+
+The endpoint SHALL update `updated_at` on all affected documents.
+
+#### Scenario: Add and remove tags in one call
+
+- **WHEN** `POST /api/v1/bulk/tags` receives `{"tags": ["agent:mybot"], "add": ["reviewed"], "remove": ["pending"]}`
+- **THEN** all documents tagged "agent:mybot" SHALL have "reviewed" added and "pending" removed
+
+### Requirement: Bulk set-tags endpoint
+
+The engine SHALL expose `POST /api/v1/bulk/set-tags` which replaces all tags on matched documents with a new set. The request body SHALL include the selection filter plus:
+
+- `new_tags` (list of str) — the replacement tag set
+
+The endpoint SHALL remove all existing tag associations from matched documents, then apply the new set. It SHALL update `updated_at` on all affected documents.
+
+#### Scenario: Replace all tags
+
+- **WHEN** `POST /api/v1/bulk/set-tags` receives `{"doc_type": "note", "new_tags": ["clean", "final"]}`
+- **THEN** all notes SHALL have their existing tags removed and replaced with "clean" and "final"
+
+### Requirement: Jobs table extension
+
+The jobs table SHALL be extended with a `job_type` column (TEXT, default "ingest") to distinguish ingestion jobs from bulk operation audit entries. Valid values: "ingest", "bulk_delete", "bulk_tags", "bulk_set_tags".
+
+Existing jobs SHALL default to `job_type = "ingest"`. The existing jobs list endpoint and CLI `kb jobs` command SHALL continue to work unchanged.
+
+#### Scenario: Migration adds column
+
+- **GIVEN** an existing database without the `job_type` column
+- **WHEN** the engine starts
+- **THEN** the column SHALL be added with default value "ingest"
+
+### Requirement: Engine config for safety threshold
+
+The engine `Config` class SHALL read `KB_BULK_SAFETY_PERCENT` from the environment as an integer (default 70, range 0-100). This value SHALL be used as the default safety threshold for all bulk endpoints.
+
+### Requirement: MCP bulk delete tool
+
+The MCP server SHALL expose a `kb_bulk_delete` tool with parameters: `document_ids` (optional list of int), `tags` (optional list of str), `doc_type` (optional str), `from_id` (optional int), `to_id` (optional int), `force` (optional bool).
+
+The tool SHALL call `POST /api/v1/bulk/delete` on the engine via the engine client and return the JSON response.
+
+The tool description SHALL clearly state that `tags` is a selection filter (which documents to delete), not tags to delete.
+
+#### Scenario: MCP bulk delete by tag
+
+- **WHEN** `kb_bulk_delete(tags=["old"])` is called
+- **THEN** the engine client SHALL send `POST /api/v1/bulk/delete` with `{"tags": ["old"]}`
+- **AND** the tool SHALL return the engine's JSON response
+
+### Requirement: MCP bulk tags tool
+
+The MCP server SHALL expose a `kb_bulk_tags` tool with parameters: `document_ids`, `tags`, `doc_type`, `from_id`, `to_id` (selection filters), plus `add` (optional list of str), `remove` (optional list of str), and `force` (optional bool).
+
+The tool description SHALL clearly distinguish `tags` (selection filter) from `add`/`remove` (tag changes to apply).
+
+#### Scenario: MCP bulk tag update
+
+- **WHEN** `kb_bulk_tags(tags=["agent:mybot"], add=["reviewed"], remove=["draft"])` is called
+- **THEN** the engine client SHALL send the appropriate `POST /api/v1/bulk/tags` request
+
+### Requirement: MCP bulk set-tags tool
+
+The MCP server SHALL expose a `kb_bulk_set_tags` tool with parameters: `document_ids`, `tags`, `doc_type`, `from_id`, `to_id` (selection filters), plus `new_tags` (list of str) and `force` (optional bool).
+
+#### Scenario: MCP bulk set tags
+
+- **WHEN** `kb_bulk_set_tags(doc_type="note", new_tags=["clean"])` is called
+- **THEN** the engine client SHALL send `POST /api/v1/bulk/set-tags` with `{"doc_type": "note", "new_tags": ["clean"]}`
+
+### Requirement: MCP engine client bulk methods
+
+The MCP engine client (`mcp/engine.py`) SHALL provide three new methods:
+
+- `bulk_delete(document_ids?, tags?, doc_type?, from_id?, to_id?, force?)` → dict
+- `bulk_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, add?, remove?, force?)` → dict
+- `bulk_set_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, new_tags?, force?)` → dict
+
+Each SHALL send a POST request to the corresponding `/api/v1/bulk/*` endpoint with the parameters as a JSON body. Each SHALL raise on non-2xx status codes, consistent with existing methods.
+
+### Requirement: CLI bulk-remove command
+
+The CLI SHALL expose a `kb bulk-remove` command with flags: `--tags` (comma-separated), `--type`, `--ids` (comma-separated), `--from-id`, `--to-id`, `--force`/`-f`, `--yes`/`-y`.
+
+Without `--yes`, the CLI SHALL first display the match count and ask for interactive confirmation before proceeding.
+
+The command SHALL call `POST /api/v1/bulk/delete` with the constructed filter.
+
+#### Scenario: CLI bulk remove with confirmation
+
+- **WHEN** `kb bulk-remove --tags "draft,old" --type note` is run without `--yes`
+- **THEN** the CLI SHALL display "This will delete N documents matching: tags=[draft,old] type=note" and prompt "Proceed? [y/N]"
+
+#### Scenario: CLI bulk remove with --yes
+
+- **WHEN** `kb bulk-remove --tags "draft" --yes` is run
+- **THEN** the CLI SHALL proceed without prompting
+
+### Requirement: CLI bulk-tag command
+
+The CLI SHALL expose a `kb bulk-tag` command with the same filter flags as `bulk-remove`, plus `--add` and `--remove` (comma-separated tag lists).
+
+The command SHALL call `POST /api/v1/bulk/tags` with the constructed filter and tag changes.
+
+### Requirement: CLI bulk-set-tags command
+
+The CLI SHALL expose a `kb bulk-set-tags` command with the filter flags, plus `--set` (comma-separated list of replacement tags).
+
+The command SHALL call `POST /api/v1/bulk/set-tags` with the constructed filter and `new_tags`.
@@ -10,7 +10,7 @@ DEVELOPER.md SHALL contain instructions for building both the engine and client

 #### Scenario: Engine build from source
 - **WHEN** a developer reads DEVELOPER.md
- **THEN** it SHALL include instructions for starting the engine from source using compose files (both NVIDIA and ROCm)
+- **THEN** it SHALL include instructions for starting the engine from source using compose files (NVIDIA and CPU)

 #### Scenario: Client build from source
 - **WHEN** a developer reads DEVELOPER.md
@@ -31,13 +31,6 @@ DEVELOPER.md SHALL document the release process for both client and engine, incl
 - **WHEN** a developer reads DEVELOPER.md
 - **THEN** it SHALL include how to check client and engine versions

-### Requirement: DEVELOPER.md contains developer notes
-DEVELOPER.md SHALL include any forward-looking developer notes such as migration plans or technical debt items.
-
-#### Scenario: ROCm migration note
- **WHEN** a developer reads DEVELOPER.md
- **THEN** it SHALL include the ROCm runtime migration note about onnxruntime and MIGraphX
-
 ### Requirement: README.md excludes developer-only content
 README.md SHALL NOT contain build-from-source instructions, release processes, or developer-only notes.

@@ -49,10 +42,6 @@ README.md SHALL NOT contain build-from-source instructions, release processes, o
 - **WHEN** a user reads README.md
 - **THEN** there SHALL be no "Building and releasing" section

-#### Scenario: No developer notes in README
- **WHEN** a user reads README.md
- **THEN** there SHALL be no "Future: ROCm runtime migration" section
-
 ### Requirement: README.md cross-references DEVELOPER.md
 README.md SHALL include a link to DEVELOPER.md for users who want to build from source or contribute.

@@ -2,7 +2,7 @@

 ## Purpose

-Docker deployment provides containerized packaging of the knowledge base engine with GPU support for NVIDIA and AMD platforms, along with Compose files for single-command deployment.
+Docker deployment provides containerized packaging of the knowledge base engine with GPU support for NVIDIA, along with Compose files for single-command deployment.

 ## Requirements

@@ -20,26 +20,12 @@ The project SHALL provide a `Dockerfile.nvidia` that builds the engine on an NVI

 ---

-### Requirement: AMD ROCm Docker image
-
-The project SHALL provide a `Dockerfile.rocm` that builds the engine on an AMD ROCm base image with GPU support for PyTorch and ONNX Runtime.
-
-#### Scenario: Build ROCm image
- **WHEN** an admin runs `docker compose -f compose.rocm.yaml build`
- **THEN** the build SHALL produce a working image with ROCm runtime, PyTorch with ROCm support, onnxruntime-rocm, and all engine dependencies
-
-#### Scenario: GPU access in ROCm container
- **WHEN** the ROCm container starts with `--device=/dev/kfd --device=/dev/dri`
- **THEN** `torch.cuda.is_available()` SHALL return True (via HIP) and the engine SHALL load the embedding model on GPU
-
---
-
 ### Requirement: Application code is GPU-vendor-agnostic

-The Python engine code SHALL NOT reference CUDA or ROCm directly. GPU vendor abstraction SHALL be handled entirely at the Docker image level (base image selection and pip package choice). The same application code SHALL run on both NVIDIA and AMD images without modification.
+The Python engine code SHALL NOT reference CUDA directly. GPU abstraction SHALL be handled at the Docker image level (base image selection and pip package choice). The same application code SHALL run on both NVIDIA and CPU images without modification.

 #### Scenario: Same engine code on both platforms
- **WHEN** the engine starts on an NVIDIA image and an AMD image with identical configuration
+- **WHEN** the engine starts on an NVIDIA image and a CPU image with identical configuration
 - **THEN** both SHALL load the model, accept requests, and return identical search results for the same query and data

 ---
@@ -59,10 +45,6 @@ The engine SHALL store all persistent state (SQLite database, HF model cache, st
 - **WHEN** an admin copies the data directory from Host A to Host B and starts the engine with the same bind mount path
 - **THEN** the engine SHALL start successfully and serve all previously ingested documents without reprocessing

-#### Scenario: Portable data across GPU vendors
- **WHEN** an admin moves the data directory from an NVIDIA host to an AMD host (same model name)
- **THEN** the engine SHALL start successfully. Embeddings in the database remain valid (they are model-specific, not GPU-vendor-specific)
-
 ---

 ### Requirement: Compose files for deployment
@@ -73,21 +55,51 @@ The project SHALL provide Docker Compose files for single-command deployment. Co
 - **WHEN** an admin runs `docker compose -f compose.nvidia.yaml up -d`
 - **THEN** the engine SHALL start with GPU access, bind-mount the data directory, and be reachable on the configured port

-#### Scenario: Start ROCm deployment
- **WHEN** an admin runs `docker compose -f compose.rocm.yaml up -d`
- **THEN** the engine SHALL start with GPU access via ROCm device passthrough, bind-mount the data directory, and be reachable on the configured port
-
 #### Scenario: Automatic restart
 - **WHEN** the engine process crashes or the host reboots
 - **THEN** Docker SHALL automatically restart the container (restart policy `unless-stopped`)

 #### Scenario: Configure via environment
- **WHEN** an admin sets environment variables in the compose file (KB_MODEL, KB_API_KEY, KB_DEVICE, etc.)
- **THEN** the engine SHALL use those values
+- **WHEN** an admin sets environment variables in the compose file (KB_MODEL, KB_API_KEY, KB_DEVICE, KB_MCP_ALLOWED_HOSTS, etc.)
+- **THEN** the engine and MCP server SHALL use those values

 #### Scenario: Pre-built image deployment
 - **WHEN** an admin wants to use a pre-built engine image without building from source
- **THEN** the engine release notes SHALL include the exact `docker pull` command with the versioned tag (e.g. `docker.dcglab.co.uk/dcg/kb/engine:engine-v2.1.0-nvidia`)
+- **THEN** the engine release notes SHALL include the exact `docker pull` command with the versioned tag (e.g. `docker.dcglab.co.uk/kb/engine:engine-v2.1.0-nvidia`)
+
+#### Scenario: MCP allowed hosts in Compose
+- **WHEN** the kb-mcp service is defined in a Compose file
+- **THEN** the environment block SHALL include `KB_MCP_ALLOWED_HOSTS` with a comment explaining its format and purpose
+
+---
+
+### Requirement: Configurable MCP allowed hosts
+
+The MCP server SHALL accept a `KB_MCP_ALLOWED_HOSTS` environment variable containing a comma-separated list of additional hosts (IP addresses or FQDNs) that are permitted to connect. The server SHALL always allow `127.0.0.1`, `localhost`, and `[::1]` regardless of this setting. DNS rebinding protection SHALL always be enabled.
+
+#### Scenario: Remote client connects with allowed host
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is set to `192.168.1.50` and a client connects with `Host: 192.168.1.50:3000`
+- **THEN** the server SHALL accept the request and process it normally
+
+#### Scenario: Remote client connects with disallowed host
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is set to `192.168.1.50` and a client connects with `Host: 10.0.0.99:3000`
+- **THEN** the server SHALL return HTTP 421 "Invalid Host header"
+
+#### Scenario: Multiple allowed hosts
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is set to `192.168.1.50,kb.example.com`
+- **THEN** the server SHALL accept requests with `Host` matching either `192.168.1.50` or `kb.example.com` on any port
+
+#### Scenario: Variable unset or empty
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is unset or empty
+- **THEN** the server SHALL allow only localhost addresses (`127.0.0.1`, `localhost`, `[::1]`) with any port
+
+#### Scenario: Localhost always allowed
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` is set to `192.168.1.50`
+- **THEN** the server SHALL still accept requests with `Host: localhost:3000` or `Host: 127.0.0.1:3000`
+
+#### Scenario: Allowed origins derived from allowed hosts
+- **WHEN** `KB_MCP_ALLOWED_HOSTS` includes `192.168.1.50`
+- **THEN** the server SHALL accept `Origin: http://192.168.1.50:3000` (and any port) in addition to localhost origins

 ---

@@ -96,7 +108,7 @@ The project SHALL provide Docker Compose files for single-command deployment. Co
 The Dockerfiles SHALL produce images that work without GPU access. If no GPU is available, the engine SHALL fall back to CPU for all operations.

 #### Scenario: No GPU available
- **WHEN** the container starts without GPU passthrough (no `--gpus`, no `/dev/kfd`)
+- **WHEN** the container starts without GPU passthrough (no `--gpus`)
 - **THEN** the engine SHALL detect no GPU, load the model on CPU, and log a warning that GPU acceleration is unavailable

 #### Scenario: Explicit CPU mode
@@ -26,11 +26,15 @@ The engine SHALL load the embedding model eagerly at startup before accepting HT

 ### Requirement: Hybrid search

-The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5) and vector similarity search (via sqlite-vec), merged using Reciprocal Rank Fusion. Search SHALL complete in under 100ms when the model is warm. The engine SHALL sanitize user query strings to prevent FTS5 syntax errors for any input.
+The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5) and vector similarity search (via sqlite-vec), merged using Reciprocal Rank Fusion with a top-rank bonus (+0.05 for rank 1, +0.02 for ranks 2-3 in either arm) that preserves exact matches. Search SHALL complete in under 100ms when the model is warm and reranking is disabled; reranked searches SHALL complete in under 500ms on GPU. The engine SHALL sanitize user query strings to prevent FTS5 syntax errors for any input.

 #### Scenario: Hybrid search with results
 - **WHEN** a client sends `POST /api/v1/search` with body `{"query": "how to change oil", "top": 5}`
- **THEN** the engine SHALL embed the query using the resident model, run both FTS5 and vector searches, merge results via RRF, and return a JSON response with matched chunks including scores, document metadata, and tags
+- **THEN** the engine SHALL embed the query using the resident model, run both FTS5 and vector searches, merge results via RRF with top-rank bonus, and return a JSON response with matched chunks including scores, `document_id`, document metadata, tags, and `tag_contexts`
+
+#### Scenario: Explain traces
+- **WHEN** a client sends `POST /api/v1/search` with `"explain": true`
+- **THEN** each result SHALL include an `explain` object with per-arm raw scores and ranks (`fts_score`, `fts_rank`, `vec_score`, `vec_rank`), RRF contributions (`rrf_fts`, `rrf_vec`), the top-rank `bonus`, rerank blend fields when reranking ran (`pre_rerank_rank`, `retrieval_norm`, `rerank_score`, `blend_weight`), and the `final_score`; fields for an arm that did not match SHALL be null

 #### Scenario: Search with filters
 - **WHEN** a client sends `POST /api/v1/search` with body `{"query": "brakes", "tags": ["maintenance"], "doc_type": "pdf", "top": 3}`
@@ -62,6 +66,28 @@ The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5

 ---

+### Requirement: Cross-encoder reranking
+
+The engine SHALL support optional server-side reranking of hybrid search results using a local cross-encoder model, enabled via `KB_RERANK_ENABLED` (default false) with the model set by `KB_RERANKER_MODEL` (default `BAAI/bge-reranker-v2-m3`). When active, the engine SHALL over-fetch candidates (`KB_RERANK_CANDIDATES`, default 40), score each (query, chunk) pair, and blend retrieval and rerank scores position-aware: 75% retrieval weight for pre-rerank ranks 1-3, 60% for 4-10, 40% for 11+, with retrieval scores min-max normalised over the candidate set. Blended scores are on a 0-1 scale distinct from RRF scores; the score threshold SHALL be applied before reranking only. Every search response SHALL include a top-level `"reranked"` boolean.
+
+#### Scenario: Reranked search
+- **WHEN** reranking is enabled with a loaded model and a client sends a hybrid search
+- **THEN** the engine SHALL rerank the top candidates and return results ordered by blended score with `"reranked": true`
+
+#### Scenario: Per-request opt-out
+- **WHEN** a client sends `POST /api/v1/search` with `"rerank": false`
+- **THEN** the engine SHALL skip reranking and return plain hybrid results with `"reranked": false`
+
+#### Scenario: Graceful degradation
+- **WHEN** reranking is requested but the model is disabled or failed to load
+- **THEN** the engine SHALL return plain hybrid results with `"reranked": false` and no error
+
+#### Scenario: Single-arm searches never rerank
+- **WHEN** a client sends a search with `fts_only` or `vec_only` set
+- **THEN** the engine SHALL NOT rerank, keeping single-arm results pure for benchmarking
+
+---
+
 ### Requirement: Async ingestion via job queue

 The engine SHALL accept file uploads and text notes for ingestion asynchronously. Uploaded content SHALL be written to a staging area and a job record created in the database. The engine SHALL return HTTP 202 immediately. A background worker SHALL process queued jobs sequentially. Before staging, the engine SHALL compute a SHA256 hash of the uploaded content and reject duplicates immediately.
@@ -128,7 +154,7 @@ The engine SHALL maintain job records in SQLite with status tracking. Jobs SHALL

 ### Requirement: Background ingestion worker

-The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.
+The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes, fixed-size text chunking for data files with minified JSON pretty-printed first), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.

 #### Scenario: Successful PDF ingestion
 - **WHEN** the background worker picks up a queued PDF job
@@ -150,15 +176,19 @@ The engine SHALL provide endpoints to list, inspect, remove, and download origin

 #### Scenario: List documents
 - **WHEN** a client sends `GET /api/v1/documents`
- **THEN** the engine SHALL return a JSON array of documents with id, title, doc_type, tags, chunk_count, and created_at
+- **THEN** the engine SHALL return a JSON array of documents with id, title, doc_type, tags, chunk_count, created_at, and updated_at

 #### Scenario: List documents with filters
 - **WHEN** a client sends `GET /api/v1/documents?type=pdf&tags=manual`
 - **THEN** the engine SHALL return only documents matching all specified filters

+#### Scenario: List documents sorted by most recent
+- **WHEN** a client requests documents sorted by date
+- **THEN** the engine SHALL use `COALESCE(updated_at, created_at)` for ordering, so un-mutated documents sort by creation time and mutated documents sort by their last update
+
 #### Scenario: Get document details
 - **WHEN** a client sends `GET /api/v1/documents/{id}`
- **THEN** the engine SHALL return the full document record including all chunks, their text content, and whether the original file is available (`has_file: true/false`)
+- **THEN** the engine SHALL return the full document record including all chunks, their text content, `updated_at`, and whether the original file is available (`has_file: true/false`)

 #### Scenario: Download original file
 - **WHEN** a client sends `GET /api/v1/documents/{id}/file`
@@ -174,13 +204,45 @@ The engine SHALL provide endpoints to list, inspect, remove, and download origin

 ---

+### Requirement: Note mutation endpoint
+
+The engine SHALL provide a `PATCH /api/v1/notes/{id}` endpoint for updating existing notes in place. See the `note-mutation` spec for full details.
+
+#### Scenario: Note update endpoint exists
+- **WHEN** a client sends `PATCH /api/v1/notes/42` with body `{"text": "new content"}`
+- **THEN** the engine SHALL process the update synchronously and return the updated document
+
+---
+
+### Requirement: Document updated_at tracking
+
+The engine SHALL track when documents are modified via an `updated_at` column. This column SHALL be NULL for documents that have never been updated.
+
+#### Scenario: New document has no updated_at
+- **WHEN** a document is first ingested
+- **THEN** `updated_at` SHALL be NULL and `created_at` SHALL be set to the ingestion timestamp
+
+#### Scenario: Note update sets updated_at
+- **WHEN** a note is updated via `PATCH /api/v1/notes/{id}`
+- **THEN** `updated_at` SHALL be set to the current timestamp
+
+#### Scenario: Tag change sets updated_at
+- **WHEN** tags are modified via `PUT /api/v1/documents/{id}/tags`
+- **THEN** `updated_at` SHALL be set to the current timestamp
+
+#### Scenario: Schema migration for updated_at
+- **WHEN** the engine starts against a v2 database without an `updated_at` column
+- **THEN** the engine SHALL automatically add `ALTER TABLE documents ADD COLUMN updated_at TEXT` and all existing documents SHALL have `updated_at = NULL`
+
+---
+
 ### Requirement: Tag management

 The engine SHALL provide endpoints to list all tags and manage tags on documents.

 #### Scenario: List all tags
 - **WHEN** a client sends `GET /api/v1/tags`
- **THEN** the engine SHALL return a JSON array of tags with name and document count
+- **THEN** the engine SHALL return a JSON array of tags with name, document count, and description (null when unset)

 #### Scenario: Add tags to a document
 - **WHEN** a client sends `PUT /api/v1/documents/{id}/tags` with body `{"add": ["manual", "v2"]}`
@@ -192,13 +254,35 @@ The engine SHALL provide endpoints to list all tags and manage tags on documents

 ---

+### Requirement: Tag context descriptions
+
+The engine SHALL support a one-line context description per tag, stored in a `description` column on the tags table (added via idempotent migration). Search results SHALL include a `tag_contexts` object mapping each of the document's described tags to its description, so consumers can judge which similar-scoring chunks answer the question.
+
+#### Scenario: Set a tag description
+- **WHEN** a client sends `PUT /api/v1/tags/{name}/description` with body `{"description": "Lab operations runbooks"}`
+- **THEN** the engine SHALL store the description (matching the tag name case-insensitively) and return `{"name": "<name>", "description": "<description>"}`
+
+#### Scenario: Clear a tag description
+- **WHEN** a client sends `PUT /api/v1/tags/{name}/description` with a null or empty description
+- **THEN** the engine SHALL clear the stored description
+
+#### Scenario: Unknown tag
+- **WHEN** a client sets a description for a tag that does not exist
+- **THEN** the engine SHALL return HTTP 404
+
+#### Scenario: Descriptions in search results
+- **WHEN** a search result's document carries tags and at least one tag has a description
+- **THEN** the result SHALL include `tag_contexts` with only the described tags; results with no described tags SHALL include an empty `tag_contexts` object
+
+---
+
 ### Requirement: Engine status and reindex

 The engine SHALL provide status information and support re-embedding all chunks. The `version` field in the status response SHALL always be present and SHALL reflect the engine's release version as read from the `VERSION` file. This field is the contract used by clients for compatibility checking.

 #### Scenario: Get engine status
 - **WHEN** a client sends `GET /api/v1/status`
- **THEN** the engine SHALL return JSON with `version` (string, from VERSION file), model_name, embedding_dim, GPU device info, database stats (document count by type, total chunks, DB size), and queue stats (queued/processing job count)
+- **THEN** the engine SHALL return JSON with `version` (string, from VERSION file), model_name, embedding_dim, GPU device info, database stats (document count by type, total chunks, DB size), queue stats (queued/processing job count), and a `rerank` object with `enabled`, `model`, `loaded`, and `candidates`

 #### Scenario: Trigger reindex
 - **WHEN** a client sends `POST /api/v1/reindex`
@@ -265,17 +265,43 @@ The client SHALL provide a `kb reindex` command that triggers re-embedding of al

 ---

+### Requirement: Update note command
+
+The client SHALL provide a `kb updatenote <id> <text>` command that updates an existing note's content via the engine's `PATCH /api/v1/notes/{id}` endpoint.
+
+#### Scenario: Update a note
+- **WHEN** the user runs `kb updatenote 42 "Updated note content"`
+- **THEN** the client SHALL send `PATCH /api/v1/notes/42` with body `{"text": "Updated note content"}` and display the result
+
+#### Scenario: Update a note with JSON output
+- **WHEN** the user runs `kb updatenote 42 "new content" --format json`
+- **THEN** the client SHALL output the raw JSON response from the engine
+
+#### Scenario: Update a non-existent document
+- **WHEN** the user runs `kb updatenote 999 "text"` and the engine returns HTTP 404
+- **THEN** the client SHALL display an error indicating the document was not found and exit with a non-zero code
+
+#### Scenario: Update a non-note document
+- **WHEN** the user runs `kb updatenote 42 "text"` and the engine returns HTTP 422
+- **THEN** the client SHALL display an error indicating that only notes can be updated and exit with a non-zero code
+
+#### Scenario: Missing arguments
+- **WHEN** the user runs `kb updatenote` or `kb updatenote 42` with insufficient arguments
+- **THEN** the client SHALL display usage help indicating that both document ID and text are required
+
+---
+
 ### Requirement: Engine version compatibility check

 The client SHALL verify that the connected engine meets a minimum version requirement before executing any API command. The minimum required engine version SHALL be embedded in the client binary at build time. If the engine version is below the minimum, the client SHALL print an error message and exit with a non-zero code. There SHALL be no flag to skip or suppress this check.

 #### Scenario: Compatible engine version
- **WHEN** the client connects to an engine reporting version `2.1.5` and `MinEngineVersion` is `2.1.0`
+- **WHEN** the client connects to an engine reporting version `3.0.0` and `MinEngineVersion` is `3.0.0`
 - **THEN** the client SHALL proceed with the command normally

 #### Scenario: Incompatible engine version
- **WHEN** the client connects to an engine reporting version `2.0.3` and `MinEngineVersion` is `2.1.0`
- **THEN** the client SHALL print to stderr: `Error: kb client vX.Y.Z requires engine v2.1.0+ (connected engine is v2.0.3)` followed by an upgrade hint, and exit with code 1
+- **WHEN** the client connects to an engine reporting version `2.1.0` and `MinEngineVersion` is `3.0.0`
+- **THEN** the client SHALL print to stderr: `Error: kb client vX.Y.Z requires engine v3.0.0+ (connected engine is v2.1.0)` followed by an upgrade hint, and exit with code 1

 #### Scenario: Engine unreachable during version check
 - **WHEN** the client cannot reach the engine's `/api/v1/status` endpoint
@@ -0,0 +1,198 @@
+# MCP Server
+
+## Purpose
+
+The MCP server provides a Model Context Protocol interface to the kb engine, exposing knowledge base operations as native MCP tools over Streamable HTTP transport. It runs as a separate Docker container alongside the engine, translating MCP tool calls into engine HTTP API calls.
+
+## Requirements
+
+### Requirement: MCP server transport and deployment
+
+The MCP server SHALL expose tools via Streamable HTTP transport. It SHALL run as a Docker container, configured to connect to the kb engine's HTTP API. It SHALL read `KB_ENGINE_URL` and `KB_API_KEY` from environment variables to connect to the engine.
+
+#### Scenario: MCP server starts and connects to engine
+- **WHEN** the MCP server container starts with `KB_ENGINE_URL=http://engine:8000` and `KB_API_KEY=secret`
+- **THEN** it SHALL begin accepting MCP connections over Streamable HTTP and use the configured URL and API key for all engine API calls
+
+#### Scenario: Engine unreachable at startup
+- **WHEN** the MCP server starts but cannot reach the engine at `KB_ENGINE_URL`
+- **THEN** it SHALL start and accept connections, but tool calls SHALL return errors indicating the engine is unreachable
+
+#### Scenario: Docker Compose deployment
+- **WHEN** the MCP server is deployed via Docker Compose alongside the engine
+- **THEN** it SHALL connect to the engine via the Docker network using the service name (e.g. `http://engine:8000`)
+
+---
+
+### Requirement: MCP server authentication
+
+The MCP server SHALL require Bearer token authentication from calling agents via the `KB_MCP_API_KEY` environment variable. This is independent of the engine's `KB_API_KEY`.
+
+#### Scenario: Valid MCP API key
+- **WHEN** `KB_MCP_API_KEY` is set and a calling agent provides a matching Bearer token
+- **THEN** the MCP server SHALL process the request normally
+
+#### Scenario: Missing MCP API key when required
+- **WHEN** `KB_MCP_API_KEY` is set and a calling agent connects without a Bearer token
+- **THEN** the MCP server SHALL reject the connection with an authentication error
+
+#### Scenario: Invalid MCP API key
+- **WHEN** `KB_MCP_API_KEY` is set and a calling agent provides a non-matching Bearer token
+- **THEN** the MCP server SHALL reject the connection with an authentication error
+
+#### Scenario: MCP auth disabled
+- **WHEN** `KB_MCP_API_KEY` is not set
+- **THEN** the MCP server SHALL accept all connections without authentication
+
+---
+
+### Requirement: Search tool
+
+The MCP server SHALL expose a `kb_search` tool that queries the knowledge base via the engine's search API.
+
+#### Scenario: Basic search
+- **WHEN** an agent calls `kb_search` with `{"query": "pension revaluation", "top": 5}`
+- **THEN** the MCP server SHALL POST to the engine's `/api/v1/search` endpoint and return the results with chunk text, scores, document metadata, and tags
+
+#### Scenario: Search with tag filter
+- **WHEN** an agent calls `kb_search` with `{"query": "email preferences", "tags": ["agent:mybot"]}`
+- **THEN** the MCP server SHALL include the tags in the filter and POST to the engine's search endpoint
+
+#### Scenario: Search with mode override
+- **WHEN** an agent calls `kb_search` with `{"query": "error log", "fts_only": true}`
+- **THEN** the MCP server SHALL pass `fts_only: true` to the engine search endpoint
+
+---
+
+### Requirement: Add note tool
+
+The MCP server SHALL expose a `kb_addnote` tool that submits a text note to the engine for ingestion.
+
+#### Scenario: Add a note
+- **WHEN** an agent calls `kb_addnote` with `{"text": "User prefers concise responses"}`
+- **THEN** the MCP server SHALL submit the note to the engine's `POST /api/v1/jobs` endpoint and return the job ID
+
+#### Scenario: Add a note with tags
+- **WHEN** an agent calls `kb_addnote` with `{"text": "User prefers concise responses", "tags": ["agent:mybot", "feedback"]}`
+- **THEN** the MCP server SHALL submit the note with exactly those tags to the engine
+
+---
+
+### Requirement: Chunked file upload tools
+
+The MCP server SHALL expose a three-step chunked file upload pattern for transferring files from remote agents to the engine.
+
+#### Scenario: Start an upload
+- **WHEN** an agent calls `kb_upload_start` with `{"filename": "report.pdf", "total_size": 5242880, "tags": ["insurance"]}`
+- **THEN** the MCP server SHALL create a staging entry, generate a UUID `upload_id`, and return `{"upload_id": "<uuid>"}`
+
+#### Scenario: Upload a chunk
+- **WHEN** an agent calls `kb_upload_chunk` with `{"upload_id": "<uuid>", "data": "<base64-encoded-data>", "chunk_index": 0}`
+- **THEN** the MCP server SHALL decode the base64 data and write it to the staging area for the given upload
+
+#### Scenario: Upload multiple chunks in sequence
+- **WHEN** an agent calls `kb_upload_chunk` multiple times with sequential `chunk_index` values for the same `upload_id`
+- **THEN** the MCP server SHALL store each chunk and track the sequence
+
+#### Scenario: Finish an upload
+- **WHEN** an agent calls `kb_upload_finish` with `{"upload_id": "<uuid>"}`
+- **THEN** the MCP server SHALL reassemble the chunks in order, forward the complete file as a multipart upload to the engine's `POST /api/v1/jobs` endpoint with the tags from `kb_upload_start`, and return the job ID
+
+#### Scenario: Upload with invalid upload_id
+- **WHEN** an agent calls `kb_upload_chunk` or `kb_upload_finish` with an `upload_id` that does not exist
+- **THEN** the MCP server SHALL return an error indicating the upload ID is not found
+
+#### Scenario: Abandoned upload cleanup
+- **WHEN** an agent starts an upload but does not call `kb_upload_finish` within 10 minutes
+- **THEN** the MCP server SHALL clean up the staged chunks and remove the upload tracking entry
+
+#### Scenario: MCP server restart during upload
+- **WHEN** the MCP server container restarts while an upload is in progress
+- **THEN** the in-progress upload SHALL be lost and the agent SHALL need to restart from `kb_upload_start`
+
+---
+
+### Requirement: Update note tool
+
+The MCP server SHALL expose a `kb_update_note` tool that updates an existing note in place via the engine's note mutation endpoint.
+
+#### Scenario: Update an existing note
+- **WHEN** an agent calls `kb_update_note` with `{"document_id": 42, "text": "Updated preference: user prefers bullet points"}`
+- **THEN** the MCP server SHALL send `PATCH /api/v1/notes/42` to the engine and return the updated document
+
+#### Scenario: Update a non-existent document
+- **WHEN** an agent calls `kb_update_note` with a `document_id` that does not exist
+- **THEN** the MCP server SHALL return an error indicating the document was not found
+
+#### Scenario: Update a non-note document
+- **WHEN** an agent calls `kb_update_note` with a `document_id` that refers to a PDF
+- **THEN** the MCP server SHALL return an error indicating that only notes can be updated
+
+---
+
+### Requirement: Get document tool
+
+The MCP server SHALL expose a `kb_get` tool that retrieves document details from the engine.
+
+#### Scenario: Get by document ID
+- **WHEN** an agent calls `kb_get` with `{"document_id": 42}`
+- **THEN** the MCP server SHALL fetch `GET /api/v1/documents/42` and return the document details with chunks
+
+#### Scenario: Get by source path
+- **WHEN** an agent calls `kb_get` with `{"source_path": "memory/feedback_testing.md"}`
+- **THEN** the MCP server SHALL query the engine's documents endpoint filtered by source path and return matching documents
+
+---
+
+### Requirement: Status tool
+
+The MCP server SHALL expose a `kb_status` tool that returns engine health and statistics.
+
+#### Scenario: Get engine status
+- **WHEN** an agent calls `kb_status` with no parameters
+- **THEN** the MCP server SHALL fetch `GET /api/v1/status` and return engine version, model info, device info, document counts, and queue state
+
+---
+
+### Requirement: Jobs tool
+
+The MCP server SHALL expose a `kb_jobs` tool that returns ingestion job status.
+
+#### Scenario: List recent jobs
+- **WHEN** an agent calls `kb_jobs` with no parameters
+- **THEN** the MCP server SHALL fetch `GET /api/v1/jobs` and return the list of recent jobs
+
+#### Scenario: Filter jobs by status
+- **WHEN** an agent calls `kb_jobs` with `{"status": "failed"}`
+- **THEN** the MCP server SHALL fetch `GET /api/v1/jobs?status=failed` and return matching jobs
+
+---
+
+### Requirement: Delete document tool
+
+The MCP server SHALL expose a `kb_delete` tool that permanently deletes a document from the knowledge base. The tool SHALL accept a `document_id` (required integer). Deletion SHALL remove the document, its chunks, embeddings, tags, and any stored file on disk.
+
+The tool SHALL return a confirmation response including the deleted document's ID and title.
+
+#### Scenario: Successful deletion
+- **WHEN** `kb_delete` is called with `document_id=42`
+- **THEN** the document, its chunks, embeddings, tag associations, and stored file SHALL be deleted
+- **AND** the response SHALL include `"status": "deleted"`, the `document_id`, and the document `title`
+
+#### Scenario: Document not found
+- **WHEN** `kb_delete` is called with a `document_id` that does not exist
+- **THEN** the tool SHALL return an error response indicating the document was not found
+
+---
+
+### Requirement: Tags-only document organisation
+
+The MCP server SHALL NOT maintain any collection abstraction. Documents SHALL be returned as-is from the engine with all tags visible. No tag stripping or collection field injection SHALL occur. Namespace isolation (e.g. separating agent memory from user documents) is achieved via tag conventions communicated through system prompts or tool descriptions.
+
+#### Scenario: Search results show all tags
+- **WHEN** `kb_search` is called and a result has tags `["agent:mybot", "collection:documents", "draft"]`
+- **THEN** all three tags SHALL be returned as-is — no stripping of `collection:*` tags
+
+#### Scenario: Add note with explicit tags only
+- **WHEN** `kb_addnote(text="hello", tags=["agent:mybot", "memory"])` is called
+- **THEN** the note SHALL be created with exactly those two tags — no default tags added
@@ -0,0 +1,43 @@
+# Note Mutation
+
+## Purpose
+
+Note mutation allows existing notes to be updated in place without requiring delete and re-add, preserving document identity (ID, creation timestamp) while updating content, embeddings, and the full-text index.
+
+## Requirements
+
+### Requirement: Note update endpoint
+
+The engine SHALL provide a `PATCH /api/v1/notes/{id}` endpoint that accepts new text for an existing note, re-chunks and re-embeds it, and returns the updated document.
+
+#### Scenario: Update an existing note
+- **WHEN** a client sends `PATCH /api/v1/notes/42` with body `{"text": "Updated note content"}`
+- **THEN** the engine SHALL delete existing chunks and embeddings for document 42, run the new text through the note chunking pipeline, generate embeddings for each chunk, insert new chunks and embeddings, update the document's `content_hash` and `updated_at`, and return the updated document with HTTP 200
+
+#### Scenario: Update preserves document identity
+- **WHEN** a note is updated via PATCH
+- **THEN** the document SHALL retain its original `id` and `created_at` values, and `updated_at` SHALL be set to the current timestamp
+
+#### Scenario: Update with long text that produces multiple chunks
+- **WHEN** a client sends `PATCH /api/v1/notes/42` with text longer than the embedding model's token window
+- **THEN** the engine SHALL chunk the text using the same note chunking pipeline as ingestion, producing multiple chunks, and embed each chunk separately
+
+#### Scenario: Update a non-existent document
+- **WHEN** a client sends `PATCH /api/v1/notes/999` and document 999 does not exist
+- **THEN** the engine SHALL return HTTP 404
+
+#### Scenario: Update a non-note document
+- **WHEN** a client sends `PATCH /api/v1/notes/42` and document 42 has `doc_type = 'pdf'`
+- **THEN** the engine SHALL return HTTP 422 with an error indicating that only notes can be updated via this endpoint
+
+#### Scenario: Embedding failure during update
+- **WHEN** a client sends `PATCH /api/v1/notes/42` but the embedding step fails
+- **THEN** the engine SHALL roll back the entire transaction, preserving the original note content, chunks, and embeddings, and return HTTP 500
+
+#### Scenario: FTS5 index updated on note mutation
+- **WHEN** a note is updated via PATCH
+- **THEN** the FTS5 virtual table SHALL be updated via the existing chunk triggers (`chunks_ad` for deletes, `chunks_ai` for inserts), keeping the full-text index consistent with the new content
+
+#### Scenario: Tags preserved on update
+- **WHEN** a note with tags `["feedback", "collection:memory"]` is updated via PATCH
+- **THEN** the document's tags SHALL be unchanged — only the text content, chunks, and embeddings are replaced
@@ -15,9 +15,17 @@ ENGINE_DIR="$SCRIPT_DIR/engine"
 VERSION_FILE="$ENGINE_DIR/VERSION"

 # Container registry
+#
+# --provenance=false --sbom=false on every build: buildx would otherwise attach
+# attestation manifests, making the image an OCI image index. The Registry v2
+# host at docker.dcglab.co.uk rejects those with a 500 on manifest PUT.
 REGISTRY="${REGISTRY:-docker.dcglab.co.uk}"
-IMAGE_ORG="${IMAGE_ORG:-dcg}"
-IMAGE_BASE="${REGISTRY}/${IMAGE_ORG}/kb"
+IMAGE_ORG="${IMAGE_ORG:-}"
+IMAGE_BASE="${REGISTRY}${IMAGE_ORG:+/${IMAGE_ORG}}/kb"
+
+# Push retries — see push_image() below
+PUSH_RETRIES="${PUSH_RETRIES:-5}"
+PUSH_RETRY_DELAY="${PUSH_RETRY_DELAY:-10}"

 #──────────────────────────────────────────────────────────────────────
 # Parse args
@@ -98,6 +106,36 @@ run() {
    fi
 }

+# Push one image tag, retrying on transient registry failures.
+#
+# The engine images carry a ~5.6GB torch layer. Uploading it intermittently
+# fails with a 502 from the reverse proxy in front of the registry, and a
+# manifest PUT can then fail with a 500 because the blob commit has not yet
+# registered. Both clear on a retry, so a whole release should not be lost to
+# one hiccup. Tune with PUSH_RETRIES / PUSH_RETRY_DELAY.
+push_image() {
+    local image="$1"
+    local attempt=1
+
+    echo "  $ docker push $image"
+    [[ "$DRY_RUN" == true ]] && return 0
+
+    while true; do
+        if docker push "$image"; then
+            return 0
+        fi
+
+        if (( attempt >= PUSH_RETRIES )); then
+            echo "Error: failed to push $image after $PUSH_RETRIES attempts" >&2
+            return 1
+        fi
+
+        echo "    push failed (attempt $attempt/$PUSH_RETRIES) — retrying in ${PUSH_RETRY_DELAY}s"
+        sleep "$PUSH_RETRY_DELAY"
+        attempt=$(( attempt + 1 ))
+    done
+}
+
 #──────────────────────────────────────────────────────────────────────
 # Determine release version
 #──────────────────────────────────────────────────────────────────────
@@ -151,15 +189,12 @@ fi
 echo "==> Building Docker engine images ($VERSION)"

 NVIDIA_IMAGE="${IMAGE_BASE}/engine:${DOCKER_TAG}-nvidia"
-ROCM_IMAGE="${IMAGE_BASE}/engine:${DOCKER_TAG}-rocm"
 CPU_IMAGE="${IMAGE_BASE}/engine:${DOCKER_TAG}-cpu"
 NVIDIA_LATEST="${IMAGE_BASE}/engine:latest-nvidia"
-ROCM_LATEST="${IMAGE_BASE}/engine:latest-rocm"
 CPU_LATEST="${IMAGE_BASE}/engine:latest-cpu"

-run docker build -t "$NVIDIA_IMAGE" -t "$NVIDIA_LATEST" -f "$ENGINE_DIR/Dockerfile.nvidia" "$ENGINE_DIR"
-run docker build -t "$ROCM_IMAGE" -t "$ROCM_LATEST" -f "$ENGINE_DIR/Dockerfile.rocm" "$ENGINE_DIR"
-run docker build -t "$CPU_IMAGE" -t "$CPU_LATEST" -f "$ENGINE_DIR/Dockerfile.cpu" "$ENGINE_DIR"
+run docker build --provenance=false --sbom=false -t "$NVIDIA_IMAGE" -t "$NVIDIA_LATEST" -f "$ENGINE_DIR/Dockerfile.nvidia" "$ENGINE_DIR"
+run docker build --provenance=false --sbom=false -t "$CPU_IMAGE" -t "$CPU_LATEST" -f "$ENGINE_DIR/Dockerfile.cpu" "$ENGINE_DIR"

 echo ""

@@ -174,7 +209,7 @@ if [[ -f "$MCP_DIR/Dockerfile" ]]; then
    MCP_IMAGE="${IMAGE_BASE}/mcp:${DOCKER_TAG}"
    MCP_LATEST="${IMAGE_BASE}/mcp:latest"

-    run docker build -t "$MCP_IMAGE" -t "$MCP_LATEST" -f "$MCP_DIR/Dockerfile" "$MCP_DIR"
+    run docker build --provenance=false --sbom=false -t "$MCP_IMAGE" -t "$MCP_LATEST" -f "$MCP_DIR/Dockerfile" "$MCP_DIR"

    echo ""
 fi
@@ -207,9 +242,6 @@ RELEASE_NOTES="## Docker images
 # NVIDIA GPU
 docker pull ${NVIDIA_IMAGE}

-# AMD GPU (ROCm)
-docker pull ${ROCM_IMAGE}
-
 # CPU only
 docker pull ${CPU_IMAGE}
 \`\`\`
@@ -239,16 +271,14 @@ echo ""
 #──────────────────────────────────────────────────────────────────────
 echo "==> Pushing Docker images to $REGISTRY"

-run docker push "$NVIDIA_IMAGE"
-run docker push "$NVIDIA_LATEST"
-run docker push "$ROCM_IMAGE"
-run docker push "$ROCM_LATEST"
-run docker push "$CPU_IMAGE"
-run docker push "$CPU_LATEST"
+push_image "$NVIDIA_IMAGE"
+push_image "$NVIDIA_LATEST"
+push_image "$CPU_IMAGE"
+push_image "$CPU_LATEST"

 if [[ -n "${MCP_IMAGE:-}" ]]; then
-    run docker push "$MCP_IMAGE"
-    run docker push "$MCP_LATEST"
+    push_image "$MCP_IMAGE"
+    push_image "$MCP_LATEST"
 fi

 echo ""
@@ -256,7 +286,6 @@ echo "==> Release $GIT_TAG complete!"
 echo ""
 echo "    Images:"
 echo "      $NVIDIA_IMAGE"
-echo "      $ROCM_IMAGE"
 echo "      $CPU_IMAGE"
 if [[ -n "${MCP_IMAGE:-}" ]]; then
    echo "      $MCP_IMAGE"
Author	SHA1	Message	Date
steve	a38d77ed23	Bump client version to 3.3.0	2026-08-21 18:37:16 +01:00
steve	f1ed5b6e23	Split the torch layer and drop 7.8GB of orphaned CUDA libs from the CPU image The CPU image was 11.2GB — larger than the CUDA one — and its single 6.72GB torch layer could not be pushed at all: the registry drops any blob upload taking longer than 60s, and that layer needed ~61s. Both problems came from install ordering. Dockerfile.cpu installed the project and sentence-transformers first, which resolved the default CUDA torch and pulled ~2.7GB of nvidia-* wheels, then reinstalled torch from the CPU index. Reinstalling replaces torch but leaves its transitive CUDA dependencies behind, orphaned and unused (torch reports 2.13.0+cpu, cuda.is_available() False). Installing CPU torch first, from the CPU index, means nothing ever requests a CUDA build. Everything else then resolves against the torch already present. CPU image 11.2GB -> 3.43GB largest layer 6.72GB -> 896MB push failed 5/5 at 60s -> succeeds in 8.6s Dockerfile.nvidia gets the same split. Its torch layer is unavoidably large (~2.8GiB compressed, ~52s at current throughput) so it stays the tightest thing in the stack, but splitting the application install off keeps that layer cached and the app layer small. Both now install torch before the source COPYs, so editing application code no longer invalidates the multi-GB layer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 18:33:34 +01:00
steve	3ab8a81c14	Retry image pushes on transient registry failures The engine images carry a ~5.6GB torch layer. Uploading it intermittently fails with a 502 from the reverse proxy in front of the registry, and the manifest PUT that follows can fail with a 500 because the blob commit has not registered yet. Both clear on a retry, but the script pushed each tag exactly once, so a whole release — including a 25GB rebuild — could be lost to one hiccup. push_image() replaces the bare `run docker push` calls and retries up to PUSH_RETRIES times (default 5) with PUSH_RETRY_DELAY seconds between attempts (default 10). Exhausting the retries still returns non-zero so set -e aborts the release rather than reporting a partial push as success. Dry runs make no docker calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 17:30:33 +01:00
steve	739c3ff30c	Disable buildx attestations so pushes to the Registry v2 host succeed buildx attaches provenance/SBOM attestation manifests by default, which makes each built image an OCI image index (application/vnd.oci.image.index.v1+json). docker.dcglab.co.uk rejects those with a 500 on the manifest PUT — blobs upload fine, then the manifest fails, so the push looks like an auth problem when it is not. Passing --provenance=false --sbom=false yields a plain image manifest, which the registry accepts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 17:20:41 +01:00
steve	3151a1b08a	Move container registry to docker.dcglab.co.uk/kb registry.dcglab.co.uk has been decommissioned and replaced by docker.dcglab.co.uk. Images move off the Gitea built-in registry (which never successfully accepted a push) to the new host, published at the root with no org segment: docker.dcglab.co.uk/kb/engine and .../kb/mcp. IMAGE_ORG is now empty by default but still honoured when set, so the documented REGISTRY=ghcr.io IMAGE_ORG=myorg override keeps working. Gitea remains the git forge — the release download URLs in README.md are unchanged and still point at gitea.dcglab.co.uk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 17:15:42 +01:00
steve	5049ba2a2f	Bump engine version to 3.3.0	2026-08-21 09:58:13 +01:00
steve	6dfc13be1d	Add reranking, RRF fusion, bench harness, tag contexts, and data ingestion Implements five of the six enhancements from docs/kb-enhancements-proposal.htm, closing the retrieval-quality gap identified in the qmd review. - Cross-encoder reranking: new kb/reranker.py loads an optional reranking model at startup (KB_RERANK_ENABLED, KB_RERANKER_MODEL, KB_RERANK_CANDIDATES). Search degrades gracefully to plain hybrid retrieval when the model is absent. Exposed via a "rerank" block in /status, a rerank flag on search, and --no-rerank in the CLI. - RRF rank fusion: FTS and vector lists now merge by reciprocal rank fusion with a top-rank bonus, replacing the old score blend. Scores are comparable across queries. - Bench harness and explain traces: kb bench runs a query fixture against each backend and reports precision@k, recall and MRR. --explain returns a per-result score breakdown. - Tag context descriptions: tags carry an optional one-line description (kb tag-describe), returned as tag_contexts with search results. Adds a tags.description column migration. - Structured data ingestion: .json/.yaml/.toml files ingest as text via the new "data" doc type, pretty-printing minified JSON before chunking. Query expansion (proposal item 5) is deliberately left out pending bench results. Requires engine v3.3.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 09:51:33 +01:00
steve	75e4a0cf73	Bump engine version to 3.2.4	2026-05-15 18:37:12 +01:00
steve	753c641e72	Switch container registry to Gitea built-in (gitea.dcglab.co.uk/steve/kb) The standalone Registry v2 host (docker.dcglab.co.uk, briefly registry.dcglab.co.uk) is being scrapped. Move all kb images to Gitea's built-in container registry.	2026-05-15 18:35:20 +01:00
steve	45e2c5ce91	Bump engine version to 3.2.3	2026-05-15 18:22:08 +01:00
steve	e6e91f1d5c	Clarify hybrid semantic + full-text search in MCP descriptions Agents were misreading kb_search as keyword-only because the vector/semantic component was only mentioned in the negative ("fts_only: no vector similarity"). Lead with hybrid semantic + BM25 + RRF in the server instructions, kb_search docstring, and MCP.md so agents recognise it as a vector search tool.	2026-05-15 18:19:42 +01:00
steve	9eccc527ae	Add next-steps.md with UX improvement ideas for kb CLI Captures pain points found while trying to locate an uploaded PDF: kb list silently ignores positional args, kb search results lack document_id, kb info dumps all chunks with no summary mode, and scan-heavy PDFs produce noisy single-char chunk hits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-13 20:34:55 +01:00
steve	d44d11e4fe	Bump engine version to 3.2.2	2026-04-14 21:48:55 +01:00
steve	574370e8d1	Remove AMD ROCm support — CPU and NVIDIA only BREAKING: Remove Dockerfile.rocm, compose.rocm.yaml, and ROCm image build/push from the release pipeline. Remove AMD quick-start and ROCm references from README and DEVELOPER docs. Update docker-deployment and developer-docs specs to reflect CPU + NVIDIA only. The ROCm variant added significant complexity (4.2GB torch wheel, >20GB container) with limited usage. Users on AMD GPUs should stay on engine v3.2.x or switch to CPU mode. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-06 16:39:37 +01:00
steve	17b19999de	Switch nvidia and rocm Dockerfiles from onnxruntime to torch Nvidia: install torch+torchvision from PyTorch cu130 index, drop onnxruntime-gpu. ROCm: use local torch wheel with rocm6.4 index for torchvision, clean up nvidia remnants from the venv. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-06 16:13:41 +01:00
steve	bb78f4ea80	Fix 500 error on notes with slashes in title, bump engine to 3.2.1 Sanitize / and \ in note titles and filenames when writing to the staging directory — a title like "/reset skill" was interpreted as a path separator, causing a FileNotFoundError and a 500 from the jobs endpoint. Also add PRAGMA busy_timeout=5000 to SQLite connections to prevent immediate failure under concurrent write load. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-06 16:12:58 +01:00
steve	223ff2cf5d	Latest changes all archived	2026-04-04 22:50:19 +01:00
steve	e9a282ddb1	Document KB_BULK_SAFETY_PERCENT in README, DEVELOPER, MCP, and SKILL docs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-04 22:43:42 +01:00
steve	b5a203d2aa	Add bulk operations and remove collections abstraction - Add bulk delete, bulk tags, and bulk set-tags engine endpoints (POST /api/v1/bulk/delete, /bulk/tags, /bulk/set-tags) - Filter-based selection: by tags, doc_type, ID list, ID range - Safety threshold (KB_BULK_SAFETY_PERCENT, default 70%) prevents accidental mass operations unless force=true - Synchronous execution with audit trail via jobs table - Add kb_bulk_delete, kb_bulk_tags, kb_bulk_set_tags MCP tools - Add kb bulk-remove, bulk-tag, bulk-set-tags CLI commands - Remove collection abstraction from MCP server (use tags instead) - Remove kb_set_collection MCP tool - Update SKILL.md, MCP.md, README.md documentation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-04 22:34:47 +01:00
steve	0c124c4ab7	Bump engine version to 3.0.1	2026-04-04 12:42:32 +01:00
steve	da5b8435bc	Add configurable allowed hosts for MCP remote access (KB_MCP_ALLOWED_HOSTS) The MCP SDK's DNS rebinding protection rejects remote clients with 421 when the Host header isn't in the allowlist. Add KB_MCP_ALLOWED_HOSTS env var (comma-separated IPs/FQDNs) to configure additional allowed hosts while keeping localhost always permitted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-04 12:39:43 +01:00
steve	e39e00a2c0	Add MCP auth status to kb_status and update server instructions - kb_status now returns authenticated: true/false so clients can verify auth - Server instructions mention Bearer token auth requirement - Add .env, .venv/, test_mcp_client.py to .gitignore Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-04 12:04:12 +01:00
steve	d078af9ad3	Split MCP docs into MCP.md with AI tool setup examples Move MCP server documentation from README into dedicated MCP.md. Add configuration examples for Claude Code, VS Code, Cursor, Windsurf, and JetBrains IDEs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-02 22:03:41 +01:00
steve	b3dce188e1	Fix version check failing on non-200 status responses When the engine returns 401 (auth required) or other non-200 responses, the version check was parsing the error body, getting an empty version string, and fatally exiting. Now skips the check on non-200 responses and lets the actual API call surface the real error. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-02 21:52:24 +01:00
steve	0dc3065979	Update README for v3.0.0 — add MCP server docs, updatenote, fix version refs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-02 21:45:31 +01:00
@@ -1 +1 @@
 .0.0
 .3.0