Add reranking, RRF fusion, bench harness, tag contexts, and data ingestion

Implements five of the six enhancements from docs/kb-enhancements-proposal.htm,
closing the retrieval-quality gap identified in the qmd review.

- Cross-encoder reranking: new kb/reranker.py loads an optional reranking
  model at startup (KB_RERANK_ENABLED, KB_RERANKER_MODEL,
  KB_RERANK_CANDIDATES). Search degrades gracefully to plain hybrid
  retrieval when the model is absent. Exposed via a "rerank" block in
  /status, a rerank flag on search, and --no-rerank in the CLI.
- RRF rank fusion: FTS and vector lists now merge by reciprocal rank
  fusion with a top-rank bonus, replacing the old score blend. Scores are
  comparable across queries.
- Bench harness and explain traces: kb bench runs a query fixture against
  each backend and reports precision@k, recall and MRR. --explain returns a
  per-result score breakdown.
- Tag context descriptions: tags carry an optional one-line description
  (kb tag-describe), returned as tag_contexts with search results. Adds a
  tags.description column migration.
- Structured data ingestion: .json/.yaml/.toml files ingest as text via the
  new "data" doc type, pretty-printing minified JSON before chunking.

Query expansion (proposal item 5) is deliberately left out pending bench
results. Requires engine v3.3.0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

openspec/specs/agent-search-patterns/spec.md

@@ -2,7 +2,7 @@
 
 ## Purpose
 
-Documents recommended patterns for agent-side query expansion and reranking, which are caller responsibilities rather than engine features. These patterns are communicated via MCP tool descriptions.
+Documents recommended patterns for agent-side query expansion, plus how agent guidance interacts with the engine's optional server-side reranking. These patterns are communicated via MCP tool descriptions.
 
 ## Requirements
 
@@ -18,18 +18,26 @@ The `kb_search` MCP tool description SHALL include guidance on query expansion a
 
 ### Requirement: Reranking guidance in tool description
 
-The `kb_search` MCP tool description SHALL include guidance on agent-side reranking as a recommended pattern for improving precision.
+The `kb_search` MCP tool description SHALL describe the engine's server-side reranking behaviour and retain agent-side reranking as a fallback pattern.
 
-#### Scenario: Tool description includes reranking pattern
+#### Scenario: Tool description covers server-side reranking
 - **WHEN** an agent reads the `kb_search` tool description
-- **THEN** the description SHALL include guidance such as: "For precision, rerank the returned results using your own judgement based on relevance to the original question"
+- **THEN** the description SHALL state that results are reranked server-side by default when the engine has a reranker enabled, that `rerank=False` skips it for lower latency, and that `kb_status` reports whether reranking is active
+
+#### Scenario: Tool description retains agent-side fallback
+- **WHEN** an agent reads the `kb_search` tool description
+- **THEN** the description SHALL include guidance that, when the engine's reranker is disabled, the agent can rerank the returned results using its own judgement of relevance to the original question
 
 ---
 
-### Requirement: No engine-side LLM dependency
+### Requirement: No external LLM dependency
 
-The engine SHALL NOT require or use any external LLM API for search operations. Query expansion and reranking SHALL remain entirely agent-side concerns.
+The engine SHALL NOT require or use any external LLM API for search operations. Query expansion SHALL remain an agent-side concern. Reranking MAY be performed engine-side using a local, opt-in cross-encoder model; it SHALL never depend on an external API.
 
-#### Scenario: Engine has no LLM dependency
+#### Scenario: Engine has no external LLM dependency
 - **WHEN** the engine is deployed without any `ANTHROPIC_API_KEY` or similar LLM API configuration
 - **THEN** all search operations SHALL function fully, with no degraded results or missing features
+
+#### Scenario: Reranking is optional and degrades gracefully
+- **WHEN** the engine is deployed with `KB_RERANK_ENABLED` unset or false, or the reranker model fails to load
+- **THEN** all search operations SHALL function fully using hybrid retrieval alone, with responses reporting `"reranked": false`

openspec/specs/engine-api/spec.md

@@ -26,11 +26,15 @@ The engine SHALL load the embedding model eagerly at startup before accepting HT
 
 ### Requirement: Hybrid search
 
-The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5) and vector similarity search (via sqlite-vec), merged using Reciprocal Rank Fusion. Search SHALL complete in under 100ms when the model is warm. The engine SHALL sanitize user query strings to prevent FTS5 syntax errors for any input.
+The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5) and vector similarity search (via sqlite-vec), merged using Reciprocal Rank Fusion with a top-rank bonus (+0.05 for rank 1, +0.02 for ranks 2-3 in either arm) that preserves exact matches. Search SHALL complete in under 100ms when the model is warm and reranking is disabled; reranked searches SHALL complete in under 500ms on GPU. The engine SHALL sanitize user query strings to prevent FTS5 syntax errors for any input.
 
 #### Scenario: Hybrid search with results
 - **WHEN** a client sends `POST /api/v1/search` with body `{"query": "how to change oil", "top": 5}`
-- **THEN** the engine SHALL embed the query using the resident model, run both FTS5 and vector searches, merge results via RRF, and return a JSON response with matched chunks including scores, document metadata, and tags
+- **THEN** the engine SHALL embed the query using the resident model, run both FTS5 and vector searches, merge results via RRF with top-rank bonus, and return a JSON response with matched chunks including scores, `document_id`, document metadata, tags, and `tag_contexts`
+
+#### Scenario: Explain traces
+- **WHEN** a client sends `POST /api/v1/search` with `"explain": true`
+- **THEN** each result SHALL include an `explain` object with per-arm raw scores and ranks (`fts_score`, `fts_rank`, `vec_score`, `vec_rank`), RRF contributions (`rrf_fts`, `rrf_vec`), the top-rank `bonus`, rerank blend fields when reranking ran (`pre_rerank_rank`, `retrieval_norm`, `rerank_score`, `blend_weight`), and the `final_score`; fields for an arm that did not match SHALL be null
 
 #### Scenario: Search with filters
 - **WHEN** a client sends `POST /api/v1/search` with body `{"query": "brakes", "tags": ["maintenance"], "doc_type": "pdf", "top": 3}`
@@ -62,6 +66,28 @@ The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5
 
 ---
 
+### Requirement: Cross-encoder reranking
+
+The engine SHALL support optional server-side reranking of hybrid search results using a local cross-encoder model, enabled via `KB_RERANK_ENABLED` (default false) with the model set by `KB_RERANKER_MODEL` (default `BAAI/bge-reranker-v2-m3`). When active, the engine SHALL over-fetch candidates (`KB_RERANK_CANDIDATES`, default 40), score each (query, chunk) pair, and blend retrieval and rerank scores position-aware: 75% retrieval weight for pre-rerank ranks 1-3, 60% for 4-10, 40% for 11+, with retrieval scores min-max normalised over the candidate set. Blended scores are on a 0-1 scale distinct from RRF scores; the score threshold SHALL be applied before reranking only. Every search response SHALL include a top-level `"reranked"` boolean.
+
+#### Scenario: Reranked search
+- **WHEN** reranking is enabled with a loaded model and a client sends a hybrid search
+- **THEN** the engine SHALL rerank the top candidates and return results ordered by blended score with `"reranked": true`
+
+#### Scenario: Per-request opt-out
+- **WHEN** a client sends `POST /api/v1/search` with `"rerank": false`
+- **THEN** the engine SHALL skip reranking and return plain hybrid results with `"reranked": false`
+
+#### Scenario: Graceful degradation
+- **WHEN** reranking is requested but the model is disabled or failed to load
+- **THEN** the engine SHALL return plain hybrid results with `"reranked": false` and no error
+
+#### Scenario: Single-arm searches never rerank
+- **WHEN** a client sends a search with `fts_only` or `vec_only` set
+- **THEN** the engine SHALL NOT rerank, keeping single-arm results pure for benchmarking
+
+---
+
 ### Requirement: Async ingestion via job queue
 
 The engine SHALL accept file uploads and text notes for ingestion asynchronously. Uploaded content SHALL be written to a staging area and a job record created in the database. The engine SHALL return HTTP 202 immediately. A background worker SHALL process queued jobs sequentially. Before staging, the engine SHALL compute a SHA256 hash of the uploaded content and reject duplicates immediately.
@@ -128,7 +154,7 @@ The engine SHALL maintain job records in SQLite with status tracking. Jobs SHALL
 
 ### Requirement: Background ingestion worker
 
-The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.
+The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes, fixed-size text chunking for data files with minified JSON pretty-printed first), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.
 
 #### Scenario: Successful PDF ingestion
 - **WHEN** the background worker picks up a queued PDF job
@@ -216,7 +242,7 @@ The engine SHALL provide endpoints to list all tags and manage tags on documents
 
 #### Scenario: List all tags
 - **WHEN** a client sends `GET /api/v1/tags`
-- **THEN** the engine SHALL return a JSON array of tags with name and document count
+- **THEN** the engine SHALL return a JSON array of tags with name, document count, and description (null when unset)
 
 #### Scenario: Add tags to a document
 - **WHEN** a client sends `PUT /api/v1/documents/{id}/tags` with body `{"add": ["manual", "v2"]}`
@@ -228,13 +254,35 @@ The engine SHALL provide endpoints to list all tags and manage tags on documents
 
 ---
 
+### Requirement: Tag context descriptions
+
+The engine SHALL support a one-line context description per tag, stored in a `description` column on the tags table (added via idempotent migration). Search results SHALL include a `tag_contexts` object mapping each of the document's described tags to its description, so consumers can judge which similar-scoring chunks answer the question.
+
+#### Scenario: Set a tag description
+- **WHEN** a client sends `PUT /api/v1/tags/{name}/description` with body `{"description": "Lab operations runbooks"}`
+- **THEN** the engine SHALL store the description (matching the tag name case-insensitively) and return `{"name": "<name>", "description": "<description>"}`
+
+#### Scenario: Clear a tag description
+- **WHEN** a client sends `PUT /api/v1/tags/{name}/description` with a null or empty description
+- **THEN** the engine SHALL clear the stored description
+
+#### Scenario: Unknown tag
+- **WHEN** a client sets a description for a tag that does not exist
+- **THEN** the engine SHALL return HTTP 404
+
+#### Scenario: Descriptions in search results
+- **WHEN** a search result's document carries tags and at least one tag has a description
+- **THEN** the result SHALL include `tag_contexts` with only the described tags; results with no described tags SHALL include an empty `tag_contexts` object
+
+---
+
 ### Requirement: Engine status and reindex
 
 The engine SHALL provide status information and support re-embedding all chunks. The `version` field in the status response SHALL always be present and SHALL reflect the engine's release version as read from the `VERSION` file. This field is the contract used by clients for compatibility checking.
 
 #### Scenario: Get engine status
 - **WHEN** a client sends `GET /api/v1/status`
-- **THEN** the engine SHALL return JSON with `version` (string, from VERSION file), model_name, embedding_dim, GPU device info, database stats (document count by type, total chunks, DB size), and queue stats (queued/processing job count)
+- **THEN** the engine SHALL return JSON with `version` (string, from VERSION file), model_name, embedding_dim, GPU device info, database stats (document count by type, total chunks, DB size), queue stats (queued/processing job count), and a `rerank` object with `enabled`, `model`, `loaded`, and `candidates`
 
 #### Scenario: Trigger reindex
 - **WHEN** a client sends `POST /api/v1/reindex`