Add reranking, RRF fusion, bench harness, tag contexts, and data ingestion

Implements five of the six enhancements from docs/kb-enhancements-proposal.htm,
closing the retrieval-quality gap identified in the qmd review.

- Cross-encoder reranking: new kb/reranker.py loads an optional reranking
  model at startup (KB_RERANK_ENABLED, KB_RERANKER_MODEL,
  KB_RERANK_CANDIDATES). Search degrades gracefully to plain hybrid
  retrieval when the model is absent. Exposed via a "rerank" block in
  /status, a rerank flag on search, and --no-rerank in the CLI.
- RRF rank fusion: FTS and vector lists now merge by reciprocal rank
  fusion with a top-rank bonus, replacing the old score blend. Scores are
  comparable across queries.
- Bench harness and explain traces: kb bench runs a query fixture against
  each backend and reports precision@k, recall and MRR. --explain returns a
  per-result score breakdown.
- Tag context descriptions: tags carry an optional one-line description
  (kb tag-describe), returned as tag_contexts with search results. Adds a
  tags.description column migration.
- Structured data ingestion: .json/.yaml/.toml files ingest as text via the
  new "data" doc type, pretty-printing minified JSON before chunking.

Query expansion (proposal item 5) is deliberately left out pending bench
results. Requires engine v3.3.0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

mcp/engine.py

@@ -18,7 +18,8 @@ def _client() -> httpx.Client:
 
 def search(query: str, top: int = 10, tags: list[str] | None = None,
            doc_type: str | None = None, fts_only: bool = False,
-           vec_only: bool = False, threshold: float | None = None) -> dict:
+           vec_only: bool = False, threshold: float | None = None,
+           explain: bool = False, rerank: bool | None = None) -> dict:
     body: dict = {"query": query, "top": top}
     if tags:
         body["tags"] = tags
@@ -30,6 +31,10 @@ def search(query: str, top: int = 10, tags: list[str] | None = None,
         body["vec_only"] = True
     if threshold is not None:
         body["threshold"] = threshold
+    if explain:
+        body["explain"] = True
+    if rerank is not None:
+        body["rerank"] = rerank
     with _client() as c:
         r = c.post("/api/v1/search", json=body)
         r.raise_for_status()

mcp/server.py

@@ -48,9 +48,11 @@ mcp = FastMCP(
         "kb_search uses dense vector embeddings (semantic similarity) fused with "
         "BM25 full-text ranking, so it finds conceptually related content even "
         "when the exact words don't match — agents can ask natural-language "
-        "questions rather than guessing keywords. Also provides tools for adding "
-        "notes, uploading files, and managing documents and tags. Use tags to "
-        "organise and filter documents (e.g. tag notes with 'agent:mybot' and "
+        "questions rather than guessing keywords. When the engine has a "
+        "cross-encoder reranker enabled, results are reranked server-side by "
+        "default (pass rerank=False for lower latency). Also provides tools for "
+        "adding notes, uploading files, and managing documents and tags. Use tags "
+        "to organise and filter documents (e.g. tag notes with 'agent:mybot' and "
         "filter searches by that tag). This server requires Bearer token "
         "authentication — all requests are authenticated via the Authorization "
         "header at the HTTP transport layer."
@@ -66,6 +68,8 @@ async def kb_search(
     tags: list[str] | None = None,
     doc_type: str | None = None,
     fts_only: bool = False,
+    explain: bool = False,
+    rerank: bool | None = None,
 ) -> str:
     """Hybrid semantic (vector) + full-text search over the knowledge base.
 
@@ -75,6 +79,11 @@ async def kb_search(
     ask natural-language questions ("what did we decide about X?") rather than
     guessing the exact keywords used in the source documents.
 
+    When the engine has a cross-encoder reranker enabled, the top candidates
+    are reranked server-side by default — you normally do NOT need to rerank
+    results yourself. Check kb_status's "rerank" block to see whether it is
+    active.
+
     Returns ranked chunks matching the query, with text content, relevance
     scores, and document metadata.
 
@@ -82,18 +91,25 @@ async def kb_search(
         query: The search query — a natural language question or keywords.
         top: Maximum number of results to return (default 10).
         tags: Filter results to documents with ALL of these tags.
-        doc_type: Filter by document type (e.g. "note", "pdf", "markdown", "code").
+        doc_type: Filter by document type (e.g. "note", "pdf", "markdown",
+            "code", "data").
         fts_only: Disable the vector/semantic component and use only BM25
             keyword matching. Default false (hybrid mode). Set true only when
             you need exact-string matching (e.g. an error code, identifier).
+        explain: Include a per-result score breakdown (BM25 score/rank, vector
+            similarity/rank, rank-fusion contributions, rerank blend) under an
+            "explain" key. Useful for diagnosing why a result ranked where it did.
+        rerank: Set false to skip server-side reranking for lower latency.
+            Default (None) uses the engine's configured behaviour.
 
     Tips for complex queries:
     - Consider expanding into 2-3 variant phrasings and calling this tool multiple
       times, then deduplicating results by chunk_id. For example, search for both
       "pension revaluation rules" and "how are pensions revalued" to cast a wider net.
-    - For precision, rerank the returned results using your own judgement based on
-      relevance to the original question.
-    - Call kb_status to see which embedding model is in use.
+    - If the engine's reranker is disabled, you can still rerank the returned
+      results yourself using your own judgement of relevance to the question.
+    - Call kb_status to see which embedding model is in use and whether
+      server-side reranking is active.
     """
     result = engine.search(
         query=query,
@@ -101,6 +117,8 @@ async def kb_search(
         tags=tags or None,
         doc_type=doc_type,
         fts_only=fts_only,
+        explain=explain,
+        rerank=rerank,
     )
 
     results_list = result if isinstance(result, list) else result.get("results", [])