steve/kb

Files

T

steve 6dfc13be1d Add reranking, RRF fusion, bench harness, tag contexts, and data ingestion

Implements five of the six enhancements from docs/kb-enhancements-proposal.htm,
closing the retrieval-quality gap identified in the qmd review.

- Cross-encoder reranking: new kb/reranker.py loads an optional reranking
  model at startup (KB_RERANK_ENABLED, KB_RERANKER_MODEL,
  KB_RERANK_CANDIDATES). Search degrades gracefully to plain hybrid
  retrieval when the model is absent. Exposed via a "rerank" block in
  /status, a rerank flag on search, and --no-rerank in the CLI.
- RRF rank fusion: FTS and vector lists now merge by reciprocal rank
  fusion with a top-rank bonus, replacing the old score blend. Scores are
  comparable across queries.
- Bench harness and explain traces: kb bench runs a query fixture against
  each backend and reports precision@k, recall and MRR. --explain returns a
  per-result score breakdown.
- Tag context descriptions: tags carry an optional one-line description
  (kb tag-describe), returned as tag_contexts with search results. Adds a
  tags.description column migration.
- Structured data ingestion: .json/.yaml/.toml files ingest as text via the
  new "data" doc type, pretty-printing minified JSON before chunking.

Query expansion (proposal item 5) is deliberately left out pending bench
results. Requires engine v3.3.0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-21 09:51:33 +01:00

2.5 KiB

Raw Blame History

Agent-Side Search Patterns

Purpose

Documents recommended patterns for agent-side query expansion, plus how agent guidance interacts with the engine's optional server-side reranking. These patterns are communicated via MCP tool descriptions.

Requirements

Requirement: Query expansion guidance in tool description

The kb_search MCP tool description SHALL include guidance on query expansion as a recommended pattern for complex queries.

Scenario: Tool description includes expansion pattern

WHEN an agent reads the kb_search tool description
THEN the description SHALL include guidance such as: "For complex queries, consider expanding into 2-3 variant phrasings and calling this tool multiple times, then deduplicating results by chunk_id"

Requirement: Reranking guidance in tool description

The kb_search MCP tool description SHALL describe the engine's server-side reranking behaviour and retain agent-side reranking as a fallback pattern.

Scenario: Tool description covers server-side reranking

WHEN an agent reads the kb_search tool description
THEN the description SHALL state that results are reranked server-side by default when the engine has a reranker enabled, that rerank=False skips it for lower latency, and that kb_status reports whether reranking is active

Scenario: Tool description retains agent-side fallback

WHEN an agent reads the kb_search tool description
THEN the description SHALL include guidance that, when the engine's reranker is disabled, the agent can rerank the returned results using its own judgement of relevance to the original question

Requirement: No external LLM dependency

The engine SHALL NOT require or use any external LLM API for search operations. Query expansion SHALL remain an agent-side concern. Reranking MAY be performed engine-side using a local, opt-in cross-encoder model; it SHALL never depend on an external API.

Scenario: Engine has no external LLM dependency

WHEN the engine is deployed without any ANTHROPIC_API_KEY or similar LLM API configuration
THEN all search operations SHALL function fully, with no degraded results or missing features

Scenario: Reranking is optional and degrades gracefully

WHEN the engine is deployed with KB_RERANK_ENABLED unset or false, or the reranker model fails to load
THEN all search operations SHALL function fully using hybrid retrieval alone, with responses reporting "reranked": false

2.5 KiB Raw Blame History