Add reranking, RRF fusion, bench harness, tag contexts, and data ingestion
Implements five of the six enhancements from docs/kb-enhancements-proposal.htm, closing the retrieval-quality gap identified in the qmd review. - Cross-encoder reranking: new kb/reranker.py loads an optional reranking model at startup (KB_RERANK_ENABLED, KB_RERANKER_MODEL, KB_RERANK_CANDIDATES). Search degrades gracefully to plain hybrid retrieval when the model is absent. Exposed via a "rerank" block in /status, a rerank flag on search, and --no-rerank in the CLI. - RRF rank fusion: FTS and vector lists now merge by reciprocal rank fusion with a top-rank bonus, replacing the old score blend. Scores are comparable across queries. - Bench harness and explain traces: kb bench runs a query fixture against each backend and reports precision@k, recall and MRR. --explain returns a per-result score breakdown. - Tag context descriptions: tags carry an optional one-line description (kb tag-describe), returned as tag_contexts with search results. Adds a tags.description column migration. - Structured data ingestion: .json/.yaml/.toml files ingest as text via the new "data" doc type, pretty-printing minified JSON before chunking. Query expansion (proposal item 5) is deliberately left out pending bench results. Requires engine v3.3.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -40,6 +40,18 @@ async def lifespan(app: FastAPI):
|
||||
init_schema(conn, model_dim)
|
||||
conn.close()
|
||||
|
||||
# Optional reranker — search degrades gracefully if this fails
|
||||
if cfg.rerank_enabled:
|
||||
from kb.reranker import load_reranker
|
||||
try:
|
||||
load_reranker(cfg.reranker_model, cfg.device)
|
||||
except Exception:
|
||||
log.warning(
|
||||
"Failed to load reranker '%s' — searches will not be reranked",
|
||||
cfg.reranker_model,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
# Start background ingestion worker
|
||||
worker_task = asyncio.create_task(ingestion_worker())
|
||||
|
||||
|
||||
Reference in New Issue
Block a user