Add reranking, RRF fusion, bench harness, tag contexts, and data ingestion
Implements five of the six enhancements from docs/kb-enhancements-proposal.htm, closing the retrieval-quality gap identified in the qmd review. - Cross-encoder reranking: new kb/reranker.py loads an optional reranking model at startup (KB_RERANK_ENABLED, KB_RERANKER_MODEL, KB_RERANK_CANDIDATES). Search degrades gracefully to plain hybrid retrieval when the model is absent. Exposed via a "rerank" block in /status, a rerank flag on search, and --no-rerank in the CLI. - RRF rank fusion: FTS and vector lists now merge by reciprocal rank fusion with a top-rank bonus, replacing the old score blend. Scores are comparable across queries. - Bench harness and explain traces: kb bench runs a query fixture against each backend and reports precision@k, recall and MRR. --explain returns a per-result score breakdown. - Tag context descriptions: tags carry an optional one-line description (kb tag-describe), returned as tag_contexts with search results. Adds a tags.description column migration. - Structured data ingestion: .json/.yaml/.toml files ingest as text via the new "data" doc type, pretty-printing minified JSON before chunking. Query expansion (proposal item 5) is deliberately left out pending bench results. Requires engine v3.3.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,128 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>kb-search Enhancements Proposal</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #f7f7f5; --fg: #1a1a1a; --muted: #666; --card: #fff;
|
||||
--border: #ddd; --accent: #2563eb; --code-bg: #eef1f5;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--bg: #16181d; --fg: #e6e6e6; --muted: #9aa0a8; --card: #1e2128;
|
||||
--border: #33363e; --accent: #7aa2f7; --code-bg: #262a33;
|
||||
}
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0; padding: 2rem 1rem 4rem; background: var(--bg); color: var(--fg);
|
||||
font: 16px/1.6 -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||
}
|
||||
main { max-width: 860px; margin: 0 auto; }
|
||||
h1 { font-size: 1.9rem; margin-bottom: .2rem; }
|
||||
h2 { margin-top: 2.2rem; border-bottom: 1px solid var(--border); padding-bottom: .3rem; }
|
||||
.meta { color: var(--muted); font-size: .9rem; margin-bottom: 2rem; }
|
||||
.card {
|
||||
background: var(--card); border: 1px solid var(--border); border-radius: 10px;
|
||||
padding: 1rem 1.3rem; margin: 1rem 0;
|
||||
}
|
||||
.card h3 { margin: .2rem 0 .5rem; }
|
||||
.badge {
|
||||
display: inline-block; font-size: .72rem; font-weight: 600; letter-spacing: .03em;
|
||||
padding: .12rem .55rem; border-radius: 999px; vertical-align: middle; margin-left: .5rem;
|
||||
}
|
||||
.b-high { background: #dc262622; color: #dc2626; }
|
||||
.b-med { background: #d9770622; color: #d97706; }
|
||||
.b-low { background: #05966922; color: #059669; }
|
||||
table { border-collapse: collapse; width: 100%; margin: 1rem 0; font-size: .93rem; }
|
||||
th, td { border: 1px solid var(--border); padding: .45rem .7rem; text-align: left; vertical-align: top; }
|
||||
th { background: var(--code-bg); }
|
||||
code { background: var(--code-bg); padding: .1rem .35rem; border-radius: 4px; font-size: .88em; }
|
||||
pre { background: var(--code-bg); padding: .8rem 1rem; border-radius: 8px; overflow-x: auto; }
|
||||
pre code { background: none; padding: 0; }
|
||||
a { color: var(--accent); }
|
||||
.muted { color: var(--muted); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>kb-search Enhancements Proposal</h1>
|
||||
<p class="meta">Bobby · 2026-07-07 · Prompted by a review of <a href="https://github.com/tobi/qmd">tobi/qmd</a> (Tobi Lütke's local hybrid search engine)</p>
|
||||
|
||||
<h2>Summary</h2>
|
||||
<p>qmd and kb-search v2 solve overlapping problems, but qmd's retrieval pipeline is measurably ahead: its own benchmarks show BM25-only at ~0.50, vector-only at ~0.70, and the full hybrid + reranked pipeline at ~1.00. Our kb does hybrid FTS + vector but stops there — no rank fusion, no reranking, no query expansion, and no way to measure whether a change helps or hurts. This proposal lists five enhancements, ordered by value-for-effort, plus the already-tracked JSON ingestion item.</p>
|
||||
|
||||
<table>
|
||||
<tr><th>#</th><th>Enhancement</th><th>Impact</th><th>Effort</th></tr>
|
||||
<tr><td>1</td><td>LLM reranking stage</td><td>High — biggest single search-quality lever</td><td>Medium</td></tr>
|
||||
<tr><td>2</td><td>RRF fusion for FTS + vector merging</td><td>Medium-high</td><td>Low</td></tr>
|
||||
<tr><td>3</td><td>Bench harness + <code>--explain</code> traces</td><td>High (enables everything else)</td><td>Low-medium</td></tr>
|
||||
<tr><td>4</td><td>Context descriptions on tags/sources</td><td>Medium</td><td>Low</td></tr>
|
||||
<tr><td>5</td><td>Query expansion</td><td>Medium</td><td>Medium-high</td></tr>
|
||||
<tr><td>6</td><td>.json file ingestion (already tracked)</td><td>Medium</td><td>Low</td></tr>
|
||||
</table>
|
||||
|
||||
<h2>Current state</h2>
|
||||
<p>kb-search v2 (engine v3.2.2) runs on the RTX 4070 box with <code>BAAI/bge-base-en-v1.5</code> (768-dim). It holds ~2,310 documents (1,944 PDFs, 237 notes, 129 markdown) in ~123k chunks. Search is hybrid FTS + vector with a blended relative score. Strengths over qmd: binary ingestion (PDF/docx/HTML), tags, ingestion job queue, dedup, original export, and multi-client API access. The proposals below close the retrieval-quality gap without giving any of that up.</p>
|
||||
|
||||
<h2>Proposals</h2>
|
||||
|
||||
<div class="card">
|
||||
<h3>1. LLM reranking stage <span class="badge b-high">HIGH IMPACT</span></h3>
|
||||
<p>Add a cross-encoder reranking pass over the top-K hybrid candidates. qmd uses <code>qwen3-reranker-0.6b</code> (~640MB GGUF) — small enough to sit alongside bge on the 4070 permanently. Flow: hybrid retrieval pulls ~40 candidates → reranker scores each (query, chunk) pair → final order blends retrieval and reranker scores.</p>
|
||||
<p>qmd's position-aware blend is worth copying wholesale: rank 1–3 keep 75% retrieval weight, 4–10 get 60%, 11+ get 40%. This stops the reranker destroying exact-match hits while letting it rescue mid-ranked semantic matches.</p>
|
||||
<p class="muted">API: add <code>rerank: bool</code> (default true) to the search endpoint, with <code>--no-rerank</code> in the CLI for latency-sensitive callers.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>2. RRF fusion <span class="badge b-med">MEDIUM-HIGH</span></h3>
|
||||
<p>Replace the current score blend with Reciprocal Rank Fusion when merging FTS and vector lists: <code>score = Σ 1/(k + rank + 1)</code>, k=60. Rank-based fusion sidesteps the incomparability of BM25 scores (unbounded) and cosine similarity (0–1). qmd adds a top-rank bonus (+0.05 for #1, +0.02 for #2–3 in any list) to preserve exact matches — cheap and effective.</p>
|
||||
<p class="muted">Pure engine-side change, no API impact. Scores become comparable across queries too, which fixes the "score is relative, not absolute" caveat in the current skill docs.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>3. Bench harness + explain traces <span class="badge b-high">DO FIRST</span></h3>
|
||||
<p>We currently have no way to know if any of the above helps. Add:</p>
|
||||
<ul>
|
||||
<li><code>kb bench fixture.json</code> — run a fixture of queries with known-relevant docs, report precision@k / recall / MRR per backend (fts-only, vec-only, hybrid, hybrid+rerank). Directly mirrors <code>qmd bench</code>.</li>
|
||||
<li><code>--explain</code> on search — per-result score breakdown (FTS score, vector score, fusion contribution, rerank score).</li>
|
||||
</ul>
|
||||
<p>A fixture of 20–30 real queries against the existing corpus (lab infra questions, manual lookups, note recall) gives a regression baseline before touching ranking. <strong>This should land before #1 and #2 so their benefit is provable.</strong></p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>4. Context descriptions <span class="badge b-med">MEDIUM</span></h3>
|
||||
<p>qmd's standout idea: attach a one-line description to a collection or path (e.g. "Meeting transcripts", "Lab infrastructure runbooks") and return it with every matching result. For kb, the natural unit is the <strong>tag</strong>: <code>kb tag-describe ops "Lab operations runbooks and procedures"</code>, returned as <code>tag_contexts</code> in search results. Helps an LLM consumer (me) judge which of several similar-scoring chunks actually answers the question — descriptions cost nothing at query time.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>5. Query expansion <span class="badge b-low">LATER</span></h3>
|
||||
<p>qmd fine-tuned a 1.7B model to generate 2 query variants, searching all three and fusing via RRF. Real quality gains, but the heaviest lift: another model resident in VRAM, ~1–2s latency, and much of the benefit is available cheaper — I already do multi-query decomposition client-side per the kb skill. Park until #1–#3 have landed and the bench shows remaining headroom.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>6. JSON ingestion <span class="badge b-low">TRACKED</span></h3>
|
||||
<p>The original scope of this task: kb rejects <code>.json</code> uploads, forcing renames to <code>.txt</code>. Add <code>.json</code> (and sensibly <code>.yaml</code>/<code>.yml</code>/<code>.toml</code>) to the accepted extensions, ingesting as text. Optional nicety: pretty-print minified JSON before chunking so chunks break on structure.</p>
|
||||
</div>
|
||||
|
||||
<h2>Suggested order</h2>
|
||||
<ol>
|
||||
<li><strong>#3 bench harness</strong> — establish the baseline (a weekend-sized job).</li>
|
||||
<li><strong>#6 JSON support</strong> — small, independent, already promised.</li>
|
||||
<li><strong>#2 RRF fusion</strong> — low-risk engine change, measure against baseline.</li>
|
||||
<li><strong>#1 reranker</strong> — the big win, measured.</li>
|
||||
<li><strong>#4 tag contexts</strong> — anytime, independent.</li>
|
||||
<li><strong>#5 query expansion</strong> — only if the bench still shows a gap.</li>
|
||||
</ol>
|
||||
|
||||
<h2>References</h2>
|
||||
<ul>
|
||||
<li><a href="https://github.com/tobi/qmd">tobi/qmd</a> — architecture, fusion weights, and bench design borrowed from here</li>
|
||||
<li>qmd score fusion detail: RRF k=60, top-rank bonus +0.05/+0.02, position-aware blend 75/60/40% retrieval weight</li>
|
||||
<li>Reranker model: <code>hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF</code> (~640MB)</li>
|
||||
</ul>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user