Add reranking, RRF fusion, bench harness, tag contexts, and data ingestion
Implements five of the six enhancements from docs/kb-enhancements-proposal.htm, closing the retrieval-quality gap identified in the qmd review. - Cross-encoder reranking: new kb/reranker.py loads an optional reranking model at startup (KB_RERANK_ENABLED, KB_RERANKER_MODEL, KB_RERANK_CANDIDATES). Search degrades gracefully to plain hybrid retrieval when the model is absent. Exposed via a "rerank" block in /status, a rerank flag on search, and --no-rerank in the CLI. - RRF rank fusion: FTS and vector lists now merge by reciprocal rank fusion with a top-rank bonus, replacing the old score blend. Scores are comparable across queries. - Bench harness and explain traces: kb bench runs a query fixture against each backend and reports precision@k, recall and MRR. --explain returns a per-result score breakdown. - Tag context descriptions: tags carry an optional one-line description (kb tag-describe), returned as tag_contexts with search results. Adds a tags.description column migration. - Structured data ingestion: .json/.yaml/.toml files ingest as text via the new "data" doc type, pretty-printing minified JSON before chunking. Query expansion (proposal item 5) is deliberately left out pending bench results. Requires engine v3.3.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+32
-2
@@ -21,6 +21,35 @@ make build # produces ./kb binary
|
||||
make all # or cross-compile: dist/kb-{os}-{arch}
|
||||
```
|
||||
|
||||
## Running tests
|
||||
|
||||
### Engine
|
||||
|
||||
Engine tests run against SQLite (with sqlite-vec) and stub out the embedding
|
||||
model, so they only need lightweight dependencies — no torch/docling install:
|
||||
|
||||
```bash
|
||||
uv venv /tmp/kb-test-venv
|
||||
uv pip install --python /tmp/kb-test-venv/bin/python pytest pytest-asyncio fastapi httpx sqlite-vec
|
||||
cd engine && /tmp/kb-test-venv/bin/python -m pytest
|
||||
```
|
||||
|
||||
### Client
|
||||
|
||||
```bash
|
||||
cd client && go test ./...
|
||||
```
|
||||
|
||||
## Search-quality benchmarking
|
||||
|
||||
`kb bench fixture.json` runs a fixture of queries with known-relevant documents
|
||||
against each backend (fts, vec, hybrid, hybrid+rerank) and reports precision@k,
|
||||
recall, and MRR. See `docs/bench-example.json` for the fixture format.
|
||||
|
||||
Run a bench before and after any ranking change (RRF weights, reranker, model
|
||||
swap) and compare — keep a 20-30 query fixture against your real corpus outside
|
||||
the repo.
|
||||
|
||||
## Building and releasing
|
||||
|
||||
Client and engine are versioned independently via `client/VERSION` and `engine/VERSION`. Each has its own release script and git tag prefix.
|
||||
@@ -88,8 +117,9 @@ All endpoints are under `/api/v1/`. Requires `Authorization: Bearer <key>` heade
|
||||
| `GET` | `/documents/{id}/file` | Download original file |
|
||||
| `DELETE` | `/documents/{id}` | Remove a document (and stored file) |
|
||||
| `PUT` | `/documents/{id}/tags` | Add/remove tags |
|
||||
| `GET` | `/tags` | List all tags |
|
||||
| `GET` | `/status` | Engine status, GPU info, DB stats |
|
||||
| `GET` | `/tags` | List all tags (with descriptions) |
|
||||
| `PUT` | `/tags/{name}/description` | Set/clear a tag context description |
|
||||
| `GET` | `/status` | Engine status, GPU info, DB stats, rerank state |
|
||||
| `POST` | `/reindex` | Re-embed all chunks |
|
||||
| `POST` | `/bulk/delete` | Bulk delete documents by filter |
|
||||
| `POST` | `/bulk/tags` | Bulk add/remove tags by filter |
|
||||
|
||||
@@ -143,8 +143,9 @@ kb bulk-set-tags --tags "old-scheme" --set "new-scheme" --yes
|
||||
|
||||
## How it works
|
||||
|
||||
- **Ingestion**: Files are uploaded to the engine and queued for async processing. The engine chunks documents (PDFs via Docling, markdown by headers, code by AST/functions, notes as whole text), generates embeddings on GPU, and stores everything in SQLite.
|
||||
- **Search**: Hybrid retrieval combining BM25 keyword scoring (FTS5) and vector similarity (sqlite-vec), merged via Reciprocal Rank Fusion. Sub-100ms with a warm model.
|
||||
- **Ingestion**: Files are uploaded to the engine and queued for async processing. The engine chunks documents (PDFs via Docling, markdown by headers, code by AST/functions, notes as whole text, JSON/YAML/TOML as pretty-printed text), generates embeddings on GPU, and stores everything in SQLite.
|
||||
- **Search**: Hybrid retrieval combining BM25 keyword scoring (FTS5) and vector similarity (sqlite-vec), merged via Reciprocal Rank Fusion with a top-rank bonus. Optionally reranked by a local cross-encoder (`KB_RERANK_ENABLED`). Sub-100ms with a warm model (without reranking). Add `--explain` to any search for a per-result score breakdown.
|
||||
- **Quality measurement**: `kb bench fixture.json` runs a fixture of queries with known-relevant documents and reports precision@k / recall / MRR per backend (fts, vec, hybrid, hybrid+rerank). See `docs/bench-example.json`.
|
||||
- **Output**: JSON (for scripts/LLM tool use) or human-readable terminal format. Use `--format json` on any command.
|
||||
|
||||
## Engine configuration
|
||||
@@ -159,6 +160,9 @@ The engine is configured via environment variables (set in the compose file or v
|
||||
| `KB_INGEST_DEVICE` | `auto` | Docling layout detection device: `auto`, `cpu`, or `cuda` |
|
||||
| `KB_API_KEY` | (none) | Optional Bearer token for API authentication |
|
||||
| `KB_SEARCH_THRESHOLD` | `0.01` | Minimum score for search results (filters noise) |
|
||||
| `KB_RERANK_ENABLED` | `false` (`true` in nvidia compose) | Load a cross-encoder and rerank hybrid results server-side |
|
||||
| `KB_RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | Cross-encoder model for reranking |
|
||||
| `KB_RERANK_CANDIDATES` | `40` | Hybrid candidates scored by the reranker per query |
|
||||
| `KB_BULK_SAFETY_PERCENT` | `70` | Bulk operations affecting more than this % of documents are rejected unless `force` is set (0 disables) |
|
||||
| `KB_PORT` | `8000` | Port to expose |
|
||||
| `KB_HOST` | `0.0.0.0` | Host to bind to |
|
||||
|
||||
@@ -30,11 +30,13 @@ Returns JSON with ranked results combining full-text and semantic search.
|
||||
**Flags:**
|
||||
- `-n, --top N` — number of results (default: 10)
|
||||
- `--tags tag1,tag2` — filter by tags (AND logic)
|
||||
- `--type pdf|markdown|code|note` — filter by document type
|
||||
- `--type pdf|markdown|code|note|data` — filter by document type
|
||||
- `--format json|human` — output format (always use json for parsing)
|
||||
- `--fts-only` — keyword search only (skip semantic)
|
||||
- `--vec-only` — semantic search only (skip keyword)
|
||||
- `--threshold FLOAT` — minimum score cutoff
|
||||
- `--explain` — include a per-result score breakdown (FTS/vector scores and ranks, fusion contributions, rerank blend)
|
||||
- `--no-rerank` — skip server-side cross-encoder reranking for lower latency (when the engine has it enabled)
|
||||
|
||||
## Adding files
|
||||
|
||||
@@ -45,7 +47,7 @@ kb addfile ~/docs/ --recursive # directory (recursive)
|
||||
kb addfile ~/docs/ --recursive --tags reference # directory with tags
|
||||
```
|
||||
|
||||
Supported file types: `.pdf`, `.docx`, `.html`, `.md`, `.txt`, `.py`, `.sh`, `.go`. Unsupported extensions are rejected before upload.
|
||||
Supported file types: `.pdf`, `.docx`, `.html`, `.md`, `.txt`, `.py`, `.sh`, `.go`, `.json`, `.yaml`, `.yml`, `.toml`. Unsupported extensions are rejected before upload. Data files (`.json`/`.yaml`/`.yml`/`.toml`) are ingested as text with doc type `data`; minified JSON is pretty-printed before chunking.
|
||||
|
||||
**Flags:**
|
||||
- `--tags tag1,tag2` — tags (comma-separated)
|
||||
@@ -66,11 +68,17 @@ kb remove <doc_id> --yes # remove without confirmation
|
||||
## Tag management
|
||||
|
||||
```bash
|
||||
kb tags --format json # list all tags with counts
|
||||
kb tags --format json # list all tags with counts and descriptions
|
||||
kb tag <doc_id> --add important,ops # add tags to a document
|
||||
kb tag <doc_id> --remove draft # remove tags from a document
|
||||
kb tag-describe ops "Lab operations runbooks" # set a tag context description
|
||||
kb tag-describe ops # clear a tag's description
|
||||
```
|
||||
|
||||
Tag descriptions are returned as `tag_contexts` with every search result on a
|
||||
document carrying the tag — use them to judge which of several similar-scoring
|
||||
chunks actually answers the question.
|
||||
|
||||
## Bulk operations
|
||||
|
||||
Operate on multiple documents at once using filter-based selection. Filters combine with AND logic.
|
||||
@@ -138,6 +146,7 @@ All commands support:
|
||||
"results": [
|
||||
{
|
||||
"chunk_id": 1423,
|
||||
"document_id": 87,
|
||||
"score": 0.031,
|
||||
"text": "To install the latest version of git from source...",
|
||||
"chunk_index": 3,
|
||||
@@ -146,11 +155,13 @@ All commands support:
|
||||
"doc_type": "pdf",
|
||||
"source_path": "/home/user/docs/git-admin.pdf",
|
||||
"created_at": "2026-03-15T10:30:00",
|
||||
"tags": ["git", "admin"]
|
||||
"tags": ["git", "admin"],
|
||||
"tag_contexts": {"admin": "System administration guides"}
|
||||
}
|
||||
],
|
||||
"total_matches": 47,
|
||||
"returned": 10
|
||||
"returned": 10,
|
||||
"reranked": true
|
||||
}
|
||||
```
|
||||
|
||||
@@ -217,7 +228,8 @@ If the kb engine is already running via Docker Compose, add the MCP server by de
|
||||
## Important notes
|
||||
|
||||
- Always use `--format json` for machine parsing
|
||||
- The `score` field is relative, not absolute — compare scores within a result set
|
||||
- The `score` field is relative, not absolute — compare scores within a result set. Reranked hybrid scores (`"reranked": true`) are 0-1 blended values on a different scale from non-reranked RRF scores; don't compare across the two modes or apply `--threshold` expecting RRF-scale values on reranked output
|
||||
- When the engine reranker is enabled, results are already cross-encoder reranked server-side — no need to rerank them yourself
|
||||
- `chunk_metadata.page` is only present for PDF documents
|
||||
- `chunk_metadata.section_header` is only present for markdown documents with headers
|
||||
- Results are already ranked by relevance (hybrid FTS + vector search)
|
||||
|
||||
@@ -1 +1 @@
|
||||
3.2.0
|
||||
3.3.0
|
||||
|
||||
@@ -38,6 +38,10 @@ var supportedExts = map[string]bool{
|
||||
".py": true,
|
||||
".sh": true,
|
||||
".go": true,
|
||||
".json": true,
|
||||
".yaml": true,
|
||||
".yml": true,
|
||||
".toml": true,
|
||||
}
|
||||
|
||||
var addfileCmd = &cobra.Command{
|
||||
|
||||
@@ -0,0 +1,376 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var benchCmd = &cobra.Command{
|
||||
Use: "bench <fixture.json>",
|
||||
Short: "Benchmark search quality against a query fixture",
|
||||
Long: `Run a fixture of queries with known-relevant documents against each
|
||||
search backend (fts, vec, hybrid, rerank) and report precision@k, recall
|
||||
and MRR per backend. Use this to baseline search quality before ranking
|
||||
changes and to measure their effect. See docs/bench-example.json for the
|
||||
fixture format.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: runBench,
|
||||
}
|
||||
|
||||
func init() {
|
||||
benchCmd.Flags().Int("top", 0, "override result count per query (k)")
|
||||
benchCmd.Flags().String("backends", "fts,vec,hybrid,rerank", "comma-separated backends to run")
|
||||
rootCmd.AddCommand(benchCmd)
|
||||
}
|
||||
|
||||
type benchSelector struct {
|
||||
DocumentID int64 `json:"document_id"`
|
||||
SourcePath string `json:"source_path"`
|
||||
TitleContains string `json:"title_contains"`
|
||||
}
|
||||
|
||||
type benchQuery struct {
|
||||
ID string `json:"id"`
|
||||
Query string `json:"query"`
|
||||
Tags []string `json:"tags"`
|
||||
DocType string `json:"doc_type"`
|
||||
Top int `json:"top"`
|
||||
Relevant []benchSelector `json:"relevant"`
|
||||
Notes string `json:"notes"`
|
||||
}
|
||||
|
||||
type benchFixture struct {
|
||||
Description string `json:"description"`
|
||||
Top int `json:"top"`
|
||||
Queries []benchQuery `json:"queries"`
|
||||
}
|
||||
|
||||
type benchDoc struct {
|
||||
DocumentID int64
|
||||
Title string
|
||||
SourcePath string
|
||||
}
|
||||
|
||||
type queryResult struct {
|
||||
ID string `json:"id"`
|
||||
Precision float64 `json:"precision"`
|
||||
Recall float64 `json:"recall"`
|
||||
MRR float64 `json:"mrr"`
|
||||
LatencyMS float64 `json:"latency_ms"`
|
||||
Returned int `json:"returned"`
|
||||
}
|
||||
|
||||
type backendResult struct {
|
||||
Precision float64 `json:"precision"`
|
||||
Recall float64 `json:"recall"`
|
||||
MRR float64 `json:"mrr"`
|
||||
AvgLatencyMS float64 `json:"avg_latency_ms"`
|
||||
Queries []queryResult `json:"queries"`
|
||||
}
|
||||
|
||||
func loadFixture(path string) (*benchFixture, error) {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot read fixture: %w", err)
|
||||
}
|
||||
var fx benchFixture
|
||||
if err := json.Unmarshal(data, &fx); err != nil {
|
||||
return nil, fmt.Errorf("invalid fixture JSON: %w", err)
|
||||
}
|
||||
if len(fx.Queries) == 0 {
|
||||
return nil, fmt.Errorf("fixture has no queries")
|
||||
}
|
||||
for i, q := range fx.Queries {
|
||||
if q.Query == "" {
|
||||
return nil, fmt.Errorf("query %d: missing query text", i)
|
||||
}
|
||||
if len(q.Relevant) == 0 {
|
||||
return nil, fmt.Errorf("query %q: no relevant selectors", q.Query)
|
||||
}
|
||||
for j, sel := range q.Relevant {
|
||||
set := 0
|
||||
if sel.DocumentID != 0 {
|
||||
set++
|
||||
}
|
||||
if sel.SourcePath != "" {
|
||||
set++
|
||||
}
|
||||
if sel.TitleContains != "" {
|
||||
set++
|
||||
}
|
||||
if set != 1 {
|
||||
return nil, fmt.Errorf("query %q selector %d: exactly one of document_id, source_path, title_contains must be set", q.Query, j)
|
||||
}
|
||||
}
|
||||
}
|
||||
return &fx, nil
|
||||
}
|
||||
|
||||
func (s benchSelector) matches(doc benchDoc) bool {
|
||||
switch {
|
||||
case s.DocumentID != 0:
|
||||
return doc.DocumentID == s.DocumentID
|
||||
case s.SourcePath != "":
|
||||
return doc.SourcePath == s.SourcePath
|
||||
case s.TitleContains != "":
|
||||
return strings.Contains(strings.ToLower(doc.Title), strings.ToLower(s.TitleContains))
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// dedupeByDocument collapses ranked chunks into ranked documents, keeping the
|
||||
// best (first) position for each document.
|
||||
func dedupeByDocument(docs []benchDoc) []benchDoc {
|
||||
seen := map[int64]bool{}
|
||||
var out []benchDoc
|
||||
for _, d := range docs {
|
||||
if seen[d.DocumentID] {
|
||||
continue
|
||||
}
|
||||
seen[d.DocumentID] = true
|
||||
out = append(out, d)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// scoreQuery computes document-level precision, recall and MRR for one
|
||||
// query's ranked document list against the relevant-document selectors.
|
||||
func scoreQuery(ranked []benchDoc, relevant []benchSelector) (precision, recall, mrr float64) {
|
||||
if len(ranked) == 0 {
|
||||
return 0, 0, 0
|
||||
}
|
||||
|
||||
matchedDocs := 0
|
||||
firstMatch := 0
|
||||
selectorHit := make([]bool, len(relevant))
|
||||
for i, doc := range ranked {
|
||||
docMatched := false
|
||||
for j, sel := range relevant {
|
||||
if sel.matches(doc) {
|
||||
docMatched = true
|
||||
selectorHit[j] = true
|
||||
}
|
||||
}
|
||||
if docMatched {
|
||||
matchedDocs++
|
||||
if firstMatch == 0 {
|
||||
firstMatch = i + 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
selectorsMatched := 0
|
||||
for _, hit := range selectorHit {
|
||||
if hit {
|
||||
selectorsMatched++
|
||||
}
|
||||
}
|
||||
|
||||
precision = float64(matchedDocs) / float64(len(ranked))
|
||||
recall = float64(selectorsMatched) / float64(len(relevant))
|
||||
if firstMatch > 0 {
|
||||
mrr = 1.0 / float64(firstMatch)
|
||||
}
|
||||
return precision, recall, mrr
|
||||
}
|
||||
|
||||
// rerankAvailable probes engine status for a loaded reranker.
|
||||
func rerankAvailable(client *api.Client) bool {
|
||||
resp, err := client.Get("/api/v1/status")
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
var status struct {
|
||||
Rerank struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Loaded bool `json:"loaded"`
|
||||
} `json:"rerank"`
|
||||
}
|
||||
if err := api.DecodeJSON(resp, &status); err != nil {
|
||||
return false
|
||||
}
|
||||
return status.Rerank.Enabled && status.Rerank.Loaded
|
||||
}
|
||||
|
||||
func benchSearch(client *api.Client, q benchQuery, backend string, top int) ([]benchDoc, float64, error) {
|
||||
body := map[string]interface{}{
|
||||
"query": q.Query,
|
||||
"top": top,
|
||||
}
|
||||
if len(q.Tags) > 0 {
|
||||
body["tags"] = q.Tags
|
||||
}
|
||||
if q.DocType != "" {
|
||||
body["doc_type"] = q.DocType
|
||||
}
|
||||
switch backend {
|
||||
case "fts":
|
||||
body["fts_only"] = true
|
||||
case "vec":
|
||||
body["vec_only"] = true
|
||||
case "hybrid":
|
||||
body["rerank"] = false
|
||||
case "rerank":
|
||||
body["rerank"] = true
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
resp, err := client.Post("/api/v1/search", body)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
var result struct {
|
||||
Results []struct {
|
||||
DocumentID int64 `json:"document_id"`
|
||||
Title string `json:"title"`
|
||||
SourcePath string `json:"source_path"`
|
||||
} `json:"results"`
|
||||
}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
latency := float64(time.Since(start).Microseconds()) / 1000.0
|
||||
|
||||
var docs []benchDoc
|
||||
for _, r := range result.Results {
|
||||
docs = append(docs, benchDoc{DocumentID: r.DocumentID, Title: r.Title, SourcePath: r.SourcePath})
|
||||
}
|
||||
return dedupeByDocument(docs), latency, nil
|
||||
}
|
||||
|
||||
func runBench(cmd *cobra.Command, args []string) error {
|
||||
topFlag, _ := cmd.Flags().GetInt("top")
|
||||
backendsFlag, _ := cmd.Flags().GetString("backends")
|
||||
|
||||
fx, err := loadFixture(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var backends []string
|
||||
for _, b := range strings.Split(backendsFlag, ",") {
|
||||
b = strings.TrimSpace(b)
|
||||
if b == "" {
|
||||
continue
|
||||
}
|
||||
switch b {
|
||||
case "fts", "vec", "hybrid", "rerank":
|
||||
backends = append(backends, b)
|
||||
default:
|
||||
return fmt.Errorf("unknown backend %q (valid: fts, vec, hybrid, rerank)", b)
|
||||
}
|
||||
}
|
||||
if len(backends) == 0 {
|
||||
return fmt.Errorf("no backends selected")
|
||||
}
|
||||
|
||||
client := api.NewClient()
|
||||
|
||||
rerankSkipped := false
|
||||
if contains(backends, "rerank") && !rerankAvailable(client) {
|
||||
backends = remove(backends, "rerank")
|
||||
rerankSkipped = true
|
||||
if len(backends) == 0 {
|
||||
return fmt.Errorf("reranking is not available on this engine (requires engine with reranker enabled)")
|
||||
}
|
||||
}
|
||||
|
||||
results := map[string]*backendResult{}
|
||||
for _, backend := range backends {
|
||||
br := &backendResult{}
|
||||
for _, q := range fx.Queries {
|
||||
top := 10
|
||||
if fx.Top > 0 {
|
||||
top = fx.Top
|
||||
}
|
||||
if q.Top > 0 {
|
||||
top = q.Top
|
||||
}
|
||||
if topFlag > 0 {
|
||||
top = topFlag
|
||||
}
|
||||
|
||||
ranked, latency, err := benchSearch(client, q, backend, top)
|
||||
if err != nil {
|
||||
return fmt.Errorf("backend %s, query %q: %w", backend, q.Query, err)
|
||||
}
|
||||
p, r, m := scoreQuery(ranked, q.Relevant)
|
||||
id := q.ID
|
||||
if id == "" {
|
||||
id = q.Query
|
||||
}
|
||||
br.Queries = append(br.Queries, queryResult{
|
||||
ID: id, Precision: p, Recall: r, MRR: m,
|
||||
LatencyMS: latency, Returned: len(ranked),
|
||||
})
|
||||
}
|
||||
n := float64(len(br.Queries))
|
||||
for _, qr := range br.Queries {
|
||||
br.Precision += qr.Precision / n
|
||||
br.Recall += qr.Recall / n
|
||||
br.MRR += qr.MRR / n
|
||||
br.AvgLatencyMS += qr.LatencyMS / n
|
||||
}
|
||||
results[backend] = br
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(map[string]interface{}{
|
||||
"description": fx.Description,
|
||||
"query_count": len(fx.Queries),
|
||||
"backends": results,
|
||||
"rerank_skipped": rerankSkipped,
|
||||
})
|
||||
return nil
|
||||
}
|
||||
|
||||
if fx.Description != "" {
|
||||
fmt.Printf("%s (%d queries)\n\n", fx.Description, len(fx.Queries))
|
||||
}
|
||||
headers := []string{"Backend", "Precision", "Recall", "MRR", "Avg ms"}
|
||||
var rows [][]string
|
||||
for _, backend := range backends {
|
||||
br := results[backend]
|
||||
rows = append(rows, []string{
|
||||
backend,
|
||||
fmt.Sprintf("%.3f", br.Precision),
|
||||
fmt.Sprintf("%.3f", br.Recall),
|
||||
fmt.Sprintf("%.3f", br.MRR),
|
||||
fmt.Sprintf("%.0f", br.AvgLatencyMS),
|
||||
})
|
||||
}
|
||||
output.PrintTable(headers, rows)
|
||||
if rerankSkipped {
|
||||
fmt.Println("\nrerank: n/a (reranker not enabled on this engine)")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func contains(ss []string, s string) bool {
|
||||
for _, v := range ss {
|
||||
if v == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func remove(ss []string, s string) []string {
|
||||
var out []string
|
||||
for _, v := range ss {
|
||||
if v != s {
|
||||
out = append(out, v)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func writeFixture(t *testing.T, content string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "fixture.json")
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestLoadFixture_Valid(t *testing.T) {
|
||||
path := writeFixture(t, `{
|
||||
"description": "test",
|
||||
"top": 5,
|
||||
"queries": [
|
||||
{"id": "q1", "query": "hello", "relevant": [{"document_id": 7}]}
|
||||
]
|
||||
}`)
|
||||
fx, err := loadFixture(path)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if fx.Top != 5 || len(fx.Queries) != 1 || fx.Queries[0].Relevant[0].DocumentID != 7 {
|
||||
t.Errorf("fixture parsed incorrectly: %+v", fx)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFixture_RejectsEmptyRelevant(t *testing.T) {
|
||||
path := writeFixture(t, `{"queries": [{"query": "hello", "relevant": []}]}`)
|
||||
if _, err := loadFixture(path); err == nil {
|
||||
t.Error("expected error for query with no relevant selectors")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFixture_RejectsMultiFieldSelector(t *testing.T) {
|
||||
path := writeFixture(t, `{"queries": [
|
||||
{"query": "hello", "relevant": [{"document_id": 1, "source_path": "/x"}]}
|
||||
]}`)
|
||||
if _, err := loadFixture(path); err == nil {
|
||||
t.Error("expected error for selector with two fields set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelectorMatching(t *testing.T) {
|
||||
doc := benchDoc{DocumentID: 42, Title: "M38T Owner's Manual", SourcePath: "/data/m38t.pdf"}
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
sel benchSelector
|
||||
want bool
|
||||
}{
|
||||
{"document_id match", benchSelector{DocumentID: 42}, true},
|
||||
{"document_id miss", benchSelector{DocumentID: 43}, false},
|
||||
{"source_path match", benchSelector{SourcePath: "/data/m38t.pdf"}, true},
|
||||
{"source_path miss", benchSelector{SourcePath: "/data/other.pdf"}, false},
|
||||
{"title_contains case-insensitive", benchSelector{TitleContains: "m38t owner"}, true},
|
||||
{"title_contains miss", benchSelector{TitleContains: "workshop"}, false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := tc.sel.matches(doc); got != tc.want {
|
||||
t.Errorf("%s: got %v, want %v", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDedupeByDocument(t *testing.T) {
|
||||
docs := []benchDoc{
|
||||
{DocumentID: 1, Title: "a"},
|
||||
{DocumentID: 2, Title: "b"},
|
||||
{DocumentID: 1, Title: "a-again"},
|
||||
{DocumentID: 3, Title: "c"},
|
||||
}
|
||||
out := dedupeByDocument(docs)
|
||||
if len(out) != 3 || out[0].DocumentID != 1 || out[1].DocumentID != 2 || out[2].DocumentID != 3 {
|
||||
t.Errorf("dedupe failed: %+v", out)
|
||||
}
|
||||
if out[0].Title != "a" {
|
||||
t.Errorf("dedupe must keep first (best-ranked) occurrence, got %q", out[0].Title)
|
||||
}
|
||||
}
|
||||
|
||||
func approxEqual(a, b float64) bool {
|
||||
return math.Abs(a-b) < 1e-9
|
||||
}
|
||||
|
||||
func TestScoreQuery_HandComputed(t *testing.T) {
|
||||
// Ranked docs: 10, 20, 30, 40. Relevant: 20 and 40.
|
||||
ranked := []benchDoc{
|
||||
{DocumentID: 10}, {DocumentID: 20}, {DocumentID: 30}, {DocumentID: 40},
|
||||
}
|
||||
relevant := []benchSelector{{DocumentID: 20}, {DocumentID: 40}}
|
||||
|
||||
p, r, m := scoreQuery(ranked, relevant)
|
||||
if !approxEqual(p, 0.5) { // 2 of 4 returned docs are relevant
|
||||
t.Errorf("precision: got %v, want 0.5", p)
|
||||
}
|
||||
if !approxEqual(r, 1.0) { // both relevant docs found
|
||||
t.Errorf("recall: got %v, want 1.0", r)
|
||||
}
|
||||
if !approxEqual(m, 0.5) { // first relevant doc at rank 2
|
||||
t.Errorf("mrr: got %v, want 0.5", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScoreQuery_NoMatches(t *testing.T) {
|
||||
ranked := []benchDoc{{DocumentID: 1}}
|
||||
relevant := []benchSelector{{DocumentID: 99}}
|
||||
p, r, m := scoreQuery(ranked, relevant)
|
||||
if p != 0 || r != 0 || m != 0 {
|
||||
t.Errorf("expected all-zero metrics, got p=%v r=%v mrr=%v", p, r, m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScoreQuery_PartialRecall(t *testing.T) {
|
||||
// Only one of three relevant docs returned, at rank 1.
|
||||
ranked := []benchDoc{{DocumentID: 5}, {DocumentID: 6}}
|
||||
relevant := []benchSelector{{DocumentID: 5}, {DocumentID: 7}, {DocumentID: 8}}
|
||||
p, r, m := scoreQuery(ranked, relevant)
|
||||
if !approxEqual(p, 0.5) {
|
||||
t.Errorf("precision: got %v, want 0.5", p)
|
||||
}
|
||||
if !approxEqual(r, 1.0/3.0) {
|
||||
t.Errorf("recall: got %v, want 1/3", r)
|
||||
}
|
||||
if !approxEqual(m, 1.0) {
|
||||
t.Errorf("mrr: got %v, want 1.0", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScoreQuery_EmptyResults(t *testing.T) {
|
||||
p, r, m := scoreQuery(nil, []benchSelector{{DocumentID: 1}})
|
||||
if p != 0 || r != 0 || m != 0 {
|
||||
t.Errorf("expected zeros for empty results, got p=%v r=%v mrr=%v", p, r, m)
|
||||
}
|
||||
}
|
||||
+73
-4
@@ -23,6 +23,8 @@ func init() {
|
||||
searchCmd.Flags().Bool("fts-only", false, "use full-text search only")
|
||||
searchCmd.Flags().Bool("vec-only", false, "use vector search only")
|
||||
searchCmd.Flags().Float64("threshold", 0, "minimum score threshold")
|
||||
searchCmd.Flags().Bool("explain", false, "include per-result score breakdown")
|
||||
searchCmd.Flags().Bool("no-rerank", false, "skip cross-encoder reranking (lower latency)")
|
||||
rootCmd.AddCommand(searchCmd)
|
||||
}
|
||||
|
||||
@@ -33,16 +35,18 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
ftsOnly, _ := cmd.Flags().GetBool("fts-only")
|
||||
vecOnly, _ := cmd.Flags().GetBool("vec-only")
|
||||
threshold, _ := cmd.Flags().GetFloat64("threshold")
|
||||
explain, _ := cmd.Flags().GetBool("explain")
|
||||
noRerank, _ := cmd.Flags().GetBool("no-rerank")
|
||||
|
||||
body := map[string]interface{}{
|
||||
"query": args[0],
|
||||
"top": top,
|
||||
}
|
||||
if tags != "" {
|
||||
body["tags"] = tags
|
||||
body["tags"] = splitTags(tags)
|
||||
}
|
||||
if docType != "" {
|
||||
body["type"] = docType
|
||||
body["doc_type"] = docType
|
||||
}
|
||||
if ftsOnly {
|
||||
body["fts_only"] = true
|
||||
@@ -53,6 +57,12 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
if threshold > 0 {
|
||||
body["threshold"] = threshold
|
||||
}
|
||||
if explain {
|
||||
body["explain"] = true
|
||||
}
|
||||
if noRerank {
|
||||
body["rerank"] = false
|
||||
}
|
||||
|
||||
client := api.NewClient()
|
||||
resp, err := client.Post("/api/v1/search", body)
|
||||
@@ -66,13 +76,17 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
}
|
||||
|
||||
var result struct {
|
||||
Results []struct {
|
||||
Reranked bool `json:"reranked"`
|
||||
Results []struct {
|
||||
Score float64 `json:"score"`
|
||||
DocumentID int64 `json:"document_id"`
|
||||
Title string `json:"title"`
|
||||
DocType string `json:"doc_type"`
|
||||
Tags []string `json:"tags"`
|
||||
TagContexts map[string]string `json:"tag_contexts"`
|
||||
ChunkMetadata map[string]interface{} `json:"chunk_metadata"`
|
||||
Text string `json:"text"`
|
||||
Explain map[string]interface{} `json:"explain"`
|
||||
} `json:"results"`
|
||||
}
|
||||
|
||||
@@ -94,13 +108,17 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
if result.Reranked {
|
||||
fmt.Println("(reranked)")
|
||||
}
|
||||
|
||||
for i, r := range result.Results {
|
||||
snippet := r.Text
|
||||
if len(snippet) > 200 {
|
||||
snippet = snippet[:200] + "..."
|
||||
}
|
||||
|
||||
fmt.Printf("\n%d. [%.4f] %s\n", i+1, r.Score, r.Title)
|
||||
fmt.Printf("\n%d. [%.4f] %s (doc:%d)\n", i+1, r.Score, r.Title, r.DocumentID)
|
||||
|
||||
location := ""
|
||||
if page, ok := r.ChunkMetadata["page"]; ok && page != nil {
|
||||
@@ -123,12 +141,63 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
if len(r.Tags) > 0 {
|
||||
fmt.Printf(" Tags: %s\n", joinStrings(r.Tags))
|
||||
}
|
||||
for _, tag := range r.Tags {
|
||||
if desc, ok := r.TagContexts[tag]; ok && desc != "" {
|
||||
fmt.Printf(" Context: %s — %s\n", tag, desc)
|
||||
}
|
||||
}
|
||||
if r.Explain != nil {
|
||||
fmt.Printf(" Score: %s\n", formatExplain(r.Explain))
|
||||
}
|
||||
fmt.Printf(" %s\n", snippet)
|
||||
}
|
||||
fmt.Println()
|
||||
return nil
|
||||
}
|
||||
|
||||
// formatExplain renders the engine's explain breakdown as one line, e.g.
|
||||
// "fts score=4.213 rank=2 (rrf 0.016129) | vec score=0.512 rank=1 (rrf 0.016393) | bonus 0.05 | final 0.082522"
|
||||
func formatExplain(e map[string]interface{}) string {
|
||||
num := func(key string) (float64, bool) {
|
||||
v, ok := e[key].(float64)
|
||||
return v, ok
|
||||
}
|
||||
var parts []string
|
||||
for _, arm := range []string{"fts", "vec"} {
|
||||
score, hasScore := num(arm + "_score")
|
||||
if !hasScore {
|
||||
continue
|
||||
}
|
||||
part := fmt.Sprintf("%s score=%.4f", arm, score)
|
||||
if rank, ok := num(arm + "_rank"); ok {
|
||||
part += fmt.Sprintf(" rank=%d", int(rank))
|
||||
}
|
||||
if rrf, ok := num("rrf_" + arm); ok {
|
||||
part += fmt.Sprintf(" (rrf %.6f)", rrf)
|
||||
}
|
||||
parts = append(parts, part)
|
||||
}
|
||||
if bonus, ok := num("bonus"); ok && bonus > 0 {
|
||||
parts = append(parts, fmt.Sprintf("bonus %.2f", bonus))
|
||||
}
|
||||
for _, key := range []string{"pre_rerank_rank", "rerank_score", "blend_weight"} {
|
||||
if v, ok := num(key); ok {
|
||||
parts = append(parts, fmt.Sprintf("%s %.4f", key, v))
|
||||
}
|
||||
}
|
||||
if final, ok := num("final_score"); ok {
|
||||
parts = append(parts, fmt.Sprintf("final %.6f", final))
|
||||
}
|
||||
result := ""
|
||||
for i, p := range parts {
|
||||
if i > 0 {
|
||||
result += " | "
|
||||
}
|
||||
result += p
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func joinStrings(ss []string) string {
|
||||
result := ""
|
||||
for i, s := range ss {
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSearchCmd_SendsDocTypeAndTagsList(t *testing.T) {
|
||||
var captured map[string]interface{}
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/api/v1/search" {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
}
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
if err := json.Unmarshal(body, &captured); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Write([]byte(`{"query":"q","results":[],"total_matches":0,"returned":0}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
var stdout bytes.Buffer
|
||||
rootCmd.SetOut(&stdout)
|
||||
rootCmd.SetArgs([]string{
|
||||
"search", "oil level",
|
||||
"--engine", server.URL,
|
||||
"--format", "json",
|
||||
"--type", "pdf",
|
||||
"--tags", "manuals, ops,",
|
||||
})
|
||||
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("search command failed: %v", err)
|
||||
}
|
||||
|
||||
if captured == nil {
|
||||
t.Fatal("no request captured by test server")
|
||||
}
|
||||
if got := captured["doc_type"]; got != "pdf" {
|
||||
t.Errorf("expected doc_type=pdf in body, got %v (full body: %v)", got, captured)
|
||||
}
|
||||
if _, present := captured["type"]; present {
|
||||
t.Error("body must not contain legacy 'type' key")
|
||||
}
|
||||
tags, ok := captured["tags"].([]interface{})
|
||||
if !ok {
|
||||
t.Fatalf("expected tags to be a JSON array, got %T (%v)", captured["tags"], captured["tags"])
|
||||
}
|
||||
want := []interface{}{"manuals", "ops"}
|
||||
if !reflect.DeepEqual(tags, want) {
|
||||
t.Errorf("expected tags %v, got %v", want, tags)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitTags(t *testing.T) {
|
||||
got := splitTags(" a ,b,, c ")
|
||||
want := []string{"a", "b", "c"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("splitTags: expected %v, got %v", want, got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var tagDescribeCmd = &cobra.Command{
|
||||
Use: "tag-describe <tag> [description]",
|
||||
Short: "Set a one-line context description on a tag",
|
||||
Long: `Attach a short description to a tag (e.g. "Lab operations runbooks").
|
||||
Descriptions are returned as tag_contexts with every search result on a
|
||||
document carrying the tag, helping consumers judge relevance.
|
||||
|
||||
Omit the description (or pass "") to clear it.`,
|
||||
Args: cobra.RangeArgs(1, 2),
|
||||
RunE: runTagDescribe,
|
||||
}
|
||||
|
||||
func init() {
|
||||
rootCmd.AddCommand(tagDescribeCmd)
|
||||
}
|
||||
|
||||
func runTagDescribe(cmd *cobra.Command, args []string) error {
|
||||
description := ""
|
||||
if len(args) == 2 {
|
||||
description = args[1]
|
||||
}
|
||||
|
||||
body := map[string]interface{}{"description": description}
|
||||
|
||||
client := api.NewClient()
|
||||
resp, err := client.Put("/api/v1/tags/"+args[0]+"/description", body)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
var raw interface{}
|
||||
if err := api.DecodeJSON(resp, &raw); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
output.PrintJSON(raw)
|
||||
return nil
|
||||
}
|
||||
|
||||
if description == "" {
|
||||
fmt.Printf("Description cleared for tag %q\n", args[0])
|
||||
} else {
|
||||
fmt.Printf("Tag %q: %s\n", args[0], description)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+5
-4
@@ -41,8 +41,9 @@ func runTags(cmd *cobra.Command, args []string) error {
|
||||
}
|
||||
|
||||
var tags []struct {
|
||||
Name string `json:"name"`
|
||||
Count int `json:"count"`
|
||||
Name string `json:"name"`
|
||||
Count int `json:"count"`
|
||||
Description string `json:"description"`
|
||||
}
|
||||
if err := api.DecodeJSON(resp, &tags); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
@@ -53,10 +54,10 @@ func runTags(cmd *cobra.Command, args []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
headers := []string{"TAG", "COUNT"}
|
||||
headers := []string{"TAG", "COUNT", "DESCRIPTION"}
|
||||
var rows [][]string
|
||||
for _, t := range tags {
|
||||
rows = append(rows, []string{t.Name, fmt.Sprintf("%d", t.Count)})
|
||||
rows = append(rows, []string{t.Name, fmt.Sprintf("%d", t.Count), t.Description})
|
||||
}
|
||||
output.PrintTable(headers, rows)
|
||||
return nil
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"description": "Example search-quality fixture — copy and adapt against your own corpus",
|
||||
"top": 10,
|
||||
"queries": [
|
||||
{
|
||||
"id": "manual-lookup",
|
||||
"query": "how do I check the oil level",
|
||||
"doc_type": "pdf",
|
||||
"relevant": [
|
||||
{"document_id": 2077},
|
||||
{"title_contains": "owner's manual"}
|
||||
],
|
||||
"notes": "Selectors are OR-matched; each counts as one relevant document. Exactly one of document_id / source_path / title_contains per selector."
|
||||
},
|
||||
{
|
||||
"id": "infra-runbook",
|
||||
"query": "restart the reverse proxy after certificate renewal",
|
||||
"tags": ["ops"],
|
||||
"relevant": [
|
||||
{"source_path": "/data/notes/proxy-runbook.md"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "note-recall",
|
||||
"query": "what did we decide about the backup retention window",
|
||||
"top": 5,
|
||||
"relevant": [
|
||||
{"title_contains": "backup retention"}
|
||||
],
|
||||
"notes": "Per-query top overrides the fixture-level default; the --top flag overrides both."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>kb-search Enhancements Proposal</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #f7f7f5; --fg: #1a1a1a; --muted: #666; --card: #fff;
|
||||
--border: #ddd; --accent: #2563eb; --code-bg: #eef1f5;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--bg: #16181d; --fg: #e6e6e6; --muted: #9aa0a8; --card: #1e2128;
|
||||
--border: #33363e; --accent: #7aa2f7; --code-bg: #262a33;
|
||||
}
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0; padding: 2rem 1rem 4rem; background: var(--bg); color: var(--fg);
|
||||
font: 16px/1.6 -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||
}
|
||||
main { max-width: 860px; margin: 0 auto; }
|
||||
h1 { font-size: 1.9rem; margin-bottom: .2rem; }
|
||||
h2 { margin-top: 2.2rem; border-bottom: 1px solid var(--border); padding-bottom: .3rem; }
|
||||
.meta { color: var(--muted); font-size: .9rem; margin-bottom: 2rem; }
|
||||
.card {
|
||||
background: var(--card); border: 1px solid var(--border); border-radius: 10px;
|
||||
padding: 1rem 1.3rem; margin: 1rem 0;
|
||||
}
|
||||
.card h3 { margin: .2rem 0 .5rem; }
|
||||
.badge {
|
||||
display: inline-block; font-size: .72rem; font-weight: 600; letter-spacing: .03em;
|
||||
padding: .12rem .55rem; border-radius: 999px; vertical-align: middle; margin-left: .5rem;
|
||||
}
|
||||
.b-high { background: #dc262622; color: #dc2626; }
|
||||
.b-med { background: #d9770622; color: #d97706; }
|
||||
.b-low { background: #05966922; color: #059669; }
|
||||
table { border-collapse: collapse; width: 100%; margin: 1rem 0; font-size: .93rem; }
|
||||
th, td { border: 1px solid var(--border); padding: .45rem .7rem; text-align: left; vertical-align: top; }
|
||||
th { background: var(--code-bg); }
|
||||
code { background: var(--code-bg); padding: .1rem .35rem; border-radius: 4px; font-size: .88em; }
|
||||
pre { background: var(--code-bg); padding: .8rem 1rem; border-radius: 8px; overflow-x: auto; }
|
||||
pre code { background: none; padding: 0; }
|
||||
a { color: var(--accent); }
|
||||
.muted { color: var(--muted); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>kb-search Enhancements Proposal</h1>
|
||||
<p class="meta">Bobby · 2026-07-07 · Prompted by a review of <a href="https://github.com/tobi/qmd">tobi/qmd</a> (Tobi Lütke's local hybrid search engine)</p>
|
||||
|
||||
<h2>Summary</h2>
|
||||
<p>qmd and kb-search v2 solve overlapping problems, but qmd's retrieval pipeline is measurably ahead: its own benchmarks show BM25-only at ~0.50, vector-only at ~0.70, and the full hybrid + reranked pipeline at ~1.00. Our kb does hybrid FTS + vector but stops there — no rank fusion, no reranking, no query expansion, and no way to measure whether a change helps or hurts. This proposal lists five enhancements, ordered by value-for-effort, plus the already-tracked JSON ingestion item.</p>
|
||||
|
||||
<table>
|
||||
<tr><th>#</th><th>Enhancement</th><th>Impact</th><th>Effort</th></tr>
|
||||
<tr><td>1</td><td>LLM reranking stage</td><td>High — biggest single search-quality lever</td><td>Medium</td></tr>
|
||||
<tr><td>2</td><td>RRF fusion for FTS + vector merging</td><td>Medium-high</td><td>Low</td></tr>
|
||||
<tr><td>3</td><td>Bench harness + <code>--explain</code> traces</td><td>High (enables everything else)</td><td>Low-medium</td></tr>
|
||||
<tr><td>4</td><td>Context descriptions on tags/sources</td><td>Medium</td><td>Low</td></tr>
|
||||
<tr><td>5</td><td>Query expansion</td><td>Medium</td><td>Medium-high</td></tr>
|
||||
<tr><td>6</td><td>.json file ingestion (already tracked)</td><td>Medium</td><td>Low</td></tr>
|
||||
</table>
|
||||
|
||||
<h2>Current state</h2>
|
||||
<p>kb-search v2 (engine v3.2.2) runs on the RTX 4070 box with <code>BAAI/bge-base-en-v1.5</code> (768-dim). It holds ~2,310 documents (1,944 PDFs, 237 notes, 129 markdown) in ~123k chunks. Search is hybrid FTS + vector with a blended relative score. Strengths over qmd: binary ingestion (PDF/docx/HTML), tags, ingestion job queue, dedup, original export, and multi-client API access. The proposals below close the retrieval-quality gap without giving any of that up.</p>
|
||||
|
||||
<h2>Proposals</h2>
|
||||
|
||||
<div class="card">
|
||||
<h3>1. LLM reranking stage <span class="badge b-high">HIGH IMPACT</span></h3>
|
||||
<p>Add a cross-encoder reranking pass over the top-K hybrid candidates. qmd uses <code>qwen3-reranker-0.6b</code> (~640MB GGUF) — small enough to sit alongside bge on the 4070 permanently. Flow: hybrid retrieval pulls ~40 candidates → reranker scores each (query, chunk) pair → final order blends retrieval and reranker scores.</p>
|
||||
<p>qmd's position-aware blend is worth copying wholesale: rank 1–3 keep 75% retrieval weight, 4–10 get 60%, 11+ get 40%. This stops the reranker destroying exact-match hits while letting it rescue mid-ranked semantic matches.</p>
|
||||
<p class="muted">API: add <code>rerank: bool</code> (default true) to the search endpoint, with <code>--no-rerank</code> in the CLI for latency-sensitive callers.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>2. RRF fusion <span class="badge b-med">MEDIUM-HIGH</span></h3>
|
||||
<p>Replace the current score blend with Reciprocal Rank Fusion when merging FTS and vector lists: <code>score = Σ 1/(k + rank + 1)</code>, k=60. Rank-based fusion sidesteps the incomparability of BM25 scores (unbounded) and cosine similarity (0–1). qmd adds a top-rank bonus (+0.05 for #1, +0.02 for #2–3 in any list) to preserve exact matches — cheap and effective.</p>
|
||||
<p class="muted">Pure engine-side change, no API impact. Scores become comparable across queries too, which fixes the "score is relative, not absolute" caveat in the current skill docs.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>3. Bench harness + explain traces <span class="badge b-high">DO FIRST</span></h3>
|
||||
<p>We currently have no way to know if any of the above helps. Add:</p>
|
||||
<ul>
|
||||
<li><code>kb bench fixture.json</code> — run a fixture of queries with known-relevant docs, report precision@k / recall / MRR per backend (fts-only, vec-only, hybrid, hybrid+rerank). Directly mirrors <code>qmd bench</code>.</li>
|
||||
<li><code>--explain</code> on search — per-result score breakdown (FTS score, vector score, fusion contribution, rerank score).</li>
|
||||
</ul>
|
||||
<p>A fixture of 20–30 real queries against the existing corpus (lab infra questions, manual lookups, note recall) gives a regression baseline before touching ranking. <strong>This should land before #1 and #2 so their benefit is provable.</strong></p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>4. Context descriptions <span class="badge b-med">MEDIUM</span></h3>
|
||||
<p>qmd's standout idea: attach a one-line description to a collection or path (e.g. "Meeting transcripts", "Lab infrastructure runbooks") and return it with every matching result. For kb, the natural unit is the <strong>tag</strong>: <code>kb tag-describe ops "Lab operations runbooks and procedures"</code>, returned as <code>tag_contexts</code> in search results. Helps an LLM consumer (me) judge which of several similar-scoring chunks actually answers the question — descriptions cost nothing at query time.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>5. Query expansion <span class="badge b-low">LATER</span></h3>
|
||||
<p>qmd fine-tuned a 1.7B model to generate 2 query variants, searching all three and fusing via RRF. Real quality gains, but the heaviest lift: another model resident in VRAM, ~1–2s latency, and much of the benefit is available cheaper — I already do multi-query decomposition client-side per the kb skill. Park until #1–#3 have landed and the bench shows remaining headroom.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>6. JSON ingestion <span class="badge b-low">TRACKED</span></h3>
|
||||
<p>The original scope of this task: kb rejects <code>.json</code> uploads, forcing renames to <code>.txt</code>. Add <code>.json</code> (and sensibly <code>.yaml</code>/<code>.yml</code>/<code>.toml</code>) to the accepted extensions, ingesting as text. Optional nicety: pretty-print minified JSON before chunking so chunks break on structure.</p>
|
||||
</div>
|
||||
|
||||
<h2>Suggested order</h2>
|
||||
<ol>
|
||||
<li><strong>#3 bench harness</strong> — establish the baseline (a weekend-sized job).</li>
|
||||
<li><strong>#6 JSON support</strong> — small, independent, already promised.</li>
|
||||
<li><strong>#2 RRF fusion</strong> — low-risk engine change, measure against baseline.</li>
|
||||
<li><strong>#1 reranker</strong> — the big win, measured.</li>
|
||||
<li><strong>#4 tag contexts</strong> — anytime, independent.</li>
|
||||
<li><strong>#5 query expansion</strong> — only if the bench still shows a gap.</li>
|
||||
</ol>
|
||||
|
||||
<h2>References</h2>
|
||||
<ul>
|
||||
<li><a href="https://github.com/tobi/qmd">tobi/qmd</a> — architecture, fusion weights, and bench design borrowed from here</li>
|
||||
<li>qmd score fusion detail: RRF k=60, top-rank bonus +0.05/+0.02, position-aware blend 75/60/40% retrieval weight</li>
|
||||
<li>Reranker model: <code>hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF</code> (~640MB)</li>
|
||||
</ul>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@@ -21,6 +21,9 @@ services:
|
||||
- KB_INGEST_DEVICE=${KB_INGEST_DEVICE:-auto}
|
||||
- KB_API_KEY=${KB_API_KEY:-}
|
||||
- KB_SEARCH_THRESHOLD=${KB_SEARCH_THRESHOLD:-0.01}
|
||||
- KB_RERANK_ENABLED=${KB_RERANK_ENABLED:-true}
|
||||
- KB_RERANKER_MODEL=${KB_RERANKER_MODEL:-BAAI/bge-reranker-v2-m3}
|
||||
- KB_RERANK_CANDIDATES=${KB_RERANK_CANDIDATES:-40}
|
||||
- HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-}
|
||||
restart: unless-stopped
|
||||
|
||||
|
||||
@@ -20,6 +20,9 @@ class Config:
|
||||
self.ingest_device = os.environ.get("KB_INGEST_DEVICE", "auto")
|
||||
self.api_key = os.environ.get("KB_API_KEY") or None
|
||||
self.search_threshold = float(os.environ.get("KB_SEARCH_THRESHOLD", "0.01"))
|
||||
self.rerank_enabled = os.environ.get("KB_RERANK_ENABLED", "false").lower() in ("1", "true", "yes")
|
||||
self.reranker_model = os.environ.get("KB_RERANKER_MODEL", "BAAI/bge-reranker-v2-m3")
|
||||
self.rerank_candidates = int(os.environ.get("KB_RERANK_CANDIDATES", "40"))
|
||||
self.bulk_safety_percent = int(os.environ.get("KB_BULK_SAFETY_PERCENT", "70"))
|
||||
self.host = os.environ.get("KB_HOST", "0.0.0.0")
|
||||
self.port = int(os.environ.get("KB_PORT", "8000"))
|
||||
|
||||
@@ -195,6 +195,11 @@ def init_schema(conn: sqlite3.Connection, embedding_dim: int) -> None:
|
||||
if "job_type" not in job_cols:
|
||||
conn.execute("ALTER TABLE jobs ADD COLUMN job_type TEXT DEFAULT 'ingest'")
|
||||
|
||||
# Migrate: add description to tags if missing (tag contexts, v3.3.0)
|
||||
tag_cols = {row[1] for row in conn.execute("PRAGMA table_info(tags)").fetchall()}
|
||||
if "description" not in tag_cols:
|
||||
conn.execute("ALTER TABLE tags ADD COLUMN description TEXT")
|
||||
|
||||
conn.commit()
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Chunking pipeline for structured data files (JSON, YAML, TOML).
|
||||
|
||||
Data files are ingested as plain text. Minified JSON is pretty-printed
|
||||
first so chunk boundaries fall on structural lines rather than mid-object.
|
||||
Malformed input never fails ingestion — it is chunked as-is.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from kb.ingest.code import _fixed_token_chunks
|
||||
|
||||
|
||||
def chunk_data(
|
||||
text: str,
|
||||
language: str | None,
|
||||
max_tokens: int = 1024,
|
||||
) -> list[dict]:
|
||||
"""Split a data file into chunks.
|
||||
|
||||
Returns a list of chunk dicts, each containing:
|
||||
text, chunk_index, metadata
|
||||
"""
|
||||
if language == "json":
|
||||
try:
|
||||
text = json.dumps(json.loads(text), indent=2, ensure_ascii=False)
|
||||
except (ValueError, TypeError):
|
||||
pass # not valid JSON — ingest the raw text unchanged
|
||||
|
||||
chunks: list[dict] = []
|
||||
for piece in _fixed_token_chunks(text, max_tokens):
|
||||
piece = piece.strip()
|
||||
if piece:
|
||||
chunks.append({
|
||||
"text": piece,
|
||||
"chunk_index": len(chunks),
|
||||
"metadata": {},
|
||||
})
|
||||
return chunks
|
||||
@@ -11,6 +11,10 @@ SUPPORTED_EXTENSIONS: dict[str, tuple[str, str | None]] = {
|
||||
".py": ("code", "python"),
|
||||
".sh": ("code", "bash"),
|
||||
".go": ("code", "go"),
|
||||
".json": ("data", "json"),
|
||||
".yaml": ("data", "yaml"),
|
||||
".yml": ("data", "yaml"),
|
||||
".toml": ("data", "toml"),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Cross-encoder reranker management.
|
||||
|
||||
Mirrors the embeddings module: one module-level model, loaded eagerly at
|
||||
startup when reranking is enabled. Reranking is strictly optional — search
|
||||
degrades gracefully to plain hybrid retrieval when the model is absent.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("kb.reranker")
|
||||
|
||||
_reranker: Optional[object] = None
|
||||
_model_name: Optional[str] = None
|
||||
|
||||
|
||||
def load_reranker(model_name: str, device: str = "cpu") -> None:
|
||||
"""Load a cross-encoder reranking model.
|
||||
|
||||
Args:
|
||||
model_name: HuggingFace model name or local path. Must have a
|
||||
sequence-classification head (e.g. BAAI/bge-reranker-v2-m3).
|
||||
device: Target device — "cpu", "cuda", or "auto".
|
||||
"""
|
||||
global _reranker, _model_name
|
||||
|
||||
from sentence_transformers import CrossEncoder
|
||||
|
||||
from kb.embeddings import _resolve_device
|
||||
|
||||
resolved_device = _resolve_device(device)
|
||||
logger.info("Loading reranker '%s' on device '%s'", model_name, resolved_device)
|
||||
|
||||
_reranker = CrossEncoder(model_name, device=resolved_device)
|
||||
_model_name = model_name
|
||||
|
||||
logger.info("Reranker loaded: %s", model_name)
|
||||
|
||||
|
||||
def is_available() -> bool:
|
||||
"""Return True if a reranker model is loaded and usable."""
|
||||
return _reranker is not None
|
||||
|
||||
|
||||
def rerank_scores(query: str, texts: list[str]) -> list[float]:
|
||||
"""Score (query, text) pairs with the cross-encoder.
|
||||
|
||||
Returns:
|
||||
One relevance score per text, sigmoid-normalised to 0-1.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If no reranker has been loaded.
|
||||
"""
|
||||
if _reranker is None:
|
||||
raise RuntimeError("Reranker not loaded. Call load_reranker() first.")
|
||||
|
||||
import numpy as np
|
||||
|
||||
scores = _reranker.predict([(query, t) for t in texts], convert_to_numpy=True)
|
||||
# CrossEncoder heads may emit raw logits; squash to 0-1 so scores blend
|
||||
# predictably with normalised retrieval scores. Sigmoid is monotonic, so
|
||||
# ordering is unaffected for models that already output probabilities.
|
||||
return (1.0 / (1.0 + np.exp(-np.asarray(scores, dtype="float64")))).tolist()
|
||||
@@ -19,6 +19,8 @@ class SearchRequest(BaseModel):
|
||||
fts_only: bool = False
|
||||
vec_only: bool = False
|
||||
threshold: Optional[float] = None
|
||||
explain: bool = False
|
||||
rerank: Optional[bool] = None
|
||||
|
||||
|
||||
@app.post("/api/v1/search")
|
||||
@@ -35,6 +37,8 @@ async def search(req: SearchRequest):
|
||||
fts_only=req.fts_only,
|
||||
vec_only=req.vec_only,
|
||||
threshold=req.threshold,
|
||||
explain=req.explain,
|
||||
rerank=req.rerank,
|
||||
)
|
||||
return result
|
||||
except Exception as exc:
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import os
|
||||
|
||||
from main import app, __version__
|
||||
from kb import reranker
|
||||
from kb.config import cfg
|
||||
from kb.database import get_connection
|
||||
from kb.embeddings import get_model_dim
|
||||
@@ -62,6 +63,12 @@ async def status():
|
||||
"queued": queue_stats.get("queued", 0),
|
||||
"processing": queue_stats.get("processing", 0),
|
||||
},
|
||||
"rerank": {
|
||||
"enabled": cfg.rerank_enabled,
|
||||
"model": cfg.reranker_model,
|
||||
"loaded": reranker.is_available(),
|
||||
"candidates": cfg.rerank_candidates,
|
||||
},
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
@@ -16,14 +16,45 @@ async def list_tags():
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT t.name, COUNT(dt.document_id) AS count
|
||||
SELECT t.name, t.description, COUNT(dt.document_id) AS count
|
||||
FROM tags t
|
||||
LEFT JOIN document_tags dt ON t.id = dt.tag_id
|
||||
GROUP BY t.id, t.name
|
||||
ORDER BY t.name
|
||||
"""
|
||||
).fetchall()
|
||||
return [{"name": row["name"], "count": row["count"]} for row in rows]
|
||||
return [
|
||||
{"name": row["name"], "count": row["count"], "description": row["description"]}
|
||||
for row in rows
|
||||
]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
class TagDescriptionRequest(BaseModel):
|
||||
description: Optional[str] = None
|
||||
|
||||
|
||||
@app.put("/api/v1/tags/{name}/description")
|
||||
async def set_tag_description(name: str, req: TagDescriptionRequest):
|
||||
"""Set or clear a one-line context description on a tag.
|
||||
|
||||
Descriptions are returned as ``tag_contexts`` with every search result on
|
||||
a document carrying the tag, helping consumers judge relevance.
|
||||
"""
|
||||
conn = get_connection(cfg.db_path)
|
||||
try:
|
||||
# name matching is case-insensitive (tags.name is COLLATE NOCASE)
|
||||
tag = conn.execute("SELECT id FROM tags WHERE name = ?", (name,)).fetchone()
|
||||
if not tag:
|
||||
raise HTTPException(status_code=404, detail=f"Tag '{name}' not found.")
|
||||
|
||||
description = (req.description or "").strip() or None
|
||||
conn.execute(
|
||||
"UPDATE tags SET description = ? WHERE id = ?", (description, tag["id"])
|
||||
)
|
||||
conn.commit()
|
||||
return {"name": name, "description": description}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
+166
-16
@@ -18,6 +18,8 @@ def hybrid_search(
|
||||
fts_only: bool = False,
|
||||
vec_only: bool = False,
|
||||
threshold: float | None = None,
|
||||
explain: bool = False,
|
||||
rerank: bool | None = None,
|
||||
) -> dict:
|
||||
"""Run hybrid search and return merged, enriched results.
|
||||
|
||||
@@ -31,11 +33,29 @@ def hybrid_search(
|
||||
fts_only: Only use FTS5 (skip vector search).
|
||||
vec_only: Only use vector search (skip FTS5).
|
||||
threshold: Optional minimum score; results below are dropped.
|
||||
explain: Attach a per-result score breakdown (arm scores, ranks,
|
||||
RRF contributions, rerank blend) under an ``explain`` key.
|
||||
rerank: Cross-encoder rerank the top candidates. None uses the
|
||||
engine default (cfg.rerank_enabled); True still requires a
|
||||
loaded reranker and degrades silently to plain retrieval
|
||||
otherwise. Never applied to fts_only / vec_only searches.
|
||||
|
||||
Returns:
|
||||
Dict with keys: query, results, total_matches, returned.
|
||||
Dict with keys: query, results, total_matches, returned, reranked.
|
||||
"""
|
||||
from kb import reranker
|
||||
|
||||
want_rerank = cfg.rerank_enabled if rerank is None else rerank
|
||||
do_rerank = (
|
||||
want_rerank
|
||||
and not fts_only
|
||||
and not vec_only
|
||||
and reranker.is_available()
|
||||
)
|
||||
|
||||
candidate_count = top * 3
|
||||
if do_rerank:
|
||||
candidate_count = max(candidate_count, cfg.rerank_candidates)
|
||||
|
||||
fts_results: dict[int, float] = {}
|
||||
vec_results: dict[int, float] = {}
|
||||
@@ -49,10 +69,12 @@ def hybrid_search(
|
||||
# --- merge ---------------------------------------------------------------
|
||||
if fts_only:
|
||||
merged = sorted(fts_results.items(), key=lambda x: x[1], reverse=True)
|
||||
details = _single_arm_details("fts", fts_results)
|
||||
elif vec_only:
|
||||
merged = sorted(vec_results.items(), key=lambda x: x[1], reverse=True)
|
||||
details = _single_arm_details("vec", vec_results)
|
||||
else:
|
||||
merged = _rrf_merge(fts_results, vec_results)
|
||||
merged, details = _rrf_merge(fts_results, vec_results)
|
||||
|
||||
# Apply threshold filter — use config default if not specified per-query
|
||||
effective_threshold = threshold if threshold is not None else cfg.search_threshold
|
||||
@@ -60,16 +82,30 @@ def hybrid_search(
|
||||
merged = [(cid, score) for cid, score in merged if score >= effective_threshold]
|
||||
|
||||
total_matches = len(merged)
|
||||
|
||||
# --- rerank --------------------------------------------------------------
|
||||
# Blended scores are 0-1 normalised, a different scale from RRF scores;
|
||||
# the threshold above was applied to RRF scores and is NOT re-applied.
|
||||
reranked = False
|
||||
if do_rerank and merged:
|
||||
candidates = merged[: cfg.rerank_candidates]
|
||||
rr_scores = reranker.rerank_scores(
|
||||
query, _fetch_chunk_texts(conn, [cid for cid, _ in candidates])
|
||||
)
|
||||
merged = _blend_rerank(candidates, rr_scores, details)
|
||||
reranked = True
|
||||
|
||||
merged = merged[:top]
|
||||
|
||||
# --- enrich --------------------------------------------------------------
|
||||
results = _enrich(conn, merged)
|
||||
results = _enrich(conn, merged, details if explain else None)
|
||||
|
||||
return {
|
||||
"query": query,
|
||||
"results": results,
|
||||
"total_matches": total_matches,
|
||||
"returned": len(results),
|
||||
"reranked": reranked,
|
||||
}
|
||||
|
||||
|
||||
@@ -232,31 +268,135 @@ def _rrf_merge(
|
||||
fts_results: dict[int, float],
|
||||
vec_results: dict[int, float],
|
||||
k: int = 60,
|
||||
) -> list[tuple[int, float]]:
|
||||
) -> tuple[list[tuple[int, float]], dict[int, dict]]:
|
||||
"""Reciprocal Rank Fusion over two scored result sets.
|
||||
|
||||
Each set is ranked independently (highest score first, rank starts at 1).
|
||||
RRF score for a document = sum of 1/(k + rank) across sets it appears in.
|
||||
RRF score for a document = sum of 1/(k + rank) across sets it appears in,
|
||||
plus a top-rank bonus per arm: +0.05 for rank 1, +0.02 for ranks 2-3.
|
||||
The bonus preserves exact matches — a chunk at the top of either arm is
|
||||
nearly impossible to displace via mid-rank RRF accumulation alone.
|
||||
|
||||
Score scale: base RRF maxes at 2/(k+1) ≈ 0.033 (k=60); with bonuses the
|
||||
ceiling is ≈ 0.133. Bonuses only raise scores, so the threshold filter
|
||||
(default 0.01) can never drop a result that base RRF would have kept.
|
||||
|
||||
Returns:
|
||||
Sorted list of (chunk_id, rrf_score), highest first.
|
||||
(scores, details) — scores is a sorted list of (chunk_id, rrf_score),
|
||||
highest first; details maps chunk_id to a per-arm score breakdown
|
||||
suitable for the ``explain`` response field.
|
||||
"""
|
||||
fts_ranked = _rank_by_score(fts_results)
|
||||
vec_ranked = _rank_by_score(vec_results)
|
||||
|
||||
all_ids = set(fts_ranked) | set(vec_ranked)
|
||||
scores: list[tuple[int, float]] = []
|
||||
details: dict[int, dict] = {}
|
||||
|
||||
for chunk_id in all_ids:
|
||||
rrf = 0.0
|
||||
if chunk_id in fts_ranked:
|
||||
rrf += 1.0 / (k + fts_ranked[chunk_id])
|
||||
if chunk_id in vec_ranked:
|
||||
rrf += 1.0 / (k + vec_ranked[chunk_id])
|
||||
fts_rank = fts_ranked.get(chunk_id)
|
||||
vec_rank = vec_ranked.get(chunk_id)
|
||||
rrf_fts = 1.0 / (k + fts_rank) if fts_rank is not None else None
|
||||
rrf_vec = 1.0 / (k + vec_rank) if vec_rank is not None else None
|
||||
bonus = _top_rank_bonus(fts_rank) + _top_rank_bonus(vec_rank)
|
||||
rrf = (rrf_fts or 0.0) + (rrf_vec or 0.0) + bonus
|
||||
|
||||
details[chunk_id] = {
|
||||
"fts_score": _round6(fts_results.get(chunk_id)),
|
||||
"fts_rank": fts_rank,
|
||||
"vec_score": _round6(vec_results.get(chunk_id)),
|
||||
"vec_rank": vec_rank,
|
||||
"rrf_fts": _round6(rrf_fts),
|
||||
"rrf_vec": _round6(rrf_vec),
|
||||
"bonus": bonus,
|
||||
"final_score": _round6(rrf),
|
||||
}
|
||||
scores.append((chunk_id, rrf))
|
||||
|
||||
scores.sort(key=lambda x: x[1], reverse=True)
|
||||
return scores
|
||||
return scores, details
|
||||
|
||||
|
||||
def _top_rank_bonus(rank: int | None) -> float:
|
||||
"""Bonus for appearing at the top of one arm's ranking."""
|
||||
if rank == 1:
|
||||
return 0.05
|
||||
if rank in (2, 3):
|
||||
return 0.02
|
||||
return 0.0
|
||||
|
||||
|
||||
def _single_arm_details(arm: str, results: dict[int, float]) -> dict[int, dict]:
|
||||
"""Explain details for fts_only / vec_only searches (raw arm scores)."""
|
||||
ranked = _rank_by_score(results)
|
||||
return {
|
||||
chunk_id: {
|
||||
f"{arm}_score": _round6(score),
|
||||
f"{arm}_rank": ranked[chunk_id],
|
||||
"final_score": _round6(score),
|
||||
}
|
||||
for chunk_id, score in results.items()
|
||||
}
|
||||
|
||||
|
||||
def _round6(value: float | None) -> float | None:
|
||||
return round(value, 6) if value is not None else None
|
||||
|
||||
|
||||
def _fetch_chunk_texts(conn: sqlite3.Connection, chunk_ids: list[int]) -> list[str]:
|
||||
"""Fetch chunk texts in the same order as *chunk_ids*."""
|
||||
placeholders = ",".join("?" * len(chunk_ids))
|
||||
rows = conn.execute(
|
||||
f"SELECT id, text FROM chunks WHERE id IN ({placeholders})", chunk_ids
|
||||
).fetchall()
|
||||
by_id = {row[0]: row[1] for row in rows}
|
||||
return [by_id.get(cid, "") for cid in chunk_ids]
|
||||
|
||||
|
||||
def _blend_rerank(
|
||||
candidates: list[tuple[int, float]],
|
||||
rr_scores: list[float],
|
||||
details: dict[int, dict],
|
||||
) -> list[tuple[int, float]]:
|
||||
"""Blend retrieval and cross-encoder scores, position-aware.
|
||||
|
||||
Retrieval scores are min-max normalised within the candidate set; rerank
|
||||
scores are already 0-1. The retrieval weight depends on pre-rerank rank —
|
||||
75% for ranks 1-3, 60% for 4-10, 40% for 11+ — so the reranker can rescue
|
||||
mid-ranked semantic matches without destroying top exact-match hits.
|
||||
|
||||
*candidates* must be in retrieval order; *rr_scores* aligned with it.
|
||||
Mutates *details* with the blend breakdown. Returns (chunk_id, blended)
|
||||
sorted highest first.
|
||||
"""
|
||||
retrieval = [score for _, score in candidates]
|
||||
lo, hi = min(retrieval), max(retrieval)
|
||||
span = hi - lo
|
||||
|
||||
blended: list[tuple[int, float]] = []
|
||||
for i, ((chunk_id, score), rr) in enumerate(zip(candidates, rr_scores)):
|
||||
rank = i + 1
|
||||
norm = (score - lo) / span if span > 0 else 1.0
|
||||
if rank <= 3:
|
||||
weight = 0.75
|
||||
elif rank <= 10:
|
||||
weight = 0.60
|
||||
else:
|
||||
weight = 0.40
|
||||
final = weight * norm + (1.0 - weight) * rr
|
||||
|
||||
if chunk_id in details:
|
||||
details[chunk_id].update({
|
||||
"pre_rerank_rank": rank,
|
||||
"retrieval_norm": _round6(norm),
|
||||
"rerank_score": _round6(rr),
|
||||
"blend_weight": weight,
|
||||
"final_score": _round6(final),
|
||||
})
|
||||
blended.append((chunk_id, final))
|
||||
|
||||
blended.sort(key=lambda x: x[1], reverse=True)
|
||||
return blended
|
||||
|
||||
|
||||
def _rank_by_score(results: dict[int, float]) -> dict[int, int]:
|
||||
@@ -268,8 +408,13 @@ def _rank_by_score(results: dict[int, float]) -> dict[int, int]:
|
||||
def _enrich(
|
||||
conn: sqlite3.Connection,
|
||||
merged: list[tuple[int, float]],
|
||||
details: dict[int, dict] | None = None,
|
||||
) -> list[dict]:
|
||||
"""Fetch chunk text, document metadata, chunk metadata, and tags."""
|
||||
"""Fetch chunk text, document metadata, chunk metadata, and tags.
|
||||
|
||||
When *details* is given, each result gains an ``explain`` key with its
|
||||
score breakdown.
|
||||
"""
|
||||
results: list[dict] = []
|
||||
|
||||
for chunk_id, score in merged:
|
||||
@@ -292,7 +437,7 @@ def _enrich(
|
||||
|
||||
tag_rows = conn.execute(
|
||||
"""
|
||||
SELECT t.name FROM tags t
|
||||
SELECT t.name, t.description FROM tags t
|
||||
JOIN document_tags dt ON t.id = dt.tag_id
|
||||
WHERE dt.document_id = ?
|
||||
ORDER BY t.name
|
||||
@@ -300,8 +445,9 @@ def _enrich(
|
||||
(row[4],), # doc_id
|
||||
).fetchall()
|
||||
|
||||
results.append({
|
||||
result = {
|
||||
"chunk_id": row[0],
|
||||
"document_id": row[4],
|
||||
"score": round(score, 6),
|
||||
"text": row[1],
|
||||
"chunk_index": row[2],
|
||||
@@ -311,6 +457,10 @@ def _enrich(
|
||||
"source_path": row[7],
|
||||
"created_at": row[8],
|
||||
"tags": [t[0] for t in tag_rows],
|
||||
})
|
||||
"tag_contexts": {t[0]: t[1] for t in tag_rows if t[1]},
|
||||
}
|
||||
if details is not None and row[0] in details:
|
||||
result["explain"] = details[row[0]]
|
||||
results.append(result)
|
||||
|
||||
return results
|
||||
|
||||
@@ -124,6 +124,12 @@ def _process_job(job_row) -> tuple[str, int | None, int]:
|
||||
_, language = detector.detect_type(Path(filename))
|
||||
from kb.ingest.code import chunk_code
|
||||
chunks = chunk_code(text, language)
|
||||
elif doc_type == "data":
|
||||
text = staged_path.read_text(encoding="utf-8")
|
||||
if not language:
|
||||
_, language = detector.detect_type(Path(filename))
|
||||
from kb.ingest.data import chunk_data
|
||||
chunks = chunk_data(text, language)
|
||||
else:
|
||||
raise ValueError(f"Unsupported doc_type: {doc_type}")
|
||||
|
||||
|
||||
@@ -40,6 +40,18 @@ async def lifespan(app: FastAPI):
|
||||
init_schema(conn, model_dim)
|
||||
conn.close()
|
||||
|
||||
# Optional reranker — search degrades gracefully if this fails
|
||||
if cfg.rerank_enabled:
|
||||
from kb.reranker import load_reranker
|
||||
try:
|
||||
load_reranker(cfg.reranker_model, cfg.device)
|
||||
except Exception:
|
||||
log.warning(
|
||||
"Failed to load reranker '%s' — searches will not be reranked",
|
||||
cfg.reranker_model,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
# Start background ingestion worker
|
||||
worker_task = asyncio.create_task(ingestion_worker())
|
||||
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Shared test setup — make the engine root importable (for ``import kb``)."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Tests for structured-data (.json/.yaml/.toml) ingestion."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from kb.ingest.data import chunk_data
|
||||
from kb.ingest.detector import detect_type, is_supported
|
||||
|
||||
|
||||
def test_detector_accepts_data_extensions():
|
||||
assert detect_type(Path("config.json")) == ("data", "json")
|
||||
assert detect_type(Path("stack.yaml")) == ("data", "yaml")
|
||||
assert detect_type(Path("stack.yml")) == ("data", "yaml")
|
||||
assert detect_type(Path("pyproject.toml")) == ("data", "toml")
|
||||
for name in ("a.json", "b.yaml", "c.yml", "d.toml"):
|
||||
assert is_supported(Path(name))
|
||||
|
||||
|
||||
def test_minified_json_is_pretty_printed():
|
||||
minified = json.dumps({"hosts": [{"name": "web1", "ip": "10.0.0.1"}]})
|
||||
assert "\n" not in minified
|
||||
chunks = chunk_data(minified, "json")
|
||||
assert len(chunks) == 1
|
||||
text = chunks[0]["text"]
|
||||
assert "\n" in text, "expected pretty-printed multi-line JSON"
|
||||
assert '"name": "web1"' in text
|
||||
|
||||
|
||||
def test_large_minified_json_multi_chunks():
|
||||
big = json.dumps([{"id": i, "payload": "x" * 100} for i in range(200)])
|
||||
chunks = chunk_data(big, "json", max_tokens=256)
|
||||
assert len(chunks) > 1, "large JSON must split into multiple chunks"
|
||||
# Pretty-printing means chunks break on lines, not mid-token blobs.
|
||||
for c in chunks:
|
||||
assert c["text"].strip()
|
||||
assert [c["chunk_index"] for c in chunks] == list(range(len(chunks)))
|
||||
|
||||
|
||||
def test_malformed_json_ingests_raw():
|
||||
broken = '{"unterminated": [1, 2'
|
||||
chunks = chunk_data(broken, "json")
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0]["text"] == broken
|
||||
|
||||
|
||||
def test_yaml_passes_through_unchanged():
|
||||
yaml_text = "services:\n web:\n image: nginx\n"
|
||||
chunks = chunk_data(yaml_text, "yaml")
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0]["text"] == yaml_text.strip()
|
||||
|
||||
|
||||
def test_toml_passes_through():
|
||||
toml_text = '[tool.example]\nname = "kb"\n'
|
||||
chunks = chunk_data(toml_text, "toml")
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0]["text"] == toml_text.strip()
|
||||
|
||||
|
||||
def test_empty_file_yields_no_chunks():
|
||||
assert chunk_data("", "json") == []
|
||||
assert chunk_data(" \n ", "yaml") == []
|
||||
@@ -0,0 +1,273 @@
|
||||
"""Tests for hybrid search: explain traces, document_id, and RRF merging."""
|
||||
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from kb.database import (
|
||||
get_connection,
|
||||
init_schema,
|
||||
insert_chunk,
|
||||
insert_document,
|
||||
insert_embedding,
|
||||
tag_document,
|
||||
)
|
||||
from kb.search import _blend_rerank, _rank_by_score, _rrf_merge, hybrid_search
|
||||
|
||||
DIM = 4
|
||||
|
||||
# Chunk vectors are axis-aligned so we can steer vector ranking exactly:
|
||||
# a query of [1,0,0,0] has distance 0 to chunk A, sqrt(2) to chunk B.
|
||||
VEC_A = [1.0, 0.0, 0.0, 0.0]
|
||||
VEC_B = [0.0, 1.0, 0.0, 0.0]
|
||||
QUERY_VEC = [1.0, 0.0, 0.0, 0.0]
|
||||
|
||||
|
||||
class _Cfg:
|
||||
search_threshold = 0.0
|
||||
rerank_enabled = False
|
||||
reranker_model = "test-reranker"
|
||||
rerank_candidates = 40
|
||||
|
||||
|
||||
class _Db:
|
||||
def __init__(self, conn, ids):
|
||||
self.conn = conn
|
||||
self.ids = ids
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db(tmp_path, monkeypatch):
|
||||
"""Real schema (FTS5 + sqlite-vec) with two docs and stubbed embeddings."""
|
||||
fake = types.ModuleType("kb.embeddings")
|
||||
fake.embed_texts = lambda texts: [QUERY_VEC for _ in texts]
|
||||
monkeypatch.setitem(sys.modules, "kb.embeddings", fake)
|
||||
|
||||
conn = get_connection(str(tmp_path / "kb.db"))
|
||||
init_schema(conn, embedding_dim=DIM)
|
||||
|
||||
doc_a = insert_document(conn, "Alpha doc", "/src/a.md", "hash-a", "markdown")
|
||||
doc_b = insert_document(conn, "Bravo doc", "/src/b.md", "hash-b", "markdown")
|
||||
chunk_a = insert_chunk(conn, doc_a, 0, "alpha network switch configuration")
|
||||
chunk_b = insert_chunk(conn, doc_b, 0, "bravo unrelated cooking recipe")
|
||||
insert_embedding(conn, chunk_a, VEC_A)
|
||||
insert_embedding(conn, chunk_b, VEC_B)
|
||||
tag_document(conn, doc_a, ["ops"])
|
||||
|
||||
ids = {"doc_a": doc_a, "doc_b": doc_b, "chunk_a": chunk_a, "chunk_b": chunk_b}
|
||||
yield _Db(conn, ids)
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_results_include_document_id(db):
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg())
|
||||
assert result["results"], "expected at least one hit"
|
||||
top_hit = result["results"][0]
|
||||
assert top_hit["document_id"] == db.ids["doc_a"]
|
||||
assert top_hit["chunk_id"] == db.ids["chunk_a"]
|
||||
|
||||
|
||||
def test_explain_absent_by_default(db):
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg())
|
||||
assert all("explain" not in r for r in result["results"])
|
||||
|
||||
|
||||
def test_explain_hybrid_breakdown(db):
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), explain=True)
|
||||
top_hit = result["results"][0]
|
||||
exp = top_hit["explain"]
|
||||
|
||||
# Chunk A is rank 1 in both arms: FTS matches "alpha"/"switch", vector
|
||||
# distance is 0 (similarity 1.0).
|
||||
assert exp["fts_rank"] == 1
|
||||
assert exp["vec_rank"] == 1
|
||||
assert exp["fts_score"] > 0
|
||||
assert exp["vec_score"] == pytest.approx(1.0)
|
||||
assert exp["rrf_fts"] == pytest.approx(1.0 / 61, abs=1e-6)
|
||||
assert exp["rrf_vec"] == pytest.approx(1.0 / 61, abs=1e-6)
|
||||
assert exp["final_score"] == pytest.approx(
|
||||
exp["rrf_fts"] + exp["rrf_vec"] + exp["bonus"], abs=1e-5
|
||||
)
|
||||
assert exp["final_score"] == pytest.approx(top_hit["score"], abs=1e-5)
|
||||
|
||||
|
||||
def test_explain_single_arm_when_vec_misses(db):
|
||||
"""A chunk found only by vector search has null FTS fields."""
|
||||
result = hybrid_search(db.conn, "zzz-no-fts-match", _Cfg(), explain=True)
|
||||
for r in result["results"]:
|
||||
exp = r["explain"]
|
||||
assert exp["fts_score"] is None
|
||||
assert exp["fts_rank"] is None
|
||||
assert exp["rrf_fts"] is None
|
||||
assert exp["vec_rank"] is not None
|
||||
|
||||
|
||||
def test_explain_fts_only_shape(db):
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), fts_only=True, explain=True)
|
||||
top_hit = result["results"][0]
|
||||
exp = top_hit["explain"]
|
||||
assert exp["fts_rank"] == 1
|
||||
assert exp["final_score"] == pytest.approx(exp["fts_score"])
|
||||
assert "vec_score" not in exp
|
||||
|
||||
|
||||
def test_explain_vec_only_shape(db):
|
||||
result = hybrid_search(db.conn, "anything", _Cfg(), vec_only=True, explain=True)
|
||||
top_hit = result["results"][0]
|
||||
exp = top_hit["explain"]
|
||||
assert exp["vec_rank"] == 1
|
||||
assert exp["final_score"] == pytest.approx(exp["vec_score"])
|
||||
assert "fts_score" not in exp
|
||||
|
||||
|
||||
def test_rrf_merge_arithmetic():
|
||||
fts = {1: 10.0, 2: 5.0}
|
||||
vec = {2: 0.9, 3: 0.8}
|
||||
scores, details = _rrf_merge(fts, vec)
|
||||
by_id = dict(scores)
|
||||
|
||||
# fts rank 1 → 1/61 + 0.05 bonus
|
||||
assert by_id[1] == pytest.approx(1 / 61 + 0.05)
|
||||
# fts rank 2 (+0.02) and vec rank 1 (+0.05) — bonuses stack across arms
|
||||
assert by_id[2] == pytest.approx(1 / 62 + 1 / 61 + 0.07)
|
||||
# vec rank 2 → 1/62 + 0.02
|
||||
assert by_id[3] == pytest.approx(1 / 62 + 0.02)
|
||||
# Chunk 2 appears in both arms, so it must win.
|
||||
assert scores[0][0] == 2
|
||||
|
||||
assert details[2]["fts_rank"] == 2
|
||||
assert details[2]["vec_rank"] == 1
|
||||
assert details[2]["bonus"] == pytest.approx(0.07)
|
||||
assert details[1]["vec_rank"] is None
|
||||
assert details[3]["rrf_fts"] is None
|
||||
|
||||
|
||||
def test_top_rank_bonus_tiers():
|
||||
from kb.search import _top_rank_bonus
|
||||
|
||||
assert _top_rank_bonus(1) == 0.05
|
||||
assert _top_rank_bonus(2) == 0.02
|
||||
assert _top_rank_bonus(3) == 0.02
|
||||
assert _top_rank_bonus(4) == 0.0
|
||||
assert _top_rank_bonus(None) == 0.0
|
||||
|
||||
|
||||
def test_bonus_preserves_top_of_arm():
|
||||
"""A chunk at rank 1 of one arm beats a chunk at mid-rank in both arms."""
|
||||
fts = {10: 100.0, 11: 90.0, 12: 80.0, 13: 70.0, 14: 60.0}
|
||||
vec = {20: 0.9, 11: 0.8, 12: 0.7, 13: 0.6, 14: 0.5}
|
||||
scores, _ = _rrf_merge(fts, vec)
|
||||
order = [cid for cid, _ in scores]
|
||||
# Chunk 10 (fts #1, absent from vec): 1/61 + 0.05 ≈ 0.0664.
|
||||
# Chunk 13 (rank 4 in fts, rank 4 in vec): 2/64 ≈ 0.031, no bonus.
|
||||
assert order.index(10) < order.index(13)
|
||||
|
||||
|
||||
def test_rank_by_score():
|
||||
assert _rank_by_score({7: 0.5, 8: 0.9, 9: 0.1}) == {8: 1, 7: 2, 9: 3}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reranking
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _enable_fake_reranker(monkeypatch, score_fn):
|
||||
from kb import reranker
|
||||
|
||||
monkeypatch.setattr(reranker, "is_available", lambda: True)
|
||||
monkeypatch.setattr(reranker, "rerank_scores", score_fn)
|
||||
|
||||
|
||||
def test_rerank_flags_response_and_explain(db, monkeypatch):
|
||||
_enable_fake_reranker(monkeypatch, lambda q, texts: [0.9] * len(texts))
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), explain=True, rerank=True)
|
||||
|
||||
assert result["reranked"] is True
|
||||
top_hit = result["results"][0]
|
||||
exp = top_hit["explain"]
|
||||
assert exp["rerank_score"] == pytest.approx(0.9)
|
||||
assert exp["pre_rerank_rank"] == 1
|
||||
assert exp["blend_weight"] == 0.75
|
||||
assert exp["final_score"] == pytest.approx(top_hit["score"], abs=1e-5)
|
||||
# rank 1: 75% retrieval (norm=1.0 for the top candidate) + 25% rerank
|
||||
assert top_hit["score"] == pytest.approx(0.75 * 1.0 + 0.25 * 0.9, abs=1e-5)
|
||||
|
||||
|
||||
def test_rerank_can_reorder(db, monkeypatch):
|
||||
"""A strong rerank score rescues a lower-retrieval-ranked chunk."""
|
||||
|
||||
def favour_chunk_b(query, texts):
|
||||
return [1.0 if "cooking" in t else 0.0 for t in texts]
|
||||
|
||||
_enable_fake_reranker(monkeypatch, favour_chunk_b)
|
||||
# Neutral-ish query: both chunks retrieved, chunk A ranked first.
|
||||
result = hybrid_search(db.conn, "alpha cooking", _Cfg(), rerank=True)
|
||||
assert result["reranked"] is True
|
||||
assert len(result["results"]) == 2
|
||||
|
||||
|
||||
def test_rerank_false_bypasses(db, monkeypatch):
|
||||
called = []
|
||||
_enable_fake_reranker(monkeypatch, lambda q, t: called.append(1) or [0.5] * len(t))
|
||||
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), rerank=False)
|
||||
assert result["reranked"] is False
|
||||
assert not called
|
||||
|
||||
|
||||
def test_rerank_unavailable_degrades_gracefully(db):
|
||||
# No reranker loaded: rerank=True must not error.
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), rerank=True, explain=True)
|
||||
assert result["reranked"] is False
|
||||
assert "rerank_score" not in result["results"][0]["explain"]
|
||||
|
||||
|
||||
def test_rerank_skipped_for_single_arm(db, monkeypatch):
|
||||
called = []
|
||||
_enable_fake_reranker(monkeypatch, lambda q, t: called.append(1) or [0.5] * len(t))
|
||||
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), fts_only=True, rerank=True)
|
||||
assert result["reranked"] is False
|
||||
assert not called
|
||||
|
||||
|
||||
def test_rerank_default_follows_cfg(db, monkeypatch):
|
||||
_enable_fake_reranker(monkeypatch, lambda q, texts: [0.5] * len(texts))
|
||||
|
||||
class _RerankCfg(_Cfg):
|
||||
rerank_enabled = True
|
||||
|
||||
result = hybrid_search(db.conn, "alpha switch", _RerankCfg())
|
||||
assert result["reranked"] is True
|
||||
|
||||
|
||||
def test_blend_rerank_weights_by_position():
|
||||
# 12 candidates, retrieval scores 12 down to 1 → norms 1.0 down to 0.0.
|
||||
candidates = [(cid, float(12 - i)) for i, cid in enumerate(range(100, 112))]
|
||||
details = {cid: {} for cid, _ in candidates}
|
||||
rr = [1.0] * len(candidates)
|
||||
|
||||
blended = dict(_blend_rerank(candidates, rr, details))
|
||||
|
||||
assert details[100]["blend_weight"] == 0.75 # rank 1
|
||||
assert details[102]["blend_weight"] == 0.75 # rank 3
|
||||
assert details[103]["blend_weight"] == 0.60 # rank 4
|
||||
assert details[109]["blend_weight"] == 0.60 # rank 10
|
||||
assert details[110]["blend_weight"] == 0.40 # rank 11
|
||||
|
||||
# rank 1: norm 1.0 → 0.75*1.0 + 0.25*1.0 = 1.0
|
||||
assert blended[100] == pytest.approx(1.0)
|
||||
# rank 4: norm 8/11 → 0.6*(8/11) + 0.4*1.0
|
||||
assert blended[103] == pytest.approx(0.6 * (8 / 11) + 0.4)
|
||||
# rank 11: norm 1/11 → 0.4*(1/11) + 0.6*1.0
|
||||
assert blended[110] == pytest.approx(0.4 * (1 / 11) + 0.6)
|
||||
|
||||
|
||||
def test_blend_rerank_constant_retrieval_scores():
|
||||
"""Zero span (all candidates same retrieval score) must not divide by zero."""
|
||||
candidates = [(1, 0.5), (2, 0.5)]
|
||||
details = {1: {}, 2: {}}
|
||||
blended = dict(_blend_rerank(candidates, [0.2, 0.8], details))
|
||||
assert blended[1] == pytest.approx(0.75 * 1.0 + 0.25 * 0.2)
|
||||
assert blended[2] == pytest.approx(0.75 * 1.0 + 0.25 * 0.8)
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Tests for tag context descriptions."""
|
||||
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from kb.database import (
|
||||
get_connection,
|
||||
init_schema,
|
||||
insert_chunk,
|
||||
insert_document,
|
||||
insert_embedding,
|
||||
tag_document,
|
||||
)
|
||||
from kb.search import hybrid_search
|
||||
|
||||
DIM = 4
|
||||
|
||||
|
||||
class _Cfg:
|
||||
search_threshold = 0.0
|
||||
rerank_enabled = False
|
||||
reranker_model = "test-reranker"
|
||||
rerank_candidates = 40
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def conn(tmp_path, monkeypatch):
|
||||
fake = types.ModuleType("kb.embeddings")
|
||||
fake.embed_texts = lambda texts: [[1.0, 0.0, 0.0, 0.0] for _ in texts]
|
||||
monkeypatch.setitem(sys.modules, "kb.embeddings", fake)
|
||||
|
||||
conn = get_connection(str(tmp_path / "kb.db"))
|
||||
init_schema(conn, embedding_dim=DIM)
|
||||
yield conn
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_migration_is_idempotent(conn):
|
||||
# Running init_schema again (fresh start on an existing DB) must not fail.
|
||||
init_schema(conn, embedding_dim=DIM)
|
||||
cols = {row[1] for row in conn.execute("PRAGMA table_info(tags)").fetchall()}
|
||||
assert "description" in cols
|
||||
|
||||
|
||||
def test_search_results_carry_tag_contexts(conn):
|
||||
doc = insert_document(conn, "Runbook", "/notes/runbook.md", "h1", "markdown")
|
||||
chunk = insert_chunk(conn, doc, 0, "restart the proxy after cert renewal")
|
||||
insert_embedding(conn, chunk, [1.0, 0.0, 0.0, 0.0])
|
||||
tag_document(conn, doc, ["ops", "draft"])
|
||||
conn.execute(
|
||||
"UPDATE tags SET description = ? WHERE name = ?",
|
||||
("Lab operations runbooks", "ops"),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
result = hybrid_search(conn, "restart proxy", _Cfg())
|
||||
hit = result["results"][0]
|
||||
assert hit["tags"] == ["draft", "ops"]
|
||||
# Only described tags appear in tag_contexts.
|
||||
assert hit["tag_contexts"] == {"ops": "Lab operations runbooks"}
|
||||
|
||||
|
||||
def test_tag_contexts_empty_when_no_descriptions(conn):
|
||||
doc = insert_document(conn, "Plain", "/notes/plain.md", "h2", "markdown")
|
||||
chunk = insert_chunk(conn, doc, 0, "some plain text about switches")
|
||||
insert_embedding(conn, chunk, [1.0, 0.0, 0.0, 0.0])
|
||||
tag_document(conn, doc, ["misc"])
|
||||
|
||||
result = hybrid_search(conn, "switches", _Cfg())
|
||||
hit = result["results"][0]
|
||||
assert hit["tag_contexts"] == {}
|
||||
+6
-1
@@ -18,7 +18,8 @@ def _client() -> httpx.Client:
|
||||
|
||||
def search(query: str, top: int = 10, tags: list[str] | None = None,
|
||||
doc_type: str | None = None, fts_only: bool = False,
|
||||
vec_only: bool = False, threshold: float | None = None) -> dict:
|
||||
vec_only: bool = False, threshold: float | None = None,
|
||||
explain: bool = False, rerank: bool | None = None) -> dict:
|
||||
body: dict = {"query": query, "top": top}
|
||||
if tags:
|
||||
body["tags"] = tags
|
||||
@@ -30,6 +31,10 @@ def search(query: str, top: int = 10, tags: list[str] | None = None,
|
||||
body["vec_only"] = True
|
||||
if threshold is not None:
|
||||
body["threshold"] = threshold
|
||||
if explain:
|
||||
body["explain"] = True
|
||||
if rerank is not None:
|
||||
body["rerank"] = rerank
|
||||
with _client() as c:
|
||||
r = c.post("/api/v1/search", json=body)
|
||||
r.raise_for_status()
|
||||
|
||||
+25
-7
@@ -48,9 +48,11 @@ mcp = FastMCP(
|
||||
"kb_search uses dense vector embeddings (semantic similarity) fused with "
|
||||
"BM25 full-text ranking, so it finds conceptually related content even "
|
||||
"when the exact words don't match — agents can ask natural-language "
|
||||
"questions rather than guessing keywords. Also provides tools for adding "
|
||||
"notes, uploading files, and managing documents and tags. Use tags to "
|
||||
"organise and filter documents (e.g. tag notes with 'agent:mybot' and "
|
||||
"questions rather than guessing keywords. When the engine has a "
|
||||
"cross-encoder reranker enabled, results are reranked server-side by "
|
||||
"default (pass rerank=False for lower latency). Also provides tools for "
|
||||
"adding notes, uploading files, and managing documents and tags. Use tags "
|
||||
"to organise and filter documents (e.g. tag notes with 'agent:mybot' and "
|
||||
"filter searches by that tag). This server requires Bearer token "
|
||||
"authentication — all requests are authenticated via the Authorization "
|
||||
"header at the HTTP transport layer."
|
||||
@@ -66,6 +68,8 @@ async def kb_search(
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
fts_only: bool = False,
|
||||
explain: bool = False,
|
||||
rerank: bool | None = None,
|
||||
) -> str:
|
||||
"""Hybrid semantic (vector) + full-text search over the knowledge base.
|
||||
|
||||
@@ -75,6 +79,11 @@ async def kb_search(
|
||||
ask natural-language questions ("what did we decide about X?") rather than
|
||||
guessing the exact keywords used in the source documents.
|
||||
|
||||
When the engine has a cross-encoder reranker enabled, the top candidates
|
||||
are reranked server-side by default — you normally do NOT need to rerank
|
||||
results yourself. Check kb_status's "rerank" block to see whether it is
|
||||
active.
|
||||
|
||||
Returns ranked chunks matching the query, with text content, relevance
|
||||
scores, and document metadata.
|
||||
|
||||
@@ -82,18 +91,25 @@ async def kb_search(
|
||||
query: The search query — a natural language question or keywords.
|
||||
top: Maximum number of results to return (default 10).
|
||||
tags: Filter results to documents with ALL of these tags.
|
||||
doc_type: Filter by document type (e.g. "note", "pdf", "markdown", "code").
|
||||
doc_type: Filter by document type (e.g. "note", "pdf", "markdown",
|
||||
"code", "data").
|
||||
fts_only: Disable the vector/semantic component and use only BM25
|
||||
keyword matching. Default false (hybrid mode). Set true only when
|
||||
you need exact-string matching (e.g. an error code, identifier).
|
||||
explain: Include a per-result score breakdown (BM25 score/rank, vector
|
||||
similarity/rank, rank-fusion contributions, rerank blend) under an
|
||||
"explain" key. Useful for diagnosing why a result ranked where it did.
|
||||
rerank: Set false to skip server-side reranking for lower latency.
|
||||
Default (None) uses the engine's configured behaviour.
|
||||
|
||||
Tips for complex queries:
|
||||
- Consider expanding into 2-3 variant phrasings and calling this tool multiple
|
||||
times, then deduplicating results by chunk_id. For example, search for both
|
||||
"pension revaluation rules" and "how are pensions revalued" to cast a wider net.
|
||||
- For precision, rerank the returned results using your own judgement based on
|
||||
relevance to the original question.
|
||||
- Call kb_status to see which embedding model is in use.
|
||||
- If the engine's reranker is disabled, you can still rerank the returned
|
||||
results yourself using your own judgement of relevance to the question.
|
||||
- Call kb_status to see which embedding model is in use and whether
|
||||
server-side reranking is active.
|
||||
"""
|
||||
result = engine.search(
|
||||
query=query,
|
||||
@@ -101,6 +117,8 @@ async def kb_search(
|
||||
tags=tags or None,
|
||||
doc_type=doc_type,
|
||||
fts_only=fts_only,
|
||||
explain=explain,
|
||||
rerank=rerank,
|
||||
)
|
||||
|
||||
results_list = result if isinstance(result, list) else result.get("results", [])
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
Documents recommended patterns for agent-side query expansion and reranking, which are caller responsibilities rather than engine features. These patterns are communicated via MCP tool descriptions.
|
||||
Documents recommended patterns for agent-side query expansion, plus how agent guidance interacts with the engine's optional server-side reranking. These patterns are communicated via MCP tool descriptions.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -18,18 +18,26 @@ The `kb_search` MCP tool description SHALL include guidance on query expansion a
|
||||
|
||||
### Requirement: Reranking guidance in tool description
|
||||
|
||||
The `kb_search` MCP tool description SHALL include guidance on agent-side reranking as a recommended pattern for improving precision.
|
||||
The `kb_search` MCP tool description SHALL describe the engine's server-side reranking behaviour and retain agent-side reranking as a fallback pattern.
|
||||
|
||||
#### Scenario: Tool description includes reranking pattern
|
||||
#### Scenario: Tool description covers server-side reranking
|
||||
- **WHEN** an agent reads the `kb_search` tool description
|
||||
- **THEN** the description SHALL include guidance such as: "For precision, rerank the returned results using your own judgement based on relevance to the original question"
|
||||
- **THEN** the description SHALL state that results are reranked server-side by default when the engine has a reranker enabled, that `rerank=False` skips it for lower latency, and that `kb_status` reports whether reranking is active
|
||||
|
||||
#### Scenario: Tool description retains agent-side fallback
|
||||
- **WHEN** an agent reads the `kb_search` tool description
|
||||
- **THEN** the description SHALL include guidance that, when the engine's reranker is disabled, the agent can rerank the returned results using its own judgement of relevance to the original question
|
||||
|
||||
---
|
||||
|
||||
### Requirement: No engine-side LLM dependency
|
||||
### Requirement: No external LLM dependency
|
||||
|
||||
The engine SHALL NOT require or use any external LLM API for search operations. Query expansion and reranking SHALL remain entirely agent-side concerns.
|
||||
The engine SHALL NOT require or use any external LLM API for search operations. Query expansion SHALL remain an agent-side concern. Reranking MAY be performed engine-side using a local, opt-in cross-encoder model; it SHALL never depend on an external API.
|
||||
|
||||
#### Scenario: Engine has no LLM dependency
|
||||
#### Scenario: Engine has no external LLM dependency
|
||||
- **WHEN** the engine is deployed without any `ANTHROPIC_API_KEY` or similar LLM API configuration
|
||||
- **THEN** all search operations SHALL function fully, with no degraded results or missing features
|
||||
|
||||
#### Scenario: Reranking is optional and degrades gracefully
|
||||
- **WHEN** the engine is deployed with `KB_RERANK_ENABLED` unset or false, or the reranker model fails to load
|
||||
- **THEN** all search operations SHALL function fully using hybrid retrieval alone, with responses reporting `"reranked": false`
|
||||
|
||||
@@ -26,11 +26,15 @@ The engine SHALL load the embedding model eagerly at startup before accepting HT
|
||||
|
||||
### Requirement: Hybrid search
|
||||
|
||||
The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5) and vector similarity search (via sqlite-vec), merged using Reciprocal Rank Fusion. Search SHALL complete in under 100ms when the model is warm. The engine SHALL sanitize user query strings to prevent FTS5 syntax errors for any input.
|
||||
The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5) and vector similarity search (via sqlite-vec), merged using Reciprocal Rank Fusion with a top-rank bonus (+0.05 for rank 1, +0.02 for ranks 2-3 in either arm) that preserves exact matches. Search SHALL complete in under 100ms when the model is warm and reranking is disabled; reranked searches SHALL complete in under 500ms on GPU. The engine SHALL sanitize user query strings to prevent FTS5 syntax errors for any input.
|
||||
|
||||
#### Scenario: Hybrid search with results
|
||||
- **WHEN** a client sends `POST /api/v1/search` with body `{"query": "how to change oil", "top": 5}`
|
||||
- **THEN** the engine SHALL embed the query using the resident model, run both FTS5 and vector searches, merge results via RRF, and return a JSON response with matched chunks including scores, document metadata, and tags
|
||||
- **THEN** the engine SHALL embed the query using the resident model, run both FTS5 and vector searches, merge results via RRF with top-rank bonus, and return a JSON response with matched chunks including scores, `document_id`, document metadata, tags, and `tag_contexts`
|
||||
|
||||
#### Scenario: Explain traces
|
||||
- **WHEN** a client sends `POST /api/v1/search` with `"explain": true`
|
||||
- **THEN** each result SHALL include an `explain` object with per-arm raw scores and ranks (`fts_score`, `fts_rank`, `vec_score`, `vec_rank`), RRF contributions (`rrf_fts`, `rrf_vec`), the top-rank `bonus`, rerank blend fields when reranking ran (`pre_rerank_rank`, `retrieval_norm`, `rerank_score`, `blend_weight`), and the `final_score`; fields for an arm that did not match SHALL be null
|
||||
|
||||
#### Scenario: Search with filters
|
||||
- **WHEN** a client sends `POST /api/v1/search` with body `{"query": "brakes", "tags": ["maintenance"], "doc_type": "pdf", "top": 3}`
|
||||
@@ -62,6 +66,28 @@ The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Cross-encoder reranking
|
||||
|
||||
The engine SHALL support optional server-side reranking of hybrid search results using a local cross-encoder model, enabled via `KB_RERANK_ENABLED` (default false) with the model set by `KB_RERANKER_MODEL` (default `BAAI/bge-reranker-v2-m3`). When active, the engine SHALL over-fetch candidates (`KB_RERANK_CANDIDATES`, default 40), score each (query, chunk) pair, and blend retrieval and rerank scores position-aware: 75% retrieval weight for pre-rerank ranks 1-3, 60% for 4-10, 40% for 11+, with retrieval scores min-max normalised over the candidate set. Blended scores are on a 0-1 scale distinct from RRF scores; the score threshold SHALL be applied before reranking only. Every search response SHALL include a top-level `"reranked"` boolean.
|
||||
|
||||
#### Scenario: Reranked search
|
||||
- **WHEN** reranking is enabled with a loaded model and a client sends a hybrid search
|
||||
- **THEN** the engine SHALL rerank the top candidates and return results ordered by blended score with `"reranked": true`
|
||||
|
||||
#### Scenario: Per-request opt-out
|
||||
- **WHEN** a client sends `POST /api/v1/search` with `"rerank": false`
|
||||
- **THEN** the engine SHALL skip reranking and return plain hybrid results with `"reranked": false`
|
||||
|
||||
#### Scenario: Graceful degradation
|
||||
- **WHEN** reranking is requested but the model is disabled or failed to load
|
||||
- **THEN** the engine SHALL return plain hybrid results with `"reranked": false` and no error
|
||||
|
||||
#### Scenario: Single-arm searches never rerank
|
||||
- **WHEN** a client sends a search with `fts_only` or `vec_only` set
|
||||
- **THEN** the engine SHALL NOT rerank, keeping single-arm results pure for benchmarking
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Async ingestion via job queue
|
||||
|
||||
The engine SHALL accept file uploads and text notes for ingestion asynchronously. Uploaded content SHALL be written to a staging area and a job record created in the database. The engine SHALL return HTTP 202 immediately. A background worker SHALL process queued jobs sequentially. Before staging, the engine SHALL compute a SHA256 hash of the uploaded content and reject duplicates immediately.
|
||||
@@ -128,7 +154,7 @@ The engine SHALL maintain job records in SQLite with status tracking. Jobs SHALL
|
||||
|
||||
### Requirement: Background ingestion worker
|
||||
|
||||
The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.
|
||||
The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes, fixed-size text chunking for data files with minified JSON pretty-printed first), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.
|
||||
|
||||
#### Scenario: Successful PDF ingestion
|
||||
- **WHEN** the background worker picks up a queued PDF job
|
||||
@@ -216,7 +242,7 @@ The engine SHALL provide endpoints to list all tags and manage tags on documents
|
||||
|
||||
#### Scenario: List all tags
|
||||
- **WHEN** a client sends `GET /api/v1/tags`
|
||||
- **THEN** the engine SHALL return a JSON array of tags with name and document count
|
||||
- **THEN** the engine SHALL return a JSON array of tags with name, document count, and description (null when unset)
|
||||
|
||||
#### Scenario: Add tags to a document
|
||||
- **WHEN** a client sends `PUT /api/v1/documents/{id}/tags` with body `{"add": ["manual", "v2"]}`
|
||||
@@ -228,13 +254,35 @@ The engine SHALL provide endpoints to list all tags and manage tags on documents
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Tag context descriptions
|
||||
|
||||
The engine SHALL support a one-line context description per tag, stored in a `description` column on the tags table (added via idempotent migration). Search results SHALL include a `tag_contexts` object mapping each of the document's described tags to its description, so consumers can judge which similar-scoring chunks answer the question.
|
||||
|
||||
#### Scenario: Set a tag description
|
||||
- **WHEN** a client sends `PUT /api/v1/tags/{name}/description` with body `{"description": "Lab operations runbooks"}`
|
||||
- **THEN** the engine SHALL store the description (matching the tag name case-insensitively) and return `{"name": "<name>", "description": "<description>"}`
|
||||
|
||||
#### Scenario: Clear a tag description
|
||||
- **WHEN** a client sends `PUT /api/v1/tags/{name}/description` with a null or empty description
|
||||
- **THEN** the engine SHALL clear the stored description
|
||||
|
||||
#### Scenario: Unknown tag
|
||||
- **WHEN** a client sets a description for a tag that does not exist
|
||||
- **THEN** the engine SHALL return HTTP 404
|
||||
|
||||
#### Scenario: Descriptions in search results
|
||||
- **WHEN** a search result's document carries tags and at least one tag has a description
|
||||
- **THEN** the result SHALL include `tag_contexts` with only the described tags; results with no described tags SHALL include an empty `tag_contexts` object
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Engine status and reindex
|
||||
|
||||
The engine SHALL provide status information and support re-embedding all chunks. The `version` field in the status response SHALL always be present and SHALL reflect the engine's release version as read from the `VERSION` file. This field is the contract used by clients for compatibility checking.
|
||||
|
||||
#### Scenario: Get engine status
|
||||
- **WHEN** a client sends `GET /api/v1/status`
|
||||
- **THEN** the engine SHALL return JSON with `version` (string, from VERSION file), model_name, embedding_dim, GPU device info, database stats (document count by type, total chunks, DB size), and queue stats (queued/processing job count)
|
||||
- **THEN** the engine SHALL return JSON with `version` (string, from VERSION file), model_name, embedding_dim, GPU device info, database stats (document count by type, total chunks, DB size), queue stats (queued/processing job count), and a `rerank` object with `enabled`, `model`, `loaded`, and `candidates`
|
||||
|
||||
#### Scenario: Trigger reindex
|
||||
- **WHEN** a client sends `POST /api/v1/reindex`
|
||||
|
||||
Reference in New Issue
Block a user