Compare commits
36 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 5b5a9ecbc3 | |||
| 932e889ee8 | |||
| 48ca873fe2 | |||
| a38d77ed23 | |||
| f1ed5b6e23 | |||
| 3ab8a81c14 | |||
| 739c3ff30c | |||
| 3151a1b08a | |||
| 5049ba2a2f | |||
| 6dfc13be1d | |||
| 75e4a0cf73 | |||
| 753c641e72 | |||
| 45e2c5ce91 | |||
| e6e91f1d5c | |||
| 9eccc527ae | |||
| d44d11e4fe | |||
| 574370e8d1 | |||
| 17b19999de | |||
| bb78f4ea80 | |||
| 223ff2cf5d | |||
| e9a282ddb1 | |||
| b5a203d2aa | |||
| 0c124c4ab7 | |||
| da5b8435bc | |||
| e39e00a2c0 | |||
| d078af9ad3 | |||
| b3dce188e1 | |||
| 0dc3065979 | |||
| e7136a4a20 | |||
| adeba21712 | |||
| 2d179af557 | |||
| a6bab5e55e | |||
| c5191df9c0 | |||
| afbe270181 | |||
| 9e957f1a9a | |||
| bbe6a5e909 |
@@ -0,0 +1,81 @@
|
||||
name: Rebuild Docker images
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: rebuild-docker-images
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
rebuild-and-push:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
REGISTRY: docker.dcglab.co.uk
|
||||
IMAGE_BASE: docker.dcglab.co.uk/public/kb
|
||||
REGISTRY_USERNAME: ${{ secrets.DOCKER_DCGLAB_CI_USERNAME }}
|
||||
REGISTRY_PASSWORD: ${{ secrets.DOCKER_DCGLAB_CI_PASSWORD }}
|
||||
steps:
|
||||
- name: Check out repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Log in to registry
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
test -n "$REGISTRY_USERNAME" || { echo "DOCKER_DCGLAB_CI_USERNAME is not available" >&2; exit 1; }
|
||||
test -n "$REGISTRY_PASSWORD" || { echo "DOCKER_DCGLAB_CI_PASSWORD is not available" >&2; exit 1; }
|
||||
printf '%s' "$REGISTRY_PASSWORD" | docker login "$REGISTRY" --username "$REGISTRY_USERNAME" --password-stdin
|
||||
|
||||
- name: Build all images from scratch
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
version="$(tr -d '[:space:]' < engine/VERSION)"
|
||||
|
||||
docker build --pull --no-cache --provenance=false --sbom=false \
|
||||
-t "$IMAGE_BASE/engine:v${version}-nvidia" \
|
||||
-t "$IMAGE_BASE/engine:latest-nvidia" \
|
||||
-f engine/Dockerfile.nvidia engine
|
||||
|
||||
docker build --pull --no-cache --provenance=false --sbom=false \
|
||||
-t "$IMAGE_BASE/engine:v${version}-cpu" \
|
||||
-t "$IMAGE_BASE/engine:latest-cpu" \
|
||||
-f engine/Dockerfile.cpu engine
|
||||
|
||||
docker build --pull --no-cache --provenance=false --sbom=false \
|
||||
-t "$IMAGE_BASE/mcp:v${version}" \
|
||||
-t "$IMAGE_BASE/mcp:latest" \
|
||||
-f mcp/Dockerfile mcp
|
||||
|
||||
- name: Push and verify all tags
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
version="$(tr -d '[:space:]' < engine/VERSION)"
|
||||
images=(
|
||||
"$IMAGE_BASE/engine:v${version}-nvidia"
|
||||
"$IMAGE_BASE/engine:latest-nvidia"
|
||||
"$IMAGE_BASE/engine:v${version}-cpu"
|
||||
"$IMAGE_BASE/engine:latest-cpu"
|
||||
"$IMAGE_BASE/mcp:v${version}"
|
||||
"$IMAGE_BASE/mcp:latest"
|
||||
)
|
||||
|
||||
push_image() {
|
||||
local image="$1"
|
||||
local attempt
|
||||
for attempt in 1 2 3 4 5; do
|
||||
docker push "$image" && return 0
|
||||
if [[ "$attempt" -eq 5 ]]; then
|
||||
echo "Failed to push $image after $attempt attempts" >&2
|
||||
return 1
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
}
|
||||
|
||||
for image in "${images[@]}"; do
|
||||
push_image "$image"
|
||||
docker manifest inspect "$image" >/dev/null
|
||||
done
|
||||
@@ -1,2 +1,9 @@
|
||||
examples/
|
||||
.claude/
|
||||
__pycache__/
|
||||
engine/data/
|
||||
|
||||
TMP/
|
||||
.env
|
||||
.venv/
|
||||
test_mcp_client.py
|
||||
|
||||
+139
@@ -0,0 +1,139 @@
|
||||
# Developer Guide
|
||||
|
||||
Instructions for building from source, releasing, and contributing to kb.
|
||||
|
||||
## Building from source
|
||||
|
||||
### Engine
|
||||
|
||||
```bash
|
||||
cd engine
|
||||
|
||||
# NVIDIA GPU
|
||||
KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d
|
||||
```
|
||||
|
||||
### Client
|
||||
|
||||
```bash
|
||||
cd client
|
||||
make build # produces ./kb binary
|
||||
make all # or cross-compile: dist/kb-{os}-{arch}
|
||||
```
|
||||
|
||||
## Running tests
|
||||
|
||||
### Engine
|
||||
|
||||
Engine tests run against SQLite (with sqlite-vec) and stub out the embedding
|
||||
model, so they only need lightweight dependencies — no torch/docling install:
|
||||
|
||||
```bash
|
||||
uv venv /tmp/kb-test-venv
|
||||
uv pip install --python /tmp/kb-test-venv/bin/python pytest pytest-asyncio fastapi httpx sqlite-vec
|
||||
cd engine && /tmp/kb-test-venv/bin/python -m pytest
|
||||
```
|
||||
|
||||
### Client
|
||||
|
||||
```bash
|
||||
cd client && go test ./...
|
||||
```
|
||||
|
||||
## Search-quality benchmarking
|
||||
|
||||
`kb bench fixture.json` runs a fixture of queries with known-relevant documents
|
||||
against each backend (fts, vec, hybrid, hybrid+rerank) and reports precision@k,
|
||||
recall, and MRR. See `docs/bench-example.json` for the fixture format.
|
||||
|
||||
Run a bench before and after any ranking change (RRF weights, reranker, model
|
||||
swap) and compare — keep a 20-30 query fixture against your real corpus outside
|
||||
the repo.
|
||||
|
||||
## Building and releasing
|
||||
|
||||
Client and engine are versioned independently via `client/VERSION` and `engine/VERSION`. Each has its own release script and git tag prefix.
|
||||
|
||||
### Release client
|
||||
|
||||
```bash
|
||||
./release-client.sh --gitea # patch bump, release via Gitea
|
||||
./release-client.sh --github --minor # minor bump, release via GitHub
|
||||
./release-client.sh --gitea --no-increment # release current version as-is
|
||||
./release-client.sh --gitea --dry-run # preview without doing anything
|
||||
```
|
||||
|
||||
Creates tag `client-vX.Y.Z`, builds Go binaries for all platforms, and creates a Gitea/GitHub release with binaries attached.
|
||||
|
||||
The client embeds a `MinEngineVersion` (from `client/MIN_ENGINE_VERSION`) and will hard-fail if the connected engine is too old.
|
||||
|
||||
### Release engine
|
||||
|
||||
```bash
|
||||
./release-engine.sh --gitea # patch bump, release via Gitea
|
||||
./release-engine.sh --github --minor # minor bump, release via GitHub
|
||||
./release-engine.sh --gitea --no-increment # release current version as-is
|
||||
./release-engine.sh --gitea --dry-run # preview without doing anything
|
||||
```
|
||||
|
||||
Creates tag `engine-vX.Y.Z`, builds NVIDIA and CPU Docker images, creates a Gitea/GitHub release, and pushes images to the registry.
|
||||
|
||||
### Checking versions
|
||||
|
||||
```bash
|
||||
# Client
|
||||
kb --version
|
||||
|
||||
# Engine
|
||||
curl http://localhost:8000/api/v1/status | jq .version
|
||||
```
|
||||
|
||||
### Docker images
|
||||
|
||||
Images are pushed to `docker.dcglab.co.uk/public/kb/engine` with tags:
|
||||
|
||||
- `engine-v2.0.6-nvidia` / `engine-v2.0.6-cpu` — versioned
|
||||
- `latest-nvidia` / `latest-cpu` — latest release
|
||||
|
||||
The release script authenticates to the registry using the
|
||||
`DOCKER_DCGLAB_CI_USERNAME` and `DOCKER_DCGLAB_CI_PASSWORD` environment
|
||||
variables.
|
||||
|
||||
Override the registry and org via environment variables:
|
||||
|
||||
```bash
|
||||
REGISTRY=ghcr.io IMAGE_ORG=myorg ./release-engine.sh --github
|
||||
```
|
||||
|
||||
Pushes are retried on transient registry failures. The engine images carry a
|
||||
~5.6GB torch layer, and uploading it can fail with a 502 from the proxy in
|
||||
front of the registry (or a 500 on the manifest PUT that follows), which
|
||||
clears on a retry. Tune with:
|
||||
|
||||
```bash
|
||||
PUSH_RETRIES=8 PUSH_RETRY_DELAY=20 ./release-engine.sh --gitea
|
||||
```
|
||||
|
||||
## API reference
|
||||
|
||||
All endpoints are under `/api/v1/`. Requires `Authorization: Bearer <key>` header when `KB_API_KEY` is set.
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/health` | Health check (bypasses auth) |
|
||||
| `POST` | `/search` | Hybrid search (JSON body) |
|
||||
| `POST` | `/jobs` | Upload file/note for ingestion (multipart, returns 202 or 409 if duplicate) |
|
||||
| `GET` | `/jobs` | List ingestion jobs |
|
||||
| `GET` | `/jobs/{id}` | Job details |
|
||||
| `GET` | `/documents` | List documents |
|
||||
| `GET` | `/documents/{id}` | Document details with chunks |
|
||||
| `GET` | `/documents/{id}/file` | Download original file |
|
||||
| `DELETE` | `/documents/{id}` | Remove a document (and stored file) |
|
||||
| `PUT` | `/documents/{id}/tags` | Add/remove tags |
|
||||
| `GET` | `/tags` | List all tags (with descriptions) |
|
||||
| `PUT` | `/tags/{name}/description` | Set/clear a tag context description |
|
||||
| `GET` | `/status` | Engine status, GPU info, DB stats, rerank state |
|
||||
| `POST` | `/reindex` | Re-embed all chunks |
|
||||
| `POST` | `/bulk/delete` | Bulk delete documents by filter |
|
||||
| `POST` | `/bulk/tags` | Bulk add/remove tags by filter |
|
||||
| `POST` | `/bulk/set-tags` | Bulk replace tags by filter |
|
||||
@@ -0,0 +1,174 @@
|
||||
# MCP Server (Agent Integration)
|
||||
|
||||
The MCP server exposes kb operations as native MCP tools, so agents can search, add notes, upload files, and manage documents without shelling out to the CLI. `kb_search` is hybrid: dense vector embeddings (semantic similarity) fused with BM25 full-text ranking via Reciprocal Rank Fusion, so agents can ask natural-language questions and find conceptually related content even when the exact words don't match.
|
||||
|
||||
## Start the MCP server
|
||||
|
||||
The compose files include a `kb-mcp` service alongside the engine. Set `KB_MCP_API_KEY` to require Bearer token auth from connecting agents:
|
||||
|
||||
```bash
|
||||
KB_API_KEY=your-engine-key KB_MCP_API_KEY=your-agent-key \
|
||||
docker compose -f engine/compose.nvidia.yaml up -d
|
||||
```
|
||||
|
||||
Or run the MCP server standalone:
|
||||
|
||||
```bash
|
||||
docker run -d --name kb-mcp \
|
||||
-p 3000:3000 \
|
||||
-e KB_ENGINE_URL=http://your-engine-host:8000 \
|
||||
-e KB_API_KEY=your-engine-key \
|
||||
-e KB_MCP_API_KEY=your-agent-key \
|
||||
--restart unless-stopped \
|
||||
docker.dcglab.co.uk/public/kb/mcp:latest
|
||||
```
|
||||
|
||||
## MCP tools
|
||||
|
||||
| Tool | Description |
|
||||
|---|---|
|
||||
| `kb_search` | Hybrid semantic (vector) + full-text search with tag/type filters |
|
||||
| `kb_addnote` | Add a text note (queued for async ingestion) |
|
||||
| `kb_update_note` | Update an existing note in place |
|
||||
| `kb_get` | Get document details by ID or source path |
|
||||
| `kb_delete` | Permanently delete a document by ID |
|
||||
| `kb_status` | Engine health and statistics |
|
||||
| `kb_jobs` | Ingestion queue status |
|
||||
| `kb_upload_start` | Start a chunked file upload |
|
||||
| `kb_upload_chunk` | Upload a base64-encoded file chunk |
|
||||
| `kb_upload_finish` | Finish upload and submit for ingestion |
|
||||
| `kb_bulk_delete` | Delete multiple documents matching a filter |
|
||||
| `kb_bulk_tags` | Add/remove tags on multiple documents |
|
||||
| `kb_bulk_set_tags` | Replace all tags on multiple documents |
|
||||
|
||||
## Organising with tags
|
||||
|
||||
Use tags to separate agent data from user documents. For example, an agent can tag all its notes with `agent:mybot` and filter by that tag when searching. This is a naming convention — configure it in your agent's system prompt. No special server-side enforcement is needed.
|
||||
|
||||
Bulk tools accept filter-based selection (by tags, doc_type, ID list, or ID range) so agents can manage thousands of documents in a single call instead of looping. A safety threshold (default 70%, configurable via engine env var `KB_BULK_SAFETY_PERCENT`) prevents accidental mass operations unless `force: true` is set.
|
||||
|
||||
## MCP server configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `KB_ENGINE_URL` | `http://localhost:8000` | Engine API URL |
|
||||
| `KB_API_KEY` | (none) | Engine API key |
|
||||
| `KB_MCP_API_KEY` | (none) | Bearer token required from agents (disabled if unset) |
|
||||
| `KB_MCP_PORT` | `3000` | Port to listen on |
|
||||
|
||||
## Connecting AI coding tools
|
||||
|
||||
The kb MCP server uses **Streamable HTTP** transport at `http://your-host:3000/mcp`. Below are configuration examples for popular AI coding tools.
|
||||
|
||||
### Claude Code (CLI / Desktop / Web)
|
||||
|
||||
Add the server to your project or user settings:
|
||||
|
||||
```bash
|
||||
claude mcp add kb-server --transport http http://localhost:3000/mcp
|
||||
```
|
||||
|
||||
Or add it manually to `.claude/settings.json` (project) or `~/.claude/settings.json` (global):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"kb-server": {
|
||||
"type": "http",
|
||||
"url": "http://localhost:3000/mcp",
|
||||
"headers": {
|
||||
"Authorization": "Bearer your-agent-key"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### VS Code (GitHub Copilot)
|
||||
|
||||
Add to your `.vscode/settings.json` (workspace) or user settings:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcp": {
|
||||
"servers": {
|
||||
"kb-server": {
|
||||
"type": "http",
|
||||
"url": "http://localhost:3000/mcp",
|
||||
"headers": {
|
||||
"Authorization": "Bearer your-agent-key"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Or add to `.vscode/mcp.json` in your workspace:
|
||||
|
||||
```json
|
||||
{
|
||||
"servers": {
|
||||
"kb-server": {
|
||||
"type": "http",
|
||||
"url": "http://localhost:3000/mcp",
|
||||
"headers": {
|
||||
"Authorization": "Bearer your-agent-key"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cursor
|
||||
|
||||
Add to `.cursor/mcp.json` in your project root:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"kb-server": {
|
||||
"type": "streamable-http",
|
||||
"url": "http://localhost:3000/mcp",
|
||||
"headers": {
|
||||
"Authorization": "Bearer your-agent-key"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Windsurf
|
||||
|
||||
Add to `~/.codeium/windsurf/mcp_config.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"kb-server": {
|
||||
"serverUrl": "http://localhost:3000/mcp",
|
||||
"headers": {
|
||||
"Authorization": "Bearer your-agent-key"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### JetBrains IDEs (IntelliJ, WebStorm, PyCharm, etc.)
|
||||
|
||||
Add to `.junie/mcp.json` in your project root, or configure via **Settings > Tools > AI Assistant > MCP Servers**:
|
||||
|
||||
```json
|
||||
{
|
||||
"servers": {
|
||||
"kb-server": {
|
||||
"type": "http",
|
||||
"url": "http://localhost:3000/mcp",
|
||||
"headers": {
|
||||
"Authorization": "Bearer your-agent-key"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -2,16 +2,19 @@
|
||||
|
||||
Personal knowledge base with hybrid search (full-text + semantic vector search).
|
||||
|
||||
v2 uses a client-server architecture: a **FastAPI engine** running in Docker (with GPU acceleration) and a lightweight **Go CLI client** that talks to it over HTTP.
|
||||
Client-server architecture: a **FastAPI engine** running in Docker (with optional GPU acceleration), a lightweight **Go CLI client**, and an **MCP server** for native agent integration.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Go CLI (kb) ──HTTP──▶ FastAPI Engine (Docker) ──▶ SQLite + GPU
|
||||
▲
|
||||
MCP Agents ──MCP/HTTP──▶ MCP Server (Docker) ──┘
|
||||
```
|
||||
|
||||
- **Engine**: Keeps the embedding model warm in GPU memory. Handles search, ingestion, and document management via REST API. Runs in Docker with NVIDIA or AMD GPU support.
|
||||
- **Engine**: Keeps the embedding model warm in memory. Handles search, ingestion, document management, and note mutation via REST API. Runs in Docker with NVIDIA GPU or CPU-only support.
|
||||
- **Client**: Single static Go binary. No Python, no ML dependencies, instant startup. Talks to the engine over HTTP.
|
||||
- **MCP Server**: Exposes kb operations as native MCP tools over Streamable HTTP. Runs as a separate Docker container alongside the engine. Use tags to scope agent data from user documents.
|
||||
- **Storage**: Single SQLite database with FTS5 (keyword search) and sqlite-vec (vector search). Portable via bind mount — just copy the data directory between hosts.
|
||||
|
||||
## Quick start
|
||||
@@ -30,72 +33,31 @@ docker run -d --name kb-engine \
|
||||
-e KB_DEVICE=auto \
|
||||
-e KB_API_KEY=your-secret-key \
|
||||
--restart unless-stopped \
|
||||
docker.dcglab.co.uk/dcg/kb/engine:latest-nvidia
|
||||
docker.dcglab.co.uk/public/kb/engine:latest-nvidia
|
||||
|
||||
# AMD GPU (ROCm)
|
||||
# CPU only (no GPU required — smaller image)
|
||||
docker run -d --name kb-engine \
|
||||
--device /dev/kfd --device /dev/dri \
|
||||
--group-add video \
|
||||
-p 8000:8000 \
|
||||
-v ~/kb-data:/data \
|
||||
-e KB_MODEL=all-MiniLM-L6-v2 \
|
||||
-e KB_DEVICE=auto \
|
||||
-e KB_API_KEY=your-secret-key \
|
||||
--restart unless-stopped \
|
||||
docker.dcglab.co.uk/dcg/kb/engine:latest-rocm
|
||||
docker.dcglab.co.uk/public/kb/engine:latest-cpu
|
||||
```
|
||||
|
||||
Or use a compose file — create `compose.yaml`:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
kb-engine:
|
||||
image: docker.dcglab.co.uk/dcg/kb/engine:latest-nvidia # or latest-rocm
|
||||
runtime: nvidia # remove for ROCm
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
# For ROCm, replace the above runtime/deploy block with:
|
||||
# devices:
|
||||
# - "/dev/kfd"
|
||||
# - "/dev/dri"
|
||||
# group_add:
|
||||
# - "video"
|
||||
ports:
|
||||
- "${KB_PORT:-8000}:8000"
|
||||
volumes:
|
||||
- ${KB_DATA_PATH:-./data}:/data
|
||||
environment:
|
||||
- KB_MODEL=${KB_MODEL:-all-MiniLM-L6-v2}
|
||||
- KB_DEVICE=${KB_DEVICE:-auto}
|
||||
- KB_INGEST_DEVICE=${KB_INGEST_DEVICE:-auto}
|
||||
- KB_API_KEY=${KB_API_KEY:-}
|
||||
- KB_SEARCH_THRESHOLD=${KB_SEARCH_THRESHOLD:-0.01}
|
||||
- HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-}
|
||||
restart: unless-stopped
|
||||
```
|
||||
Or use a compose file from the repo:
|
||||
|
||||
```bash
|
||||
KB_DATA_PATH=~/kb-data docker compose up -d
|
||||
```
|
||||
|
||||
**From source** (for development):
|
||||
|
||||
```bash
|
||||
cd engine
|
||||
|
||||
# NVIDIA GPU
|
||||
KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d
|
||||
KB_DATA_PATH=~/kb-data docker compose -f engine/compose.nvidia.yaml up -d
|
||||
|
||||
# AMD GPU (ROCm)
|
||||
KB_DATA_PATH=~/kb-data docker compose -f compose.rocm.yaml up -d
|
||||
# CPU only
|
||||
KB_DATA_PATH=~/kb-data docker compose -f engine/compose.cpu.yaml up -d
|
||||
```
|
||||
|
||||
The engine will download the embedding model on first start (~90MB) and load it onto the GPU. Check readiness:
|
||||
See [DEVELOPER.md](DEVELOPER.md) to run the engine from source.
|
||||
|
||||
The engine will download the embedding model on first start (~90MB) and load it into memory (GPU or CPU). Check readiness:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/api/v1/health
|
||||
@@ -110,7 +72,7 @@ Check [releases](https://gitea.dcglab.co.uk/steve/kb/releases) for the latest cl
|
||||
|
||||
```bash
|
||||
# Set the version tag
|
||||
TAG=client-v2.1.0
|
||||
TAG=client-v3.0.0
|
||||
|
||||
# Linux (amd64)
|
||||
curl -L -o kb https://gitea.dcglab.co.uk/steve/kb/releases/download/${TAG}/kb-linux-amd64
|
||||
@@ -129,13 +91,7 @@ chmod +x kb
|
||||
sudo mv kb /usr/local/bin/
|
||||
```
|
||||
|
||||
**From source** (for development):
|
||||
|
||||
```bash
|
||||
cd client
|
||||
make build # produces ./kb binary
|
||||
make all # or cross-compile: dist/kb-{os}-{arch}
|
||||
```
|
||||
See [DEVELOPER.md](DEVELOPER.md) to build the client from source.
|
||||
|
||||
### 3. Configure the client
|
||||
|
||||
@@ -152,13 +108,15 @@ Override via environment variables (`KB_ENGINE_URL`, `KB_API_KEY`) or CLI flags
|
||||
### 4. Use it
|
||||
|
||||
```bash
|
||||
# Quick notes (shorthand — no subcommand needed)
|
||||
kb "Always restart nginx after config changes"
|
||||
kb "Server room is building 3, floor 2" --tags ops
|
||||
# Add notes
|
||||
kb addnote "Always restart nginx after config changes"
|
||||
kb addnote "Server room is building 3, floor 2" --tags ops
|
||||
kb addnote "Deploy checklist" --wait
|
||||
|
||||
# Add files (async — uploads and exits immediately)
|
||||
# Add files (async by default; --wait blocks until ingestion finishes)
|
||||
kb addfile ~/docs/manual.pdf --tags admin
|
||||
kb addfile ~/notes/ --recursive
|
||||
kb addfile ~/docs/manual.pdf --wait
|
||||
|
||||
# Check ingestion progress
|
||||
kb jobs
|
||||
@@ -166,21 +124,33 @@ kb jobs
|
||||
# Search
|
||||
kb search "how to install git"
|
||||
kb search "deploy process" --tags ops --type pdf
|
||||
kb find "vehicle handbook" --type pdf
|
||||
|
||||
# Update a note in place
|
||||
kb updatenote 42 "revised note content"
|
||||
|
||||
# Manage
|
||||
kb list
|
||||
kb info 1
|
||||
kb list --title handbook
|
||||
kb list --filename M38T_PHEV
|
||||
kb info 1 --no-chunks
|
||||
kb tags
|
||||
kb tag 1 --add important
|
||||
kb export 1 -o manual.pdf # download original file
|
||||
kb remove 3 --yes
|
||||
kb status
|
||||
|
||||
# Bulk operations
|
||||
kb bulk-remove --tags "draft,old" --type note --yes
|
||||
kb bulk-tag --type note --add "archived" --yes
|
||||
kb bulk-set-tags --tags "old-scheme" --set "new-scheme" --yes
|
||||
```
|
||||
|
||||
## How it works
|
||||
|
||||
- **Ingestion**: Files are uploaded to the engine and queued for async processing. The engine chunks documents (PDFs via Docling, markdown by headers, code by AST/functions, notes as whole text), generates embeddings on GPU, and stores everything in SQLite.
|
||||
- **Search**: Hybrid retrieval combining BM25 keyword scoring (FTS5) and vector similarity (sqlite-vec), merged via Reciprocal Rank Fusion. Sub-100ms with a warm model.
|
||||
- **Ingestion**: Files are uploaded to the engine and queued for async processing. The engine chunks documents (PDFs via Docling, markdown by headers, code by AST/functions, notes as whole text, JSON/YAML/TOML as pretty-printed text), generates embeddings on GPU, and stores everything in SQLite.
|
||||
- **Search**: Hybrid retrieval combining BM25 keyword scoring (FTS5) and vector similarity (sqlite-vec), merged via Reciprocal Rank Fusion with a top-rank bonus. Optionally reranked by a local cross-encoder (`KB_RERANK_ENABLED`). Sub-100ms with a warm model (without reranking). Add `--explain` to any search for a per-result score breakdown.
|
||||
- **Quality measurement**: `kb bench fixture.json` runs a fixture of queries with known-relevant documents and reports precision@k / recall / MRR per backend (fts, vec, hybrid, hybrid+rerank). See `docs/bench-example.json`.
|
||||
- **Output**: JSON (for scripts/LLM tool use) or human-readable terminal format. Use `--format json` on any command.
|
||||
|
||||
## Engine configuration
|
||||
@@ -195,11 +165,31 @@ The engine is configured via environment variables (set in the compose file or v
|
||||
| `KB_INGEST_DEVICE` | `auto` | Docling layout detection device: `auto`, `cpu`, or `cuda` |
|
||||
| `KB_API_KEY` | (none) | Optional Bearer token for API authentication |
|
||||
| `KB_SEARCH_THRESHOLD` | `0.01` | Minimum score for search results (filters noise) |
|
||||
| `KB_MIN_CHUNK_ALNUM` | `3` | Minimum alphanumeric characters retained in Docling PDF/DOCX/HTML chunks (`0` disables) |
|
||||
| `KB_RERANK_ENABLED` | `false` (`true` in nvidia compose) | Load a cross-encoder and rerank hybrid results server-side |
|
||||
| `KB_RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | Cross-encoder model for reranking |
|
||||
| `KB_RERANK_CANDIDATES` | `40` | Hybrid candidates scored by the reranker per query |
|
||||
| `KB_BULK_SAFETY_PERCENT` | `70` | Bulk operations affecting more than this % of documents are rejected unless `force` is set (0 disables) |
|
||||
| `KB_PORT` | `8000` | Port to expose |
|
||||
| `KB_HOST` | `0.0.0.0` | Host to bind to |
|
||||
| `HF_HUB_OFFLINE` | (none) | Set to `1` to prevent model downloads (use cached only) |
|
||||
| `KB_DATA_PATH` | `./data` | Host path for bind mount (compose variable, not used by engine) |
|
||||
|
||||
### Repairing legacy note titles
|
||||
|
||||
Notes created by older clients may have a synthetic `<uuid>_note.note` title.
|
||||
The maintenance command previews only unambiguous matches by default:
|
||||
|
||||
```bash
|
||||
cd engine
|
||||
python -m kb.maintenance.backfill_note_titles
|
||||
python -m kb.maintenance.backfill_note_titles --apply
|
||||
```
|
||||
|
||||
The apply mode reloads the configured embedding model and refreshes each
|
||||
affected note's title, enriched full-text content, and vector embedding. Back
|
||||
up the data directory before running it against production.
|
||||
|
||||
## Data portability
|
||||
|
||||
The data directory contains everything: SQLite database, model cache, and staging files. To migrate between hosts:
|
||||
@@ -212,83 +202,14 @@ rsync -a ~/kb-data/ user@target:/home/user/kb-data/
|
||||
KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d
|
||||
```
|
||||
|
||||
Data is GPU-vendor-agnostic — you can ingest on NVIDIA and serve from AMD (or vice versa) with the same data directory.
|
||||
Data is device-agnostic — you can ingest on NVIDIA and serve from CPU (or vice versa) with the same data directory.
|
||||
|
||||
## API reference
|
||||
## MCP server (agent integration)
|
||||
|
||||
All endpoints are under `/api/v1/`. Requires `Authorization: Bearer <key>` header when `KB_API_KEY` is set.
|
||||
The MCP server exposes kb operations as native MCP tools over Streamable HTTP, so agents can search, add notes, upload files, and manage documents without shelling out to the CLI. Includes setup guides for Claude Code, VS Code, Cursor, Windsurf, and JetBrains IDEs.
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/health` | Health check (bypasses auth) |
|
||||
| `POST` | `/search` | Hybrid search (JSON body) |
|
||||
| `POST` | `/jobs` | Upload file/note for ingestion (multipart, returns 202 or 409 if duplicate) |
|
||||
| `GET` | `/jobs` | List ingestion jobs |
|
||||
| `GET` | `/jobs/{id}` | Job details |
|
||||
| `GET` | `/documents` | List documents |
|
||||
| `GET` | `/documents/{id}` | Document details with chunks |
|
||||
| `GET` | `/documents/{id}/file` | Download original file |
|
||||
| `DELETE` | `/documents/{id}` | Remove a document (and stored file) |
|
||||
| `PUT` | `/documents/{id}/tags` | Add/remove tags |
|
||||
| `GET` | `/tags` | List all tags |
|
||||
| `GET` | `/status` | Engine status, GPU info, DB stats |
|
||||
| `POST` | `/reindex` | Re-embed all chunks |
|
||||
See **[MCP.md](MCP.md)** for full details — server setup, available tools, tag-based organisation, configuration, and client examples.
|
||||
|
||||
## Building and releasing
|
||||
## Agent skill
|
||||
|
||||
Client and engine are versioned independently via `client/VERSION` and `engine/VERSION`. Each has its own release script and git tag prefix.
|
||||
|
||||
### Release client
|
||||
|
||||
```bash
|
||||
./release-client.sh --gitea # patch bump, release via Gitea
|
||||
./release-client.sh --github --minor # minor bump, release via GitHub
|
||||
./release-client.sh --gitea --no-increment # release current version as-is
|
||||
./release-client.sh --gitea --dry-run # preview without doing anything
|
||||
```
|
||||
|
||||
Creates tag `client-vX.Y.Z`, builds Go binaries for all platforms, and creates a Gitea/GitHub release with binaries attached.
|
||||
|
||||
The client embeds a `MinEngineVersion` (from `client/MIN_ENGINE_VERSION`) and will hard-fail if the connected engine is too old.
|
||||
|
||||
### Release engine
|
||||
|
||||
```bash
|
||||
./release-engine.sh --gitea # patch bump, release via Gitea
|
||||
./release-engine.sh --github --minor # minor bump, release via GitHub
|
||||
./release-engine.sh --gitea --no-increment # release current version as-is
|
||||
./release-engine.sh --gitea --dry-run # preview without doing anything
|
||||
```
|
||||
|
||||
Creates tag `engine-vX.Y.Z`, builds NVIDIA and ROCm Docker images, creates a Gitea/GitHub release, and pushes images to the registry.
|
||||
|
||||
### Checking versions
|
||||
|
||||
```bash
|
||||
# Client
|
||||
kb --version
|
||||
|
||||
# Engine
|
||||
curl http://localhost:8000/api/v1/status | jq .version
|
||||
```
|
||||
|
||||
### Docker images
|
||||
|
||||
Images are pushed to `docker.dcglab.co.uk/dcg/kb/engine` with tags:
|
||||
|
||||
- `engine-v2.0.6-nvidia` / `engine-v2.0.6-rocm` — versioned
|
||||
- `latest-nvidia` / `latest-rocm` — latest release
|
||||
|
||||
Override the registry and org via environment variables:
|
||||
|
||||
```bash
|
||||
REGISTRY=ghcr.io IMAGE_ORG=myorg ./release-engine.sh --github
|
||||
```
|
||||
|
||||
## Future: ROCm runtime migration
|
||||
|
||||
The `onnxruntime-rocm` execution provider was removed from onnxruntime as of v1.23. AMD is pushing toward the **MIGraphX execution provider** as the replacement for ROCm GPU inference. When upgrading onnxruntime beyond v1.22, the ROCm Dockerfile will need to switch from `onnxruntime-rocm` to `onnxruntime` with the MIGraphX EP and install the `migraphx` runtime libraries instead.
|
||||
|
||||
## Claude Code skill
|
||||
|
||||
This tool is designed to be wrapped as a Claude Code skill. See `SKILL.md` for the skill definition.
|
||||
If you are restricted from using MCP server, or you just prefer to utilise Agent SKILLS, please also see `SKILL.md` for the skill definition.
|
||||
|
||||
@@ -10,14 +10,14 @@ Search, manage, and add to the user's personal knowledge base containing PDFs, W
|
||||
- User asks "how do I..." style questions that their knowledge base likely covers
|
||||
- User wants to save a note, add a file, or manage their knowledge base
|
||||
|
||||
## Quick notes
|
||||
## Adding notes
|
||||
|
||||
```bash
|
||||
kb "remember to update DNS records" # add a note
|
||||
kb "server room is building 3, floor 2" --tags ops # add a tagged note
|
||||
kb addnote "remember to update DNS records" # add a note
|
||||
kb addnote "server room is building 3, floor 2" --tags ops # add a tagged note
|
||||
```
|
||||
|
||||
Bare text without a subcommand is treated as a note and submitted for ingestion.
|
||||
The note text must be a single quoted argument.
|
||||
|
||||
## Search (primary use case)
|
||||
|
||||
@@ -30,11 +30,13 @@ Returns JSON with ranked results combining full-text and semantic search.
|
||||
**Flags:**
|
||||
- `-n, --top N` — number of results (default: 10)
|
||||
- `--tags tag1,tag2` — filter by tags (AND logic)
|
||||
- `--type pdf|markdown|code|note` — filter by document type
|
||||
- `--type pdf|markdown|code|note|data` — filter by document type
|
||||
- `--format json|human` — output format (always use json for parsing)
|
||||
- `--fts-only` — keyword search only (skip semantic)
|
||||
- `--vec-only` — semantic search only (skip keyword)
|
||||
- `--threshold FLOAT` — minimum score cutoff
|
||||
- `--explain` — include a per-result score breakdown (FTS/vector scores and ranks, fusion contributions, rerank blend)
|
||||
- `--no-rerank` — skip server-side cross-encoder reranking for lower latency (when the engine has it enabled)
|
||||
|
||||
## Adding files
|
||||
|
||||
@@ -45,7 +47,7 @@ kb addfile ~/docs/ --recursive # directory (recursive)
|
||||
kb addfile ~/docs/ --recursive --tags reference # directory with tags
|
||||
```
|
||||
|
||||
Supported file types: `.pdf`, `.docx`, `.html`, `.md`, `.txt`, `.py`, `.sh`, `.go`. Unsupported extensions are rejected before upload.
|
||||
Supported file types: `.pdf`, `.docx`, `.html`, `.md`, `.txt`, `.py`, `.sh`, `.go`, `.json`, `.yaml`, `.yml`, `.toml`. Unsupported extensions are rejected before upload. Data files (`.json`/`.yaml`/`.yml`/`.toml`) are ingested as text with doc type `data`; minified JSON is pretty-printed before chunking.
|
||||
|
||||
**Flags:**
|
||||
- `--tags tag1,tag2` — tags (comma-separated)
|
||||
@@ -66,11 +68,48 @@ kb remove <doc_id> --yes # remove without confirmation
|
||||
## Tag management
|
||||
|
||||
```bash
|
||||
kb tags --format json # list all tags with counts
|
||||
kb tags --format json # list all tags with counts and descriptions
|
||||
kb tag <doc_id> --add important,ops # add tags to a document
|
||||
kb tag <doc_id> --remove draft # remove tags from a document
|
||||
kb tag-describe ops "Lab operations runbooks" # set a tag context description
|
||||
kb tag-describe ops # clear a tag's description
|
||||
```
|
||||
|
||||
Tag descriptions are returned as `tag_contexts` with every search result on a
|
||||
document carrying the tag — use them to judge which of several similar-scoring
|
||||
chunks actually answers the question.
|
||||
|
||||
## Bulk operations
|
||||
|
||||
Operate on multiple documents at once using filter-based selection. Filters combine with AND logic.
|
||||
|
||||
**Filter flags (shared across all bulk commands):**
|
||||
- `--tags tag1,tag2` — match documents with ALL specified tags
|
||||
- `--type pdf|note|...` — match by document type
|
||||
- `--ids 1,5,12` — match specific document IDs
|
||||
- `--from-id N` — match documents with id >= N
|
||||
- `--to-id N` — match documents with id <= N
|
||||
- `--force` / `-f` — override safety threshold (blocks operations affecting >70% of all documents)
|
||||
- `--yes` / `-y` — skip confirmation prompt
|
||||
|
||||
```bash
|
||||
# Bulk delete
|
||||
kb bulk-remove --tags "draft,old" --type note --yes # delete matching docs
|
||||
kb bulk-remove --from-id 10 --to-id 50 --yes # delete by ID range
|
||||
kb bulk-remove --ids "3,7,12" --yes # delete specific IDs
|
||||
|
||||
# Bulk tag add/remove
|
||||
kb bulk-tag --tags "agent:mybot" --add "reviewed" --remove "pending" --yes
|
||||
kb bulk-tag --type note --add "archived" --yes # tag all notes
|
||||
|
||||
# Bulk replace tags
|
||||
kb bulk-set-tags --tags "old-scheme" --set "new-scheme,migrated" --yes
|
||||
```
|
||||
|
||||
All bulk commands return a summary: matched count, succeeded count, failed count, and errors.
|
||||
A safety threshold prevents accidentally affecting more than 70% of documents unless `--force` is used.
|
||||
The threshold is configurable on the engine via `KB_BULK_SAFETY_PERCENT` (integer 0-100, default 70; 0 disables).
|
||||
|
||||
## Jobs (ingestion queue)
|
||||
|
||||
```bash
|
||||
@@ -79,6 +118,12 @@ kb jobs --status failed --format json # filter by status
|
||||
kb jobs <job_id> --format json # job details
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
kb examples # show common usage examples
|
||||
```
|
||||
|
||||
## Engine status and maintenance
|
||||
|
||||
```bash
|
||||
@@ -101,23 +146,22 @@ All commands support:
|
||||
"results": [
|
||||
{
|
||||
"chunk_id": 1423,
|
||||
"document_id": 87,
|
||||
"score": 0.031,
|
||||
"score_breakdown": {"fts": 0.016, "vector": 0.015},
|
||||
"text": "To install the latest version of git from source...",
|
||||
"source": {
|
||||
"document_id": 42,
|
||||
"title": "Git Admin Guide",
|
||||
"path": "/home/user/docs/git-admin.pdf",
|
||||
"type": "pdf",
|
||||
"page": 12,
|
||||
"chunk_index": 3,
|
||||
"total_chunks": 28,
|
||||
"tags": ["git", "admin"]
|
||||
}
|
||||
"chunk_metadata": {"page": 12},
|
||||
"title": "Git Admin Guide",
|
||||
"doc_type": "pdf",
|
||||
"source_path": "/home/user/docs/git-admin.pdf",
|
||||
"created_at": "2026-03-15T10:30:00",
|
||||
"tags": ["git", "admin"],
|
||||
"tag_contexts": {"admin": "System administration guides"}
|
||||
}
|
||||
],
|
||||
"total_matches": 47,
|
||||
"returned": 10
|
||||
"returned": 10,
|
||||
"reranked": true
|
||||
}
|
||||
```
|
||||
|
||||
@@ -156,11 +200,37 @@ Use filters when the question implies a specific domain:
|
||||
- From a specific topic → `--tags <topic>`
|
||||
- Check available tags first: `kb tags --format json`
|
||||
|
||||
## Updating notes
|
||||
|
||||
```bash
|
||||
kb updatenote 42 "revised note content" # update note by ID
|
||||
```
|
||||
|
||||
Updates the text of an existing note in place, preserving its ID, creation timestamp, and tags. Re-chunks and re-embeds the new text.
|
||||
|
||||
## MCP server (agent integration)
|
||||
|
||||
For agent-to-agent integration, kb provides an MCP server alongside the CLI. The MCP server
|
||||
exposes the same operations as native MCP tools over Streamable HTTP transport, which agents
|
||||
can connect to directly without subprocess overhead.
|
||||
|
||||
**MCP tools:** `kb_search`, `kb_addnote`, `kb_update_note`, `kb_get`, `kb_delete`, `kb_status`,
|
||||
`kb_jobs`, `kb_upload_start`, `kb_upload_chunk`, `kb_upload_finish`, `kb_bulk_delete`,
|
||||
`kb_bulk_tags`, `kb_bulk_set_tags`.
|
||||
|
||||
Use tags to separate agent data from user documents (e.g. tag all agent notes with
|
||||
`agent:mybot` and filter by that tag when searching). This convention is communicated
|
||||
via system prompt — no special server-side enforcement needed.
|
||||
|
||||
If the kb engine is already running via Docker Compose, add the MCP server by deploying the
|
||||
`kb-mcp` service from the same compose file. Agents connect to it on port 3000 (default).
|
||||
|
||||
## Important notes
|
||||
|
||||
- Always use `--format json` for machine parsing
|
||||
- The `score` field is relative, not absolute — compare scores within a result set
|
||||
- `source.page` is only present for PDF documents
|
||||
- `source.section_header` is only present for markdown documents with headers
|
||||
- The `score` field is relative, not absolute — compare scores within a result set. Reranked hybrid scores (`"reranked": true`) are 0-1 blended values on a different scale from non-reranked RRF scores; don't compare across the two modes or apply `--threshold` expecting RRF-scale values on reranked output
|
||||
- When the engine reranker is enabled, results are already cross-encoder reranked server-side — no need to rerank them yourself
|
||||
- `chunk_metadata.page` is only present for PDF documents
|
||||
- `chunk_metadata.section_header` is only present for markdown documents with headers
|
||||
- Results are already ranked by relevance (hybrid FTS + vector search)
|
||||
- Duplicate files are detected at upload time (HTTP 409) — the client handles this gracefully
|
||||
|
||||
@@ -1 +1 @@
|
||||
2.0.0
|
||||
3.3.0
|
||||
|
||||
+1
-1
@@ -1 +1 @@
|
||||
2.1.1
|
||||
3.3.0
|
||||
|
||||
+48
-50
@@ -8,6 +8,7 @@ import (
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
@@ -38,6 +39,10 @@ var supportedExts = map[string]bool{
|
||||
".py": true,
|
||||
".sh": true,
|
||||
".go": true,
|
||||
".json": true,
|
||||
".yaml": true,
|
||||
".yml": true,
|
||||
".toml": true,
|
||||
}
|
||||
|
||||
var addfileCmd = &cobra.Command{
|
||||
@@ -50,12 +55,16 @@ var addfileCmd = &cobra.Command{
|
||||
func init() {
|
||||
addfileCmd.Flags().String("tags", "", "tags (comma-separated)")
|
||||
addfileCmd.Flags().BoolP("recursive", "r", false, "recursively add directory contents")
|
||||
addfileCmd.Flags().Bool("wait", false, "wait for ingestion to finish")
|
||||
addfileCmd.Flags().Duration("wait-timeout", 10*time.Minute, "maximum time to wait per ingestion job")
|
||||
rootCmd.AddCommand(addfileCmd)
|
||||
}
|
||||
|
||||
func runAddfile(cmd *cobra.Command, args []string) error {
|
||||
tags, _ := cmd.Flags().GetString("tags")
|
||||
recursive, _ := cmd.Flags().GetBool("recursive")
|
||||
wait, _ := cmd.Flags().GetBool("wait")
|
||||
timeout, _ := cmd.Flags().GetDuration("wait-timeout")
|
||||
|
||||
client := api.NewClient()
|
||||
|
||||
@@ -85,12 +94,25 @@ func runAddfile(cmd *cobra.Command, args []string) error {
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
if !wait || result.Duplicate {
|
||||
output.PrintJSON([]interface{}{result.Raw})
|
||||
}
|
||||
} else if result.Duplicate {
|
||||
fmt.Println(result.duplicateMsg())
|
||||
} else {
|
||||
fmt.Printf("Queued: %s\n", filepath.Base(path))
|
||||
}
|
||||
if wait && !result.Duplicate {
|
||||
job, err := waitForJob(client, int(result.JobID), timeout)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON([]interface{}{job})
|
||||
} else {
|
||||
fmt.Printf("Ingested: %s (doc ID: %d, chunks: %d)\n", filepath.Base(path), job.DocumentID, job.ChunkCount)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -118,6 +140,7 @@ func runAddfile(cmd *cobra.Command, args []string) error {
|
||||
}
|
||||
|
||||
var results []interface{}
|
||||
var pending []*uploadResult
|
||||
queued := 0
|
||||
duplicates := 0
|
||||
for _, f := range files {
|
||||
@@ -126,7 +149,9 @@ func runAddfile(cmd *cobra.Command, args []string) error {
|
||||
fmt.Fprintf(os.Stderr, "Error uploading %s: %v\n", f, err)
|
||||
continue
|
||||
}
|
||||
if !wait || result.Duplicate {
|
||||
results = append(results, result.Raw)
|
||||
}
|
||||
if result.Duplicate {
|
||||
duplicates++
|
||||
if !output.IsJSON() {
|
||||
@@ -134,11 +159,25 @@ func runAddfile(cmd *cobra.Command, args []string) error {
|
||||
}
|
||||
} else {
|
||||
queued++
|
||||
pending = append(pending, result)
|
||||
if !output.IsJSON() {
|
||||
fmt.Printf("Queued: %s\n", filepath.Base(f))
|
||||
}
|
||||
}
|
||||
}
|
||||
if wait {
|
||||
for _, result := range pending {
|
||||
job, err := waitForJob(client, int(result.JobID), timeout)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if output.IsJSON() {
|
||||
results = append(results, job)
|
||||
} else {
|
||||
fmt.Printf("Ingested job %d (doc ID: %d, chunks: %d)\n", int(result.JobID), job.DocumentID, job.ChunkCount)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(results)
|
||||
@@ -199,60 +238,19 @@ func uploadFile(client *api.Client, path, tags string) (*uploadResult, error) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var result interface{}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
var raw json.RawMessage
|
||||
if err := api.DecodeJSON(resp, &raw); err != nil {
|
||||
return nil, fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
return &uploadResult{Raw: result}, nil
|
||||
var queued struct {
|
||||
JobID float64 `json:"job_id"`
|
||||
}
|
||||
|
||||
func submitNote(client *api.Client, note, tags string) error {
|
||||
fields := map[string]string{
|
||||
"note": note,
|
||||
if err := json.Unmarshal(raw, &queued); err != nil {
|
||||
return nil, fmt.Errorf("failed to decode queued job: %w", err)
|
||||
}
|
||||
if tags != "" {
|
||||
fields["tags"] = tags
|
||||
}
|
||||
|
||||
resp, err := client.PostMultipart("/api/v1/jobs", fields, nil)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if resp.StatusCode == http.StatusConflict {
|
||||
var result interface{}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
if err := json.Unmarshal(raw, &result); err != nil {
|
||||
return nil, fmt.Errorf("failed to decode queued response: %w", err)
|
||||
}
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(result)
|
||||
} else {
|
||||
if m, ok := result.(map[string]interface{}); ok {
|
||||
if docID, ok := m["document_id"].(float64); ok {
|
||||
fmt.Printf("Already imported: %s (doc ID: %.0f)\n", m["title"], docID)
|
||||
} else if jobID, ok := m["job_id"].(float64); ok {
|
||||
fmt.Printf("Already queued: %s (job ID: %.0f)\n", m["title"], jobID)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
var result interface{}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(result)
|
||||
} else {
|
||||
fmt.Println("Queued: note")
|
||||
}
|
||||
return nil
|
||||
return &uploadResult{Raw: result, JobID: queued.JobID}, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var addnoteCmd = &cobra.Command{
|
||||
Use: "addnote <text>",
|
||||
Short: "Add a text note to the knowledge base",
|
||||
Args: func(cmd *cobra.Command, args []string) error {
|
||||
if len(args) == 0 {
|
||||
return fmt.Errorf("requires a note text argument\n\n Usage: kb addnote \"your note text here\"")
|
||||
}
|
||||
if len(args) > 1 {
|
||||
return fmt.Errorf("accepts 1 arg but received %d — quote your note text, e.g. kb addnote \"your note text here\"", len(args))
|
||||
}
|
||||
return nil
|
||||
},
|
||||
RunE: runAddnote,
|
||||
}
|
||||
|
||||
func init() {
|
||||
addnoteCmd.Flags().String("tags", "", "tags (comma-separated)")
|
||||
addnoteCmd.Flags().Bool("wait", false, "wait for ingestion to finish")
|
||||
addnoteCmd.Flags().Duration("wait-timeout", 10*time.Minute, "maximum time to wait for ingestion")
|
||||
rootCmd.AddCommand(addnoteCmd)
|
||||
}
|
||||
|
||||
func runAddnote(cmd *cobra.Command, args []string) error {
|
||||
tags, _ := cmd.Flags().GetString("tags")
|
||||
wait, _ := cmd.Flags().GetBool("wait")
|
||||
timeout, _ := cmd.Flags().GetDuration("wait-timeout")
|
||||
client := api.NewClient()
|
||||
return submitNote(client, args[0], tags, wait, timeout)
|
||||
}
|
||||
|
||||
func submitNote(client *api.Client, note, tags string, wait bool, timeout time.Duration) error {
|
||||
fields := map[string]string{
|
||||
"note": note,
|
||||
}
|
||||
if tags != "" {
|
||||
fields["tags"] = tags
|
||||
}
|
||||
|
||||
resp, err := client.PostMultipart("/api/v1/jobs", fields, nil)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if resp.StatusCode == http.StatusConflict {
|
||||
var result interface{}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(result)
|
||||
} else {
|
||||
if m, ok := result.(map[string]interface{}); ok {
|
||||
if docID, ok := m["document_id"].(float64); ok {
|
||||
fmt.Printf("Already imported: %s (doc ID: %.0f)\n", m["title"], docID)
|
||||
} else if jobID, ok := m["job_id"].(float64); ok {
|
||||
fmt.Printf("Already queued: %s (job ID: %.0f)\n", m["title"], jobID)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
var result struct {
|
||||
JobID int `json:"job_id"`
|
||||
Status string `json:"status"`
|
||||
Filename string `json:"filename"`
|
||||
}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
if !wait {
|
||||
output.PrintJSON(result)
|
||||
}
|
||||
} else {
|
||||
fmt.Println("Queued: note")
|
||||
}
|
||||
if wait {
|
||||
job, err := waitForJob(client, result.JobID, timeout)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(job)
|
||||
} else {
|
||||
fmt.Printf("Ingested: note (doc ID: %d, chunks: %d)\n", job.DocumentID, job.ChunkCount)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,376 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var benchCmd = &cobra.Command{
|
||||
Use: "bench <fixture.json>",
|
||||
Short: "Benchmark search quality against a query fixture",
|
||||
Long: `Run a fixture of queries with known-relevant documents against each
|
||||
search backend (fts, vec, hybrid, rerank) and report precision@k, recall
|
||||
and MRR per backend. Use this to baseline search quality before ranking
|
||||
changes and to measure their effect. See docs/bench-example.json for the
|
||||
fixture format.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: runBench,
|
||||
}
|
||||
|
||||
func init() {
|
||||
benchCmd.Flags().Int("top", 0, "override result count per query (k)")
|
||||
benchCmd.Flags().String("backends", "fts,vec,hybrid,rerank", "comma-separated backends to run")
|
||||
rootCmd.AddCommand(benchCmd)
|
||||
}
|
||||
|
||||
type benchSelector struct {
|
||||
DocumentID int64 `json:"document_id"`
|
||||
SourcePath string `json:"source_path"`
|
||||
TitleContains string `json:"title_contains"`
|
||||
}
|
||||
|
||||
type benchQuery struct {
|
||||
ID string `json:"id"`
|
||||
Query string `json:"query"`
|
||||
Tags []string `json:"tags"`
|
||||
DocType string `json:"doc_type"`
|
||||
Top int `json:"top"`
|
||||
Relevant []benchSelector `json:"relevant"`
|
||||
Notes string `json:"notes"`
|
||||
}
|
||||
|
||||
type benchFixture struct {
|
||||
Description string `json:"description"`
|
||||
Top int `json:"top"`
|
||||
Queries []benchQuery `json:"queries"`
|
||||
}
|
||||
|
||||
type benchDoc struct {
|
||||
DocumentID int64
|
||||
Title string
|
||||
SourcePath string
|
||||
}
|
||||
|
||||
type queryResult struct {
|
||||
ID string `json:"id"`
|
||||
Precision float64 `json:"precision"`
|
||||
Recall float64 `json:"recall"`
|
||||
MRR float64 `json:"mrr"`
|
||||
LatencyMS float64 `json:"latency_ms"`
|
||||
Returned int `json:"returned"`
|
||||
}
|
||||
|
||||
type backendResult struct {
|
||||
Precision float64 `json:"precision"`
|
||||
Recall float64 `json:"recall"`
|
||||
MRR float64 `json:"mrr"`
|
||||
AvgLatencyMS float64 `json:"avg_latency_ms"`
|
||||
Queries []queryResult `json:"queries"`
|
||||
}
|
||||
|
||||
func loadFixture(path string) (*benchFixture, error) {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot read fixture: %w", err)
|
||||
}
|
||||
var fx benchFixture
|
||||
if err := json.Unmarshal(data, &fx); err != nil {
|
||||
return nil, fmt.Errorf("invalid fixture JSON: %w", err)
|
||||
}
|
||||
if len(fx.Queries) == 0 {
|
||||
return nil, fmt.Errorf("fixture has no queries")
|
||||
}
|
||||
for i, q := range fx.Queries {
|
||||
if q.Query == "" {
|
||||
return nil, fmt.Errorf("query %d: missing query text", i)
|
||||
}
|
||||
if len(q.Relevant) == 0 {
|
||||
return nil, fmt.Errorf("query %q: no relevant selectors", q.Query)
|
||||
}
|
||||
for j, sel := range q.Relevant {
|
||||
set := 0
|
||||
if sel.DocumentID != 0 {
|
||||
set++
|
||||
}
|
||||
if sel.SourcePath != "" {
|
||||
set++
|
||||
}
|
||||
if sel.TitleContains != "" {
|
||||
set++
|
||||
}
|
||||
if set != 1 {
|
||||
return nil, fmt.Errorf("query %q selector %d: exactly one of document_id, source_path, title_contains must be set", q.Query, j)
|
||||
}
|
||||
}
|
||||
}
|
||||
return &fx, nil
|
||||
}
|
||||
|
||||
func (s benchSelector) matches(doc benchDoc) bool {
|
||||
switch {
|
||||
case s.DocumentID != 0:
|
||||
return doc.DocumentID == s.DocumentID
|
||||
case s.SourcePath != "":
|
||||
return doc.SourcePath == s.SourcePath
|
||||
case s.TitleContains != "":
|
||||
return strings.Contains(strings.ToLower(doc.Title), strings.ToLower(s.TitleContains))
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// dedupeByDocument collapses ranked chunks into ranked documents, keeping the
|
||||
// best (first) position for each document.
|
||||
func dedupeByDocument(docs []benchDoc) []benchDoc {
|
||||
seen := map[int64]bool{}
|
||||
var out []benchDoc
|
||||
for _, d := range docs {
|
||||
if seen[d.DocumentID] {
|
||||
continue
|
||||
}
|
||||
seen[d.DocumentID] = true
|
||||
out = append(out, d)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// scoreQuery computes document-level precision, recall and MRR for one
|
||||
// query's ranked document list against the relevant-document selectors.
|
||||
func scoreQuery(ranked []benchDoc, relevant []benchSelector) (precision, recall, mrr float64) {
|
||||
if len(ranked) == 0 {
|
||||
return 0, 0, 0
|
||||
}
|
||||
|
||||
matchedDocs := 0
|
||||
firstMatch := 0
|
||||
selectorHit := make([]bool, len(relevant))
|
||||
for i, doc := range ranked {
|
||||
docMatched := false
|
||||
for j, sel := range relevant {
|
||||
if sel.matches(doc) {
|
||||
docMatched = true
|
||||
selectorHit[j] = true
|
||||
}
|
||||
}
|
||||
if docMatched {
|
||||
matchedDocs++
|
||||
if firstMatch == 0 {
|
||||
firstMatch = i + 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
selectorsMatched := 0
|
||||
for _, hit := range selectorHit {
|
||||
if hit {
|
||||
selectorsMatched++
|
||||
}
|
||||
}
|
||||
|
||||
precision = float64(matchedDocs) / float64(len(ranked))
|
||||
recall = float64(selectorsMatched) / float64(len(relevant))
|
||||
if firstMatch > 0 {
|
||||
mrr = 1.0 / float64(firstMatch)
|
||||
}
|
||||
return precision, recall, mrr
|
||||
}
|
||||
|
||||
// rerankAvailable probes engine status for a loaded reranker.
|
||||
func rerankAvailable(client *api.Client) bool {
|
||||
resp, err := client.Get("/api/v1/status")
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
var status struct {
|
||||
Rerank struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Loaded bool `json:"loaded"`
|
||||
} `json:"rerank"`
|
||||
}
|
||||
if err := api.DecodeJSON(resp, &status); err != nil {
|
||||
return false
|
||||
}
|
||||
return status.Rerank.Enabled && status.Rerank.Loaded
|
||||
}
|
||||
|
||||
func benchSearch(client *api.Client, q benchQuery, backend string, top int) ([]benchDoc, float64, error) {
|
||||
body := map[string]interface{}{
|
||||
"query": q.Query,
|
||||
"top": top,
|
||||
}
|
||||
if len(q.Tags) > 0 {
|
||||
body["tags"] = q.Tags
|
||||
}
|
||||
if q.DocType != "" {
|
||||
body["doc_type"] = q.DocType
|
||||
}
|
||||
switch backend {
|
||||
case "fts":
|
||||
body["fts_only"] = true
|
||||
case "vec":
|
||||
body["vec_only"] = true
|
||||
case "hybrid":
|
||||
body["rerank"] = false
|
||||
case "rerank":
|
||||
body["rerank"] = true
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
resp, err := client.Post("/api/v1/search", body)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
var result struct {
|
||||
Results []struct {
|
||||
DocumentID int64 `json:"document_id"`
|
||||
Title string `json:"title"`
|
||||
SourcePath string `json:"source_path"`
|
||||
} `json:"results"`
|
||||
}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
latency := float64(time.Since(start).Microseconds()) / 1000.0
|
||||
|
||||
var docs []benchDoc
|
||||
for _, r := range result.Results {
|
||||
docs = append(docs, benchDoc{DocumentID: r.DocumentID, Title: r.Title, SourcePath: r.SourcePath})
|
||||
}
|
||||
return dedupeByDocument(docs), latency, nil
|
||||
}
|
||||
|
||||
func runBench(cmd *cobra.Command, args []string) error {
|
||||
topFlag, _ := cmd.Flags().GetInt("top")
|
||||
backendsFlag, _ := cmd.Flags().GetString("backends")
|
||||
|
||||
fx, err := loadFixture(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var backends []string
|
||||
for _, b := range strings.Split(backendsFlag, ",") {
|
||||
b = strings.TrimSpace(b)
|
||||
if b == "" {
|
||||
continue
|
||||
}
|
||||
switch b {
|
||||
case "fts", "vec", "hybrid", "rerank":
|
||||
backends = append(backends, b)
|
||||
default:
|
||||
return fmt.Errorf("unknown backend %q (valid: fts, vec, hybrid, rerank)", b)
|
||||
}
|
||||
}
|
||||
if len(backends) == 0 {
|
||||
return fmt.Errorf("no backends selected")
|
||||
}
|
||||
|
||||
client := api.NewClient()
|
||||
|
||||
rerankSkipped := false
|
||||
if contains(backends, "rerank") && !rerankAvailable(client) {
|
||||
backends = remove(backends, "rerank")
|
||||
rerankSkipped = true
|
||||
if len(backends) == 0 {
|
||||
return fmt.Errorf("reranking is not available on this engine (requires engine with reranker enabled)")
|
||||
}
|
||||
}
|
||||
|
||||
results := map[string]*backendResult{}
|
||||
for _, backend := range backends {
|
||||
br := &backendResult{}
|
||||
for _, q := range fx.Queries {
|
||||
top := 10
|
||||
if fx.Top > 0 {
|
||||
top = fx.Top
|
||||
}
|
||||
if q.Top > 0 {
|
||||
top = q.Top
|
||||
}
|
||||
if topFlag > 0 {
|
||||
top = topFlag
|
||||
}
|
||||
|
||||
ranked, latency, err := benchSearch(client, q, backend, top)
|
||||
if err != nil {
|
||||
return fmt.Errorf("backend %s, query %q: %w", backend, q.Query, err)
|
||||
}
|
||||
p, r, m := scoreQuery(ranked, q.Relevant)
|
||||
id := q.ID
|
||||
if id == "" {
|
||||
id = q.Query
|
||||
}
|
||||
br.Queries = append(br.Queries, queryResult{
|
||||
ID: id, Precision: p, Recall: r, MRR: m,
|
||||
LatencyMS: latency, Returned: len(ranked),
|
||||
})
|
||||
}
|
||||
n := float64(len(br.Queries))
|
||||
for _, qr := range br.Queries {
|
||||
br.Precision += qr.Precision / n
|
||||
br.Recall += qr.Recall / n
|
||||
br.MRR += qr.MRR / n
|
||||
br.AvgLatencyMS += qr.LatencyMS / n
|
||||
}
|
||||
results[backend] = br
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(map[string]interface{}{
|
||||
"description": fx.Description,
|
||||
"query_count": len(fx.Queries),
|
||||
"backends": results,
|
||||
"rerank_skipped": rerankSkipped,
|
||||
})
|
||||
return nil
|
||||
}
|
||||
|
||||
if fx.Description != "" {
|
||||
fmt.Printf("%s (%d queries)\n\n", fx.Description, len(fx.Queries))
|
||||
}
|
||||
headers := []string{"Backend", "Precision", "Recall", "MRR", "Avg ms"}
|
||||
var rows [][]string
|
||||
for _, backend := range backends {
|
||||
br := results[backend]
|
||||
rows = append(rows, []string{
|
||||
backend,
|
||||
fmt.Sprintf("%.3f", br.Precision),
|
||||
fmt.Sprintf("%.3f", br.Recall),
|
||||
fmt.Sprintf("%.3f", br.MRR),
|
||||
fmt.Sprintf("%.0f", br.AvgLatencyMS),
|
||||
})
|
||||
}
|
||||
output.PrintTable(headers, rows)
|
||||
if rerankSkipped {
|
||||
fmt.Println("\nrerank: n/a (reranker not enabled on this engine)")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func contains(ss []string, s string) bool {
|
||||
for _, v := range ss {
|
||||
if v == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func remove(ss []string, s string) []string {
|
||||
var out []string
|
||||
for _, v := range ss {
|
||||
if v != s {
|
||||
out = append(out, v)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func writeFixture(t *testing.T, content string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "fixture.json")
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestLoadFixture_Valid(t *testing.T) {
|
||||
path := writeFixture(t, `{
|
||||
"description": "test",
|
||||
"top": 5,
|
||||
"queries": [
|
||||
{"id": "q1", "query": "hello", "relevant": [{"document_id": 7}]}
|
||||
]
|
||||
}`)
|
||||
fx, err := loadFixture(path)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if fx.Top != 5 || len(fx.Queries) != 1 || fx.Queries[0].Relevant[0].DocumentID != 7 {
|
||||
t.Errorf("fixture parsed incorrectly: %+v", fx)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFixture_RejectsEmptyRelevant(t *testing.T) {
|
||||
path := writeFixture(t, `{"queries": [{"query": "hello", "relevant": []}]}`)
|
||||
if _, err := loadFixture(path); err == nil {
|
||||
t.Error("expected error for query with no relevant selectors")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFixture_RejectsMultiFieldSelector(t *testing.T) {
|
||||
path := writeFixture(t, `{"queries": [
|
||||
{"query": "hello", "relevant": [{"document_id": 1, "source_path": "/x"}]}
|
||||
]}`)
|
||||
if _, err := loadFixture(path); err == nil {
|
||||
t.Error("expected error for selector with two fields set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelectorMatching(t *testing.T) {
|
||||
doc := benchDoc{DocumentID: 42, Title: "M38T Owner's Manual", SourcePath: "/data/m38t.pdf"}
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
sel benchSelector
|
||||
want bool
|
||||
}{
|
||||
{"document_id match", benchSelector{DocumentID: 42}, true},
|
||||
{"document_id miss", benchSelector{DocumentID: 43}, false},
|
||||
{"source_path match", benchSelector{SourcePath: "/data/m38t.pdf"}, true},
|
||||
{"source_path miss", benchSelector{SourcePath: "/data/other.pdf"}, false},
|
||||
{"title_contains case-insensitive", benchSelector{TitleContains: "m38t owner"}, true},
|
||||
{"title_contains miss", benchSelector{TitleContains: "workshop"}, false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := tc.sel.matches(doc); got != tc.want {
|
||||
t.Errorf("%s: got %v, want %v", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDedupeByDocument(t *testing.T) {
|
||||
docs := []benchDoc{
|
||||
{DocumentID: 1, Title: "a"},
|
||||
{DocumentID: 2, Title: "b"},
|
||||
{DocumentID: 1, Title: "a-again"},
|
||||
{DocumentID: 3, Title: "c"},
|
||||
}
|
||||
out := dedupeByDocument(docs)
|
||||
if len(out) != 3 || out[0].DocumentID != 1 || out[1].DocumentID != 2 || out[2].DocumentID != 3 {
|
||||
t.Errorf("dedupe failed: %+v", out)
|
||||
}
|
||||
if out[0].Title != "a" {
|
||||
t.Errorf("dedupe must keep first (best-ranked) occurrence, got %q", out[0].Title)
|
||||
}
|
||||
}
|
||||
|
||||
func approxEqual(a, b float64) bool {
|
||||
return math.Abs(a-b) < 1e-9
|
||||
}
|
||||
|
||||
func TestScoreQuery_HandComputed(t *testing.T) {
|
||||
// Ranked docs: 10, 20, 30, 40. Relevant: 20 and 40.
|
||||
ranked := []benchDoc{
|
||||
{DocumentID: 10}, {DocumentID: 20}, {DocumentID: 30}, {DocumentID: 40},
|
||||
}
|
||||
relevant := []benchSelector{{DocumentID: 20}, {DocumentID: 40}}
|
||||
|
||||
p, r, m := scoreQuery(ranked, relevant)
|
||||
if !approxEqual(p, 0.5) { // 2 of 4 returned docs are relevant
|
||||
t.Errorf("precision: got %v, want 0.5", p)
|
||||
}
|
||||
if !approxEqual(r, 1.0) { // both relevant docs found
|
||||
t.Errorf("recall: got %v, want 1.0", r)
|
||||
}
|
||||
if !approxEqual(m, 0.5) { // first relevant doc at rank 2
|
||||
t.Errorf("mrr: got %v, want 0.5", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScoreQuery_NoMatches(t *testing.T) {
|
||||
ranked := []benchDoc{{DocumentID: 1}}
|
||||
relevant := []benchSelector{{DocumentID: 99}}
|
||||
p, r, m := scoreQuery(ranked, relevant)
|
||||
if p != 0 || r != 0 || m != 0 {
|
||||
t.Errorf("expected all-zero metrics, got p=%v r=%v mrr=%v", p, r, m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScoreQuery_PartialRecall(t *testing.T) {
|
||||
// Only one of three relevant docs returned, at rank 1.
|
||||
ranked := []benchDoc{{DocumentID: 5}, {DocumentID: 6}}
|
||||
relevant := []benchSelector{{DocumentID: 5}, {DocumentID: 7}, {DocumentID: 8}}
|
||||
p, r, m := scoreQuery(ranked, relevant)
|
||||
if !approxEqual(p, 0.5) {
|
||||
t.Errorf("precision: got %v, want 0.5", p)
|
||||
}
|
||||
if !approxEqual(r, 1.0/3.0) {
|
||||
t.Errorf("recall: got %v, want 1/3", r)
|
||||
}
|
||||
if !approxEqual(m, 1.0) {
|
||||
t.Errorf("mrr: got %v, want 1.0", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScoreQuery_EmptyResults(t *testing.T) {
|
||||
p, r, m := scoreQuery(nil, []benchSelector{{DocumentID: 1}})
|
||||
if p != 0 || r != 0 || m != 0 {
|
||||
t.Errorf("expected zeros for empty results, got p=%v r=%v mrr=%v", p, r, m)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var bulkRemoveCmd = &cobra.Command{
|
||||
Use: "bulk-remove",
|
||||
Short: "Delete multiple documents matching a filter",
|
||||
RunE: runBulkRemove,
|
||||
}
|
||||
|
||||
func init() {
|
||||
addBulkFilterFlags(bulkRemoveCmd)
|
||||
rootCmd.AddCommand(bulkRemoveCmd)
|
||||
}
|
||||
|
||||
func runBulkRemove(cmd *cobra.Command, args []string) error {
|
||||
body, err := buildBulkBody(cmd)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
yes, _ := cmd.Flags().GetBool("yes")
|
||||
if !yes {
|
||||
desc := describeBulkFilter(cmd)
|
||||
fmt.Printf("This will delete documents matching: %s\nProceed? [y/N] ", desc)
|
||||
reader := bufio.NewReader(os.Stdin)
|
||||
answer, _ := reader.ReadString('\n')
|
||||
answer = strings.TrimSpace(strings.ToLower(answer))
|
||||
if answer != "y" && answer != "yes" {
|
||||
fmt.Println("Cancelled.")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
client := api.NewClient()
|
||||
resp, err := client.Post("/api/v1/bulk/delete", body)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
var result map[string]interface{}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(result)
|
||||
} else {
|
||||
printBulkResult("Deleted", result)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Shared helpers for all bulk commands
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
func addBulkFilterFlags(cmd *cobra.Command) {
|
||||
cmd.Flags().String("tags", "", "filter by tags (comma-separated)")
|
||||
cmd.Flags().String("type", "", "filter by document type")
|
||||
cmd.Flags().String("ids", "", "filter by document IDs (comma-separated)")
|
||||
cmd.Flags().Int("from-id", 0, "filter by id >= value")
|
||||
cmd.Flags().Int("to-id", 0, "filter by id <= value")
|
||||
cmd.Flags().BoolP("force", "f", false, "override safety threshold")
|
||||
cmd.Flags().BoolP("yes", "y", false, "skip confirmation prompt")
|
||||
}
|
||||
|
||||
func buildBulkBody(cmd *cobra.Command) (map[string]interface{}, error) {
|
||||
body := map[string]interface{}{}
|
||||
|
||||
tagsStr, _ := cmd.Flags().GetString("tags")
|
||||
if tagsStr != "" {
|
||||
body["tags"] = splitTags(tagsStr)
|
||||
}
|
||||
|
||||
docType, _ := cmd.Flags().GetString("type")
|
||||
if docType != "" {
|
||||
body["doc_type"] = docType
|
||||
}
|
||||
|
||||
idsStr, _ := cmd.Flags().GetString("ids")
|
||||
if idsStr != "" {
|
||||
ids, err := parseIntList(idsStr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid --ids: %w", err)
|
||||
}
|
||||
body["document_ids"] = ids
|
||||
}
|
||||
|
||||
fromID, _ := cmd.Flags().GetInt("from-id")
|
||||
if fromID > 0 {
|
||||
body["from_id"] = fromID
|
||||
}
|
||||
|
||||
toID, _ := cmd.Flags().GetInt("to-id")
|
||||
if toID > 0 {
|
||||
body["to_id"] = toID
|
||||
}
|
||||
|
||||
force, _ := cmd.Flags().GetBool("force")
|
||||
if force {
|
||||
body["force"] = true
|
||||
}
|
||||
|
||||
// Ensure at least one filter
|
||||
hasFilter := tagsStr != "" || docType != "" || idsStr != "" || fromID > 0 || toID > 0
|
||||
if !hasFilter {
|
||||
return nil, fmt.Errorf("at least one filter is required (--tags, --type, --ids, --from-id, --to-id)")
|
||||
}
|
||||
|
||||
return body, nil
|
||||
}
|
||||
|
||||
func describeBulkFilter(cmd *cobra.Command) string {
|
||||
var parts []string
|
||||
|
||||
tagsStr, _ := cmd.Flags().GetString("tags")
|
||||
if tagsStr != "" {
|
||||
parts = append(parts, fmt.Sprintf("tags=[%s]", tagsStr))
|
||||
}
|
||||
|
||||
docType, _ := cmd.Flags().GetString("type")
|
||||
if docType != "" {
|
||||
parts = append(parts, fmt.Sprintf("type=%s", docType))
|
||||
}
|
||||
|
||||
idsStr, _ := cmd.Flags().GetString("ids")
|
||||
if idsStr != "" {
|
||||
parts = append(parts, fmt.Sprintf("ids=[%s]", idsStr))
|
||||
}
|
||||
|
||||
fromID, _ := cmd.Flags().GetInt("from-id")
|
||||
if fromID > 0 {
|
||||
parts = append(parts, fmt.Sprintf("from_id=%d", fromID))
|
||||
}
|
||||
|
||||
toID, _ := cmd.Flags().GetInt("to-id")
|
||||
if toID > 0 {
|
||||
parts = append(parts, fmt.Sprintf("to_id=%d", toID))
|
||||
}
|
||||
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
|
||||
func printBulkResult(action string, result map[string]interface{}) {
|
||||
matched := int(result["matched"].(float64))
|
||||
succeeded := int(result["succeeded"].(float64))
|
||||
failed := int(result["failed"].(float64))
|
||||
|
||||
fmt.Printf("%s %d of %d documents", action, succeeded, matched)
|
||||
if failed > 0 {
|
||||
fmt.Printf(" (%d failed)", failed)
|
||||
}
|
||||
fmt.Println()
|
||||
}
|
||||
|
||||
func parseIntList(s string) ([]int, error) {
|
||||
var ids []int
|
||||
for _, part := range strings.Split(s, ",") {
|
||||
part = strings.TrimSpace(part)
|
||||
if part == "" {
|
||||
continue
|
||||
}
|
||||
id, err := strconv.Atoi(part)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid ID %q: %w", part, err)
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var bulkSetTagsCmd = &cobra.Command{
|
||||
Use: "bulk-set-tags",
|
||||
Short: "Replace all tags on multiple documents matching a filter",
|
||||
RunE: runBulkSetTags,
|
||||
}
|
||||
|
||||
func init() {
|
||||
addBulkFilterFlags(bulkSetTagsCmd)
|
||||
bulkSetTagsCmd.Flags().String("set", "", "replacement tags (comma-separated)")
|
||||
rootCmd.AddCommand(bulkSetTagsCmd)
|
||||
}
|
||||
|
||||
func runBulkSetTags(cmd *cobra.Command, args []string) error {
|
||||
body, err := buildBulkBody(cmd)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
setStr, _ := cmd.Flags().GetString("set")
|
||||
if setStr == "" {
|
||||
return fmt.Errorf("--set is required (comma-separated list of replacement tags)")
|
||||
}
|
||||
body["new_tags"] = splitTags(setStr)
|
||||
|
||||
yes, _ := cmd.Flags().GetBool("yes")
|
||||
if !yes {
|
||||
desc := describeBulkFilter(cmd)
|
||||
fmt.Printf("This will replace all tags with [%s] on documents matching: %s\nProceed? [y/N] ", setStr, desc)
|
||||
reader := bufio.NewReader(os.Stdin)
|
||||
answer, _ := reader.ReadString('\n')
|
||||
answer = strings.TrimSpace(strings.ToLower(answer))
|
||||
if answer != "y" && answer != "yes" {
|
||||
fmt.Println("Cancelled.")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
client := api.NewClient()
|
||||
resp, err := client.Post("/api/v1/bulk/set-tags", body)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
var result map[string]interface{}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(result)
|
||||
} else {
|
||||
printBulkResult("Set tags on", result)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var bulkTagCmd = &cobra.Command{
|
||||
Use: "bulk-tag",
|
||||
Short: "Add or remove tags on multiple documents matching a filter",
|
||||
RunE: runBulkTag,
|
||||
}
|
||||
|
||||
func init() {
|
||||
addBulkFilterFlags(bulkTagCmd)
|
||||
bulkTagCmd.Flags().String("add", "", "tags to add (comma-separated)")
|
||||
bulkTagCmd.Flags().String("remove", "", "tags to remove (comma-separated)")
|
||||
rootCmd.AddCommand(bulkTagCmd)
|
||||
}
|
||||
|
||||
func runBulkTag(cmd *cobra.Command, args []string) error {
|
||||
body, err := buildBulkBody(cmd)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
addStr, _ := cmd.Flags().GetString("add")
|
||||
removeStr, _ := cmd.Flags().GetString("remove")
|
||||
|
||||
if addStr == "" && removeStr == "" {
|
||||
return fmt.Errorf("specify --add and/or --remove")
|
||||
}
|
||||
|
||||
if addStr != "" {
|
||||
body["add"] = splitTags(addStr)
|
||||
}
|
||||
if removeStr != "" {
|
||||
body["remove"] = splitTags(removeStr)
|
||||
}
|
||||
|
||||
yes, _ := cmd.Flags().GetBool("yes")
|
||||
if !yes {
|
||||
desc := describeBulkFilter(cmd)
|
||||
action := ""
|
||||
if addStr != "" {
|
||||
action += fmt.Sprintf("add=[%s]", addStr)
|
||||
}
|
||||
if removeStr != "" {
|
||||
if action != "" {
|
||||
action += " "
|
||||
}
|
||||
action += fmt.Sprintf("remove=[%s]", removeStr)
|
||||
}
|
||||
fmt.Printf("This will update tags (%s) on documents matching: %s\nProceed? [y/N] ", action, desc)
|
||||
reader := bufio.NewReader(os.Stdin)
|
||||
answer, _ := reader.ReadString('\n')
|
||||
answer = strings.TrimSpace(strings.ToLower(answer))
|
||||
if answer != "y" && answer != "yes" {
|
||||
fmt.Println("Cancelled.")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
client := api.NewClient()
|
||||
resp, err := client.Post("/api/v1/bulk/tags", body)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
var result map[string]interface{}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(result)
|
||||
} else {
|
||||
printBulkResult("Tagged", result)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+11
-4
@@ -11,21 +11,28 @@ var examplesCmd = &cobra.Command{
|
||||
Short: "Show common usage examples",
|
||||
Args: cobra.NoArgs,
|
||||
Run: func(cmd *cobra.Command, args []string) {
|
||||
fmt.Print(`Quick notes:
|
||||
kb "Remember to update DNS records"
|
||||
kb "Server room is building 3" --tags ops
|
||||
fmt.Print(`Add notes:
|
||||
kb addnote "Remember to update DNS records"
|
||||
kb addnote "Server room is building 3" --tags ops
|
||||
kb addnote "Deploy checklist" --wait
|
||||
|
||||
Add files:
|
||||
kb addfile report.pdf
|
||||
kb addfile ~/docs/ --recursive --tags reference
|
||||
kb addfile report.pdf --wait
|
||||
|
||||
Search:
|
||||
kb search "how to restart nginx"
|
||||
kb search "deploy" --tags ops --top 5
|
||||
kb find "quarterly report" --type pdf
|
||||
|
||||
Update notes:
|
||||
kb updatenote 42 "revised note content"
|
||||
|
||||
Manage documents:
|
||||
kb list --type pdf
|
||||
kb info 3
|
||||
kb list --filename report.pdf
|
||||
kb info 3 --no-chunks
|
||||
kb tag 3 --add important,ops
|
||||
kb remove 3 --yes
|
||||
`)
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var findCmd = &cobra.Command{
|
||||
Use: "find <query>",
|
||||
Short: "Find documents by their indexed content",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: runFind,
|
||||
}
|
||||
|
||||
func init() {
|
||||
findCmd.Flags().IntP("top", "n", 10, "number of documents to return")
|
||||
findCmd.Flags().String("tags", "", "filter by tags (comma-separated)")
|
||||
findCmd.Flags().String("type", "", "filter by document type")
|
||||
rootCmd.AddCommand(findCmd)
|
||||
}
|
||||
|
||||
func runFind(cmd *cobra.Command, args []string) error {
|
||||
top, _ := cmd.Flags().GetInt("top")
|
||||
tags, _ := cmd.Flags().GetString("tags")
|
||||
docType, _ := cmd.Flags().GetString("type")
|
||||
body := map[string]interface{}{"query": args[0], "top": top}
|
||||
if tags != "" {
|
||||
body["tags"] = splitTags(tags)
|
||||
}
|
||||
if docType != "" {
|
||||
body["doc_type"] = docType
|
||||
}
|
||||
|
||||
resp, err := api.NewClient().Post("/api/v1/documents/find", body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
return err
|
||||
}
|
||||
if output.IsJSON() {
|
||||
var raw interface{}
|
||||
if err := api.DecodeJSON(resp, &raw); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
output.PrintJSON(raw)
|
||||
return nil
|
||||
}
|
||||
|
||||
var docs []struct {
|
||||
DocumentID int `json:"document_id"`
|
||||
Title string `json:"title"`
|
||||
Filename string `json:"original_filename"`
|
||||
Type string `json:"doc_type"`
|
||||
Score float64 `json:"score"`
|
||||
HitCount int `json:"hit_count"`
|
||||
TopChunk string `json:"top_chunk"`
|
||||
}
|
||||
if err := api.DecodeJSON(resp, &docs); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
if len(docs) == 0 {
|
||||
fmt.Println("No documents found.")
|
||||
return nil
|
||||
}
|
||||
for i, doc := range docs {
|
||||
preview := doc.TopChunk
|
||||
if len(preview) > 200 {
|
||||
preview = preview[:200] + "..."
|
||||
}
|
||||
fmt.Printf("\n%d. [%.4f] %s (doc:%d, hits:%d)\n", i+1, doc.Score, doc.Title, doc.DocumentID, doc.HitCount)
|
||||
if doc.Filename != "" {
|
||||
fmt.Printf(" Filename: %s\n", doc.Filename)
|
||||
}
|
||||
if doc.Type != "" {
|
||||
fmt.Printf(" Type: %s\n", doc.Type)
|
||||
}
|
||||
fmt.Printf(" %s\n", preview)
|
||||
}
|
||||
fmt.Println()
|
||||
return nil
|
||||
}
|
||||
+9
-2
@@ -17,12 +17,18 @@ var infoCmd = &cobra.Command{
|
||||
}
|
||||
|
||||
func init() {
|
||||
infoCmd.Flags().Bool("no-chunks", false, "return document metadata without chunk details")
|
||||
rootCmd.AddCommand(infoCmd)
|
||||
}
|
||||
|
||||
func runInfo(cmd *cobra.Command, args []string) error {
|
||||
client := api.NewClient()
|
||||
resp, err := client.Get("/api/v1/documents/" + args[0])
|
||||
noChunks, _ := cmd.Flags().GetBool("no-chunks")
|
||||
path := "/api/v1/documents/" + args[0]
|
||||
if noChunks {
|
||||
path += "?include_chunks=false"
|
||||
}
|
||||
resp, err := client.Get(path)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
@@ -48,6 +54,7 @@ func runInfo(cmd *cobra.Command, args []string) error {
|
||||
Tags []string `json:"tags"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
ChunkCount int `json:"chunk_count"`
|
||||
Chunks []struct {
|
||||
ID int `json:"id"`
|
||||
Page interface{} `json:"page"`
|
||||
@@ -65,7 +72,7 @@ func runInfo(cmd *cobra.Command, args []string) error {
|
||||
{"Tags", joinStrings(doc.Tags)},
|
||||
{"Created", doc.CreatedAt},
|
||||
{"Updated", doc.UpdatedAt},
|
||||
{"Chunks", fmt.Sprintf("%d", len(doc.Chunks))},
|
||||
{"Chunks", fmt.Sprintf("%d", doc.ChunkCount)},
|
||||
}
|
||||
output.PrintKeyValue(pairs)
|
||||
|
||||
|
||||
+14
-2
@@ -13,18 +13,23 @@ import (
|
||||
var listCmd = &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "List documents in the knowledge base",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runList,
|
||||
}
|
||||
|
||||
func init() {
|
||||
listCmd.Flags().String("type", "", "filter by document type")
|
||||
listCmd.Flags().String("tags", "", "filter by tags (comma-separated)")
|
||||
listCmd.Flags().String("title", "", "filter by title substring")
|
||||
listCmd.Flags().String("filename", "", "filter by original filename substring")
|
||||
rootCmd.AddCommand(listCmd)
|
||||
}
|
||||
|
||||
func runList(cmd *cobra.Command, args []string) error {
|
||||
docType, _ := cmd.Flags().GetString("type")
|
||||
tags, _ := cmd.Flags().GetString("tags")
|
||||
title, _ := cmd.Flags().GetString("title")
|
||||
filename, _ := cmd.Flags().GetString("filename")
|
||||
|
||||
params := url.Values{}
|
||||
if docType != "" {
|
||||
@@ -33,6 +38,12 @@ func runList(cmd *cobra.Command, args []string) error {
|
||||
if tags != "" {
|
||||
params.Set("tags", tags)
|
||||
}
|
||||
if title != "" {
|
||||
params.Set("title", title)
|
||||
}
|
||||
if filename != "" {
|
||||
params.Set("filename", filename)
|
||||
}
|
||||
|
||||
path := "/api/v1/documents"
|
||||
if len(params) > 0 {
|
||||
@@ -62,6 +73,7 @@ func runList(cmd *cobra.Command, args []string) error {
|
||||
var docs []struct {
|
||||
ID int `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Filename string `json:"original_filename"`
|
||||
Type string `json:"doc_type"`
|
||||
Tags []string `json:"tags"`
|
||||
}
|
||||
@@ -74,10 +86,10 @@ func runList(cmd *cobra.Command, args []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
headers := []string{"ID", "TITLE", "TYPE", "TAGS"}
|
||||
headers := []string{"ID", "TITLE", "FILENAME", "TYPE", "TAGS"}
|
||||
var rows [][]string
|
||||
for _, d := range docs {
|
||||
rows = append(rows, []string{fmt.Sprintf("%d", d.ID), d.Title, d.Type, joinStrings(d.Tags)})
|
||||
rows = append(rows, []string{fmt.Sprintf("%d", d.ID), d.Title, d.Filename, d.Type, joinStrings(d.Tags)})
|
||||
}
|
||||
output.PrintTable(headers, rows)
|
||||
return nil
|
||||
|
||||
+1
-33
@@ -3,7 +3,6 @@ package cmd
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/config"
|
||||
@@ -23,10 +22,9 @@ var (
|
||||
)
|
||||
|
||||
var rootCmd = &cobra.Command{
|
||||
Use: "kb [\"note text\" | command]",
|
||||
Use: "kb [command]",
|
||||
Short: "kb-search CLI client",
|
||||
Long: "A CLI client for the kb-search v2 engine API.\nRun 'kb examples' for common usage patterns.",
|
||||
Args: cobra.ArbitraryArgs,
|
||||
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
|
||||
if err := config.Load(); err != nil {
|
||||
return err
|
||||
@@ -34,44 +32,14 @@ var rootCmd = &cobra.Command{
|
||||
config.ApplyFlags(flagEngine, flagFormat, flagAPIKey)
|
||||
return nil
|
||||
},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if len(args) == 0 {
|
||||
return cmd.Help()
|
||||
}
|
||||
if len(args) == 1 {
|
||||
return fmt.Errorf("unknown command %q\nTo add a note, use: kb \"%s ...\" or pass multiple words", args[0], args[0])
|
||||
}
|
||||
note := strings.Join(args, " ")
|
||||
tags, _ := cmd.Flags().GetString("tags")
|
||||
client := api.NewClient()
|
||||
return submitNote(client, note, tags)
|
||||
},
|
||||
}
|
||||
|
||||
func init() {
|
||||
api.SetVersionInfo(Version, MinEngineVersion)
|
||||
rootCmd.Version = Version
|
||||
rootCmd.SetUsageTemplate(`Quick note taking (must be more than one word):
|
||||
kb "note text here" [flags]
|
||||
|
||||
Normal usage:
|
||||
kb [command] [flags]{{if .HasAvailableSubCommands}}
|
||||
|
||||
Available Commands:{{range .Commands}}{{if (or .IsAvailableCommand (eq .Name "help"))}}
|
||||
{{rpad .Name .NamePadding }} {{.Short}}{{end}}{{end}}{{end}}{{if .HasAvailableLocalFlags}}
|
||||
|
||||
Flags:
|
||||
{{.LocalFlags.FlagUsages | trimTrailingWhitespaces}}{{end}}{{if .HasAvailableInheritedFlags}}
|
||||
|
||||
Global Flags:
|
||||
{{.InheritedFlags.FlagUsages | trimTrailingWhitespaces}}{{end}}
|
||||
|
||||
Use "{{.CommandPath}} [command] --help" for more information about a command.
|
||||
`)
|
||||
rootCmd.PersistentFlags().StringVar(&flagEngine, "engine", "", "engine API URL")
|
||||
rootCmd.PersistentFlags().StringVar(&flagFormat, "format", "", "output format (human|json)")
|
||||
rootCmd.PersistentFlags().StringVar(&flagAPIKey, "api-key", "", "API key for authentication")
|
||||
rootCmd.Flags().String("tags", "", "tags for note shorthand (comma-separated)")
|
||||
}
|
||||
|
||||
// Execute runs the root command.
|
||||
|
||||
+58
-30
@@ -6,36 +6,6 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRootCmd_SingleWordRejected(t *testing.T) {
|
||||
rootCmd.SetArgs([]string{"infow"})
|
||||
|
||||
var stderr bytes.Buffer
|
||||
rootCmd.SetErr(&stderr)
|
||||
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for single bare word, got nil")
|
||||
}
|
||||
|
||||
errMsg := err.Error()
|
||||
if !strings.Contains(errMsg, `unknown command "infow"`) {
|
||||
t.Errorf("expected error to mention unknown command, got: %s", errMsg)
|
||||
}
|
||||
if !strings.Contains(errMsg, "multiple words") {
|
||||
t.Errorf("expected error to suggest multiple words, got: %s", errMsg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRootCmd_MultipleWordsNotRejected(t *testing.T) {
|
||||
rootCmd.SetArgs([]string{"remember", "to", "update", "dns"})
|
||||
|
||||
err := rootCmd.Execute()
|
||||
// Will fail at API call (no server), but should NOT be the "unknown command" error
|
||||
if err != nil && strings.Contains(err.Error(), "unknown command") {
|
||||
t.Errorf("multi-word input should not be rejected as unknown command, got: %s", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRootCmd_NoArgs_ShowsHelp(t *testing.T) {
|
||||
rootCmd.SetArgs([]string{})
|
||||
|
||||
@@ -52,3 +22,61 @@ func TestRootCmd_NoArgs_ShowsHelp(t *testing.T) {
|
||||
t.Errorf("expected help output, got: %s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRootCmd_UnknownCommand_ReturnsError(t *testing.T) {
|
||||
rootCmd.SetArgs([]string{"notacommand"})
|
||||
|
||||
var stderr bytes.Buffer
|
||||
rootCmd.SetErr(&stderr)
|
||||
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for unknown command, got nil")
|
||||
}
|
||||
|
||||
errMsg := err.Error()
|
||||
if !strings.Contains(errMsg, "unknown command") {
|
||||
t.Errorf("expected 'unknown command' error, got: %s", errMsg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddnoteCmd_NoArgs_ReturnsError(t *testing.T) {
|
||||
rootCmd.SetArgs([]string{"addnote"})
|
||||
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for addnote with no args, got nil")
|
||||
}
|
||||
|
||||
errMsg := err.Error()
|
||||
if !strings.Contains(errMsg, "requires a note text argument") {
|
||||
t.Errorf("expected 'requires a note text argument' error, got: %s", errMsg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddnoteCmd_TooManyArgs_ReturnsError(t *testing.T) {
|
||||
rootCmd.SetArgs([]string{"addnote", "hello", "world"})
|
||||
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for addnote with too many args, got nil")
|
||||
}
|
||||
|
||||
errMsg := err.Error()
|
||||
if !strings.Contains(errMsg, "quote your note text") {
|
||||
t.Errorf("expected 'accepts 1 arg' error, got: %s", errMsg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListCmd_PositionalArgReturnsError(t *testing.T) {
|
||||
rootCmd.SetArgs([]string{"list", "ignored-title"})
|
||||
|
||||
err := rootCmd.Execute()
|
||||
if err == nil {
|
||||
t.Fatal("expected error for positional list argument, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "unknown command") &&
|
||||
!strings.Contains(err.Error(), "accepts 0 arg") {
|
||||
t.Errorf("expected positional argument error, got: %s", err)
|
||||
}
|
||||
}
|
||||
|
||||
+84
-16
@@ -23,6 +23,8 @@ func init() {
|
||||
searchCmd.Flags().Bool("fts-only", false, "use full-text search only")
|
||||
searchCmd.Flags().Bool("vec-only", false, "use vector search only")
|
||||
searchCmd.Flags().Float64("threshold", 0, "minimum score threshold")
|
||||
searchCmd.Flags().Bool("explain", false, "include per-result score breakdown")
|
||||
searchCmd.Flags().Bool("no-rerank", false, "skip cross-encoder reranking (lower latency)")
|
||||
rootCmd.AddCommand(searchCmd)
|
||||
}
|
||||
|
||||
@@ -33,16 +35,18 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
ftsOnly, _ := cmd.Flags().GetBool("fts-only")
|
||||
vecOnly, _ := cmd.Flags().GetBool("vec-only")
|
||||
threshold, _ := cmd.Flags().GetFloat64("threshold")
|
||||
explain, _ := cmd.Flags().GetBool("explain")
|
||||
noRerank, _ := cmd.Flags().GetBool("no-rerank")
|
||||
|
||||
body := map[string]interface{}{
|
||||
"query": args[0],
|
||||
"top": top,
|
||||
}
|
||||
if tags != "" {
|
||||
body["tags"] = tags
|
||||
body["tags"] = splitTags(tags)
|
||||
}
|
||||
if docType != "" {
|
||||
body["type"] = docType
|
||||
body["doc_type"] = docType
|
||||
}
|
||||
if ftsOnly {
|
||||
body["fts_only"] = true
|
||||
@@ -53,6 +57,12 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
if threshold > 0 {
|
||||
body["threshold"] = threshold
|
||||
}
|
||||
if explain {
|
||||
body["explain"] = true
|
||||
}
|
||||
if noRerank {
|
||||
body["rerank"] = false
|
||||
}
|
||||
|
||||
client := api.NewClient()
|
||||
resp, err := client.Post("/api/v1/search", body)
|
||||
@@ -66,16 +76,17 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
}
|
||||
|
||||
var result struct {
|
||||
Reranked bool `json:"reranked"`
|
||||
Results []struct {
|
||||
Score float64 `json:"score"`
|
||||
Document struct {
|
||||
DocumentID int64 `json:"document_id"`
|
||||
Title string `json:"title"`
|
||||
Type string `json:"doc_type"`
|
||||
DocType string `json:"doc_type"`
|
||||
Tags []string `json:"tags"`
|
||||
} `json:"document"`
|
||||
Page interface{} `json:"page"`
|
||||
Section string `json:"section"`
|
||||
TagContexts map[string]string `json:"tag_contexts"`
|
||||
ChunkMetadata map[string]interface{} `json:"chunk_metadata"`
|
||||
Text string `json:"text"`
|
||||
Explain map[string]interface{} `json:"explain"`
|
||||
} `json:"results"`
|
||||
}
|
||||
|
||||
@@ -97,32 +108,46 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
if result.Reranked {
|
||||
fmt.Println("(reranked)")
|
||||
}
|
||||
|
||||
for i, r := range result.Results {
|
||||
snippet := r.Text
|
||||
if len(snippet) > 200 {
|
||||
snippet = snippet[:200] + "..."
|
||||
}
|
||||
|
||||
fmt.Printf("\n%d. [%.4f] %s\n", i+1, r.Score, r.Document.Title)
|
||||
fmt.Printf("\n%d. [%.4f] %s (doc:%d)\n", i+1, r.Score, r.Title, r.DocumentID)
|
||||
|
||||
location := ""
|
||||
if r.Page != nil {
|
||||
location = fmt.Sprintf("Page %v", r.Page)
|
||||
if page, ok := r.ChunkMetadata["page"]; ok && page != nil {
|
||||
location = fmt.Sprintf("Page %v", page)
|
||||
}
|
||||
if r.Section != "" {
|
||||
if section, ok := r.ChunkMetadata["section_header"]; ok && section != nil {
|
||||
if s, ok := section.(string); ok && s != "" {
|
||||
if location != "" {
|
||||
location += " / "
|
||||
}
|
||||
location += r.Section
|
||||
location += s
|
||||
}
|
||||
}
|
||||
if location != "" {
|
||||
fmt.Printf(" Location: %s\n", location)
|
||||
}
|
||||
if r.Document.Type != "" {
|
||||
fmt.Printf(" Type: %s\n", r.Document.Type)
|
||||
if r.DocType != "" {
|
||||
fmt.Printf(" Type: %s\n", r.DocType)
|
||||
}
|
||||
if len(r.Document.Tags) > 0 {
|
||||
fmt.Printf(" Tags: %s\n", joinStrings(r.Document.Tags))
|
||||
if len(r.Tags) > 0 {
|
||||
fmt.Printf(" Tags: %s\n", joinStrings(r.Tags))
|
||||
}
|
||||
for _, tag := range r.Tags {
|
||||
if desc, ok := r.TagContexts[tag]; ok && desc != "" {
|
||||
fmt.Printf(" Context: %s — %s\n", tag, desc)
|
||||
}
|
||||
}
|
||||
if r.Explain != nil {
|
||||
fmt.Printf(" Score: %s\n", formatExplain(r.Explain))
|
||||
}
|
||||
fmt.Printf(" %s\n", snippet)
|
||||
}
|
||||
@@ -130,6 +155,49 @@ func runSearch(cmd *cobra.Command, args []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// formatExplain renders the engine's explain breakdown as one line, e.g.
|
||||
// "fts score=4.213 rank=2 (rrf 0.016129) | vec score=0.512 rank=1 (rrf 0.016393) | bonus 0.05 | final 0.082522"
|
||||
func formatExplain(e map[string]interface{}) string {
|
||||
num := func(key string) (float64, bool) {
|
||||
v, ok := e[key].(float64)
|
||||
return v, ok
|
||||
}
|
||||
var parts []string
|
||||
for _, arm := range []string{"fts", "vec"} {
|
||||
score, hasScore := num(arm + "_score")
|
||||
if !hasScore {
|
||||
continue
|
||||
}
|
||||
part := fmt.Sprintf("%s score=%.4f", arm, score)
|
||||
if rank, ok := num(arm + "_rank"); ok {
|
||||
part += fmt.Sprintf(" rank=%d", int(rank))
|
||||
}
|
||||
if rrf, ok := num("rrf_" + arm); ok {
|
||||
part += fmt.Sprintf(" (rrf %.6f)", rrf)
|
||||
}
|
||||
parts = append(parts, part)
|
||||
}
|
||||
if bonus, ok := num("bonus"); ok && bonus > 0 {
|
||||
parts = append(parts, fmt.Sprintf("bonus %.2f", bonus))
|
||||
}
|
||||
for _, key := range []string{"pre_rerank_rank", "rerank_score", "blend_weight"} {
|
||||
if v, ok := num(key); ok {
|
||||
parts = append(parts, fmt.Sprintf("%s %.4f", key, v))
|
||||
}
|
||||
}
|
||||
if final, ok := num("final_score"); ok {
|
||||
parts = append(parts, fmt.Sprintf("final %.6f", final))
|
||||
}
|
||||
result := ""
|
||||
for i, p := range parts {
|
||||
if i > 0 {
|
||||
result += " | "
|
||||
}
|
||||
result += p
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func joinStrings(ss []string) string {
|
||||
result := ""
|
||||
for i, s := range ss {
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSearchCmd_SendsDocTypeAndTagsList(t *testing.T) {
|
||||
var captured map[string]interface{}
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/api/v1/search" {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
}
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
if err := json.Unmarshal(body, &captured); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Write([]byte(`{"query":"q","results":[],"total_matches":0,"returned":0}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
var stdout bytes.Buffer
|
||||
rootCmd.SetOut(&stdout)
|
||||
rootCmd.SetArgs([]string{
|
||||
"search", "oil level",
|
||||
"--engine", server.URL,
|
||||
"--format", "json",
|
||||
"--type", "pdf",
|
||||
"--tags", "manuals, ops,",
|
||||
})
|
||||
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
t.Fatalf("search command failed: %v", err)
|
||||
}
|
||||
|
||||
if captured == nil {
|
||||
t.Fatal("no request captured by test server")
|
||||
}
|
||||
if got := captured["doc_type"]; got != "pdf" {
|
||||
t.Errorf("expected doc_type=pdf in body, got %v (full body: %v)", got, captured)
|
||||
}
|
||||
if _, present := captured["type"]; present {
|
||||
t.Error("body must not contain legacy 'type' key")
|
||||
}
|
||||
tags, ok := captured["tags"].([]interface{})
|
||||
if !ok {
|
||||
t.Fatalf("expected tags to be a JSON array, got %T (%v)", captured["tags"], captured["tags"])
|
||||
}
|
||||
want := []interface{}{"manuals", "ops"}
|
||||
if !reflect.DeepEqual(tags, want) {
|
||||
t.Errorf("expected tags %v, got %v", want, tags)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitTags(t *testing.T) {
|
||||
got := splitTags(" a ,b,, c ")
|
||||
want := []string{"a", "b", "c"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("splitTags: expected %v, got %v", want, got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var tagDescribeCmd = &cobra.Command{
|
||||
Use: "tag-describe <tag> [description]",
|
||||
Short: "Set a one-line context description on a tag",
|
||||
Long: `Attach a short description to a tag (e.g. "Lab operations runbooks").
|
||||
Descriptions are returned as tag_contexts with every search result on a
|
||||
document carrying the tag, helping consumers judge relevance.
|
||||
|
||||
Omit the description (or pass "") to clear it.`,
|
||||
Args: cobra.RangeArgs(1, 2),
|
||||
RunE: runTagDescribe,
|
||||
}
|
||||
|
||||
func init() {
|
||||
rootCmd.AddCommand(tagDescribeCmd)
|
||||
}
|
||||
|
||||
func runTagDescribe(cmd *cobra.Command, args []string) error {
|
||||
description := ""
|
||||
if len(args) == 2 {
|
||||
description = args[1]
|
||||
}
|
||||
|
||||
body := map[string]interface{}{"description": description}
|
||||
|
||||
client := api.NewClient()
|
||||
resp, err := client.Put("/api/v1/tags/"+args[0]+"/description", body)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
var raw interface{}
|
||||
if err := api.DecodeJSON(resp, &raw); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
output.PrintJSON(raw)
|
||||
return nil
|
||||
}
|
||||
|
||||
if description == "" {
|
||||
fmt.Printf("Description cleared for tag %q\n", args[0])
|
||||
} else {
|
||||
fmt.Printf("Tag %q: %s\n", args[0], description)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+3
-2
@@ -43,6 +43,7 @@ func runTags(cmd *cobra.Command, args []string) error {
|
||||
var tags []struct {
|
||||
Name string `json:"name"`
|
||||
Count int `json:"count"`
|
||||
Description string `json:"description"`
|
||||
}
|
||||
if err := api.DecodeJSON(resp, &tags); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
@@ -53,10 +54,10 @@ func runTags(cmd *cobra.Command, args []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
headers := []string{"TAG", "COUNT"}
|
||||
headers := []string{"TAG", "COUNT", "DESCRIPTION"}
|
||||
var rows [][]string
|
||||
for _, t := range tags {
|
||||
rows = append(rows, []string{t.Name, fmt.Sprintf("%d", t.Count)})
|
||||
rows = append(rows, []string{t.Name, fmt.Sprintf("%d", t.Count), t.Description})
|
||||
}
|
||||
output.PrintTable(headers, rows)
|
||||
return nil
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
"github.com/kb-search/kb/internal/output"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var updatenoteCmd = &cobra.Command{
|
||||
Use: "updatenote <id> <text>",
|
||||
Short: "Update an existing note's content",
|
||||
Args: func(cmd *cobra.Command, args []string) error {
|
||||
if len(args) < 2 {
|
||||
return fmt.Errorf("requires document ID and text arguments\n\n Usage: kb updatenote 42 \"updated note text\"")
|
||||
}
|
||||
if _, err := strconv.Atoi(args[0]); err != nil {
|
||||
return fmt.Errorf("document ID must be an integer, got %q", args[0])
|
||||
}
|
||||
return nil
|
||||
},
|
||||
RunE: runUpdatenote,
|
||||
}
|
||||
|
||||
func init() {
|
||||
rootCmd.AddCommand(updatenoteCmd)
|
||||
}
|
||||
|
||||
func runUpdatenote(cmd *cobra.Command, args []string) error {
|
||||
docID := args[0]
|
||||
text := args[1]
|
||||
|
||||
client := api.NewClient()
|
||||
|
||||
body := map[string]string{"text": text}
|
||||
resp, err := client.Patch(fmt.Sprintf("/api/v1/notes/%s", docID), body)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
var result interface{}
|
||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||
return fmt.Errorf("failed to decode response: %w", err)
|
||||
}
|
||||
|
||||
if output.IsJSON() {
|
||||
output.PrintJSON(result)
|
||||
} else {
|
||||
fmt.Printf("Updated note %s\n", docID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/kb-search/kb/internal/api"
|
||||
)
|
||||
|
||||
type jobStatus struct {
|
||||
ID int `json:"id"`
|
||||
Status string `json:"status"`
|
||||
DocumentID int `json:"document_id"`
|
||||
ChunkCount int `json:"chunk_count"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
|
||||
func waitForJob(client *api.Client, jobID int, timeout time.Duration) (*jobStatus, error) {
|
||||
deadline := time.Now().Add(timeout)
|
||||
for {
|
||||
resp, err := client.Get(fmt.Sprintf("/api/v1/jobs/%d", jobID))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := api.CheckError(resp); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var job jobStatus
|
||||
if err := api.DecodeJSON(resp, &job); err != nil {
|
||||
return nil, fmt.Errorf("failed to decode job status: %w", err)
|
||||
}
|
||||
switch job.Status {
|
||||
case "done", "skipped":
|
||||
return &job, nil
|
||||
case "failed":
|
||||
return nil, fmt.Errorf("ingestion job %d failed: %s", jobID, job.Error)
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return nil, fmt.Errorf("timed out waiting for ingestion job %d", jobID)
|
||||
}
|
||||
time.Sleep(time.Second)
|
||||
}
|
||||
}
|
||||
@@ -94,6 +94,10 @@ func (c *Client) checkEngineVersion() {
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return // auth error or other issue — let the actual request surface it
|
||||
}
|
||||
|
||||
var status struct {
|
||||
Version string `json:"version"`
|
||||
}
|
||||
@@ -217,6 +221,20 @@ func (c *Client) Put(path string, body interface{}) (*http.Response, error) {
|
||||
return c.do(req)
|
||||
}
|
||||
|
||||
// Patch performs a PATCH request with a JSON body.
|
||||
func (c *Client) Patch(path string, body interface{}) (*http.Response, error) {
|
||||
data, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to marshal request body: %w", err)
|
||||
}
|
||||
req, err := c.newRequest(http.MethodPatch, path, bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
return c.do(req)
|
||||
}
|
||||
|
||||
// DecodeJSON reads the response body and decodes it into target.
|
||||
func DecodeJSON(resp *http.Response, target interface{}) error {
|
||||
defer resp.Body.Close()
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"description": "Example search-quality fixture — copy and adapt against your own corpus",
|
||||
"top": 10,
|
||||
"queries": [
|
||||
{
|
||||
"id": "manual-lookup",
|
||||
"query": "how do I check the oil level",
|
||||
"doc_type": "pdf",
|
||||
"relevant": [
|
||||
{"document_id": 2077},
|
||||
{"title_contains": "owner's manual"}
|
||||
],
|
||||
"notes": "Selectors are OR-matched; each counts as one relevant document. Exactly one of document_id / source_path / title_contains per selector."
|
||||
},
|
||||
{
|
||||
"id": "infra-runbook",
|
||||
"query": "restart the reverse proxy after certificate renewal",
|
||||
"tags": ["ops"],
|
||||
"relevant": [
|
||||
{"source_path": "/data/notes/proxy-runbook.md"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "note-recall",
|
||||
"query": "what did we decide about the backup retention window",
|
||||
"top": 5,
|
||||
"relevant": [
|
||||
{"title_contains": "backup retention"}
|
||||
],
|
||||
"notes": "Per-query top overrides the fixture-level default; the --top flag overrides both."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>kb-search Enhancements Proposal</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #f7f7f5; --fg: #1a1a1a; --muted: #666; --card: #fff;
|
||||
--border: #ddd; --accent: #2563eb; --code-bg: #eef1f5;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--bg: #16181d; --fg: #e6e6e6; --muted: #9aa0a8; --card: #1e2128;
|
||||
--border: #33363e; --accent: #7aa2f7; --code-bg: #262a33;
|
||||
}
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0; padding: 2rem 1rem 4rem; background: var(--bg); color: var(--fg);
|
||||
font: 16px/1.6 -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||
}
|
||||
main { max-width: 860px; margin: 0 auto; }
|
||||
h1 { font-size: 1.9rem; margin-bottom: .2rem; }
|
||||
h2 { margin-top: 2.2rem; border-bottom: 1px solid var(--border); padding-bottom: .3rem; }
|
||||
.meta { color: var(--muted); font-size: .9rem; margin-bottom: 2rem; }
|
||||
.card {
|
||||
background: var(--card); border: 1px solid var(--border); border-radius: 10px;
|
||||
padding: 1rem 1.3rem; margin: 1rem 0;
|
||||
}
|
||||
.card h3 { margin: .2rem 0 .5rem; }
|
||||
.badge {
|
||||
display: inline-block; font-size: .72rem; font-weight: 600; letter-spacing: .03em;
|
||||
padding: .12rem .55rem; border-radius: 999px; vertical-align: middle; margin-left: .5rem;
|
||||
}
|
||||
.b-high { background: #dc262622; color: #dc2626; }
|
||||
.b-med { background: #d9770622; color: #d97706; }
|
||||
.b-low { background: #05966922; color: #059669; }
|
||||
table { border-collapse: collapse; width: 100%; margin: 1rem 0; font-size: .93rem; }
|
||||
th, td { border: 1px solid var(--border); padding: .45rem .7rem; text-align: left; vertical-align: top; }
|
||||
th { background: var(--code-bg); }
|
||||
code { background: var(--code-bg); padding: .1rem .35rem; border-radius: 4px; font-size: .88em; }
|
||||
pre { background: var(--code-bg); padding: .8rem 1rem; border-radius: 8px; overflow-x: auto; }
|
||||
pre code { background: none; padding: 0; }
|
||||
a { color: var(--accent); }
|
||||
.muted { color: var(--muted); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>kb-search Enhancements Proposal</h1>
|
||||
<p class="meta">Bobby · 2026-07-07 · Prompted by a review of <a href="https://github.com/tobi/qmd">tobi/qmd</a> (Tobi Lütke's local hybrid search engine)</p>
|
||||
|
||||
<h2>Summary</h2>
|
||||
<p>qmd and kb-search v2 solve overlapping problems, but qmd's retrieval pipeline is measurably ahead: its own benchmarks show BM25-only at ~0.50, vector-only at ~0.70, and the full hybrid + reranked pipeline at ~1.00. Our kb does hybrid FTS + vector but stops there — no rank fusion, no reranking, no query expansion, and no way to measure whether a change helps or hurts. This proposal lists five enhancements, ordered by value-for-effort, plus the already-tracked JSON ingestion item.</p>
|
||||
|
||||
<table>
|
||||
<tr><th>#</th><th>Enhancement</th><th>Impact</th><th>Effort</th></tr>
|
||||
<tr><td>1</td><td>LLM reranking stage</td><td>High — biggest single search-quality lever</td><td>Medium</td></tr>
|
||||
<tr><td>2</td><td>RRF fusion for FTS + vector merging</td><td>Medium-high</td><td>Low</td></tr>
|
||||
<tr><td>3</td><td>Bench harness + <code>--explain</code> traces</td><td>High (enables everything else)</td><td>Low-medium</td></tr>
|
||||
<tr><td>4</td><td>Context descriptions on tags/sources</td><td>Medium</td><td>Low</td></tr>
|
||||
<tr><td>5</td><td>Query expansion</td><td>Medium</td><td>Medium-high</td></tr>
|
||||
<tr><td>6</td><td>.json file ingestion (already tracked)</td><td>Medium</td><td>Low</td></tr>
|
||||
</table>
|
||||
|
||||
<h2>Current state</h2>
|
||||
<p>kb-search v2 (engine v3.2.2) runs on the RTX 4070 box with <code>BAAI/bge-base-en-v1.5</code> (768-dim). It holds ~2,310 documents (1,944 PDFs, 237 notes, 129 markdown) in ~123k chunks. Search is hybrid FTS + vector with a blended relative score. Strengths over qmd: binary ingestion (PDF/docx/HTML), tags, ingestion job queue, dedup, original export, and multi-client API access. The proposals below close the retrieval-quality gap without giving any of that up.</p>
|
||||
|
||||
<h2>Proposals</h2>
|
||||
|
||||
<div class="card">
|
||||
<h3>1. LLM reranking stage <span class="badge b-high">HIGH IMPACT</span></h3>
|
||||
<p>Add a cross-encoder reranking pass over the top-K hybrid candidates. qmd uses <code>qwen3-reranker-0.6b</code> (~640MB GGUF) — small enough to sit alongside bge on the 4070 permanently. Flow: hybrid retrieval pulls ~40 candidates → reranker scores each (query, chunk) pair → final order blends retrieval and reranker scores.</p>
|
||||
<p>qmd's position-aware blend is worth copying wholesale: rank 1–3 keep 75% retrieval weight, 4–10 get 60%, 11+ get 40%. This stops the reranker destroying exact-match hits while letting it rescue mid-ranked semantic matches.</p>
|
||||
<p class="muted">API: add <code>rerank: bool</code> (default true) to the search endpoint, with <code>--no-rerank</code> in the CLI for latency-sensitive callers.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>2. RRF fusion <span class="badge b-med">MEDIUM-HIGH</span></h3>
|
||||
<p>Replace the current score blend with Reciprocal Rank Fusion when merging FTS and vector lists: <code>score = Σ 1/(k + rank + 1)</code>, k=60. Rank-based fusion sidesteps the incomparability of BM25 scores (unbounded) and cosine similarity (0–1). qmd adds a top-rank bonus (+0.05 for #1, +0.02 for #2–3 in any list) to preserve exact matches — cheap and effective.</p>
|
||||
<p class="muted">Pure engine-side change, no API impact. Scores become comparable across queries too, which fixes the "score is relative, not absolute" caveat in the current skill docs.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>3. Bench harness + explain traces <span class="badge b-high">DO FIRST</span></h3>
|
||||
<p>We currently have no way to know if any of the above helps. Add:</p>
|
||||
<ul>
|
||||
<li><code>kb bench fixture.json</code> — run a fixture of queries with known-relevant docs, report precision@k / recall / MRR per backend (fts-only, vec-only, hybrid, hybrid+rerank). Directly mirrors <code>qmd bench</code>.</li>
|
||||
<li><code>--explain</code> on search — per-result score breakdown (FTS score, vector score, fusion contribution, rerank score).</li>
|
||||
</ul>
|
||||
<p>A fixture of 20–30 real queries against the existing corpus (lab infra questions, manual lookups, note recall) gives a regression baseline before touching ranking. <strong>This should land before #1 and #2 so their benefit is provable.</strong></p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>4. Context descriptions <span class="badge b-med">MEDIUM</span></h3>
|
||||
<p>qmd's standout idea: attach a one-line description to a collection or path (e.g. "Meeting transcripts", "Lab infrastructure runbooks") and return it with every matching result. For kb, the natural unit is the <strong>tag</strong>: <code>kb tag-describe ops "Lab operations runbooks and procedures"</code>, returned as <code>tag_contexts</code> in search results. Helps an LLM consumer (me) judge which of several similar-scoring chunks actually answers the question — descriptions cost nothing at query time.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>5. Query expansion <span class="badge b-low">LATER</span></h3>
|
||||
<p>qmd fine-tuned a 1.7B model to generate 2 query variants, searching all three and fusing via RRF. Real quality gains, but the heaviest lift: another model resident in VRAM, ~1–2s latency, and much of the benefit is available cheaper — I already do multi-query decomposition client-side per the kb skill. Park until #1–#3 have landed and the bench shows remaining headroom.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>6. JSON ingestion <span class="badge b-low">TRACKED</span></h3>
|
||||
<p>The original scope of this task: kb rejects <code>.json</code> uploads, forcing renames to <code>.txt</code>. Add <code>.json</code> (and sensibly <code>.yaml</code>/<code>.yml</code>/<code>.toml</code>) to the accepted extensions, ingesting as text. Optional nicety: pretty-print minified JSON before chunking so chunks break on structure.</p>
|
||||
</div>
|
||||
|
||||
<h2>Suggested order</h2>
|
||||
<ol>
|
||||
<li><strong>#3 bench harness</strong> — establish the baseline (a weekend-sized job).</li>
|
||||
<li><strong>#6 JSON support</strong> — small, independent, already promised.</li>
|
||||
<li><strong>#2 RRF fusion</strong> — low-risk engine change, measure against baseline.</li>
|
||||
<li><strong>#1 reranker</strong> — the big win, measured.</li>
|
||||
<li><strong>#4 tag contexts</strong> — anytime, independent.</li>
|
||||
<li><strong>#5 query expansion</strong> — only if the bench still shows a gap.</li>
|
||||
</ol>
|
||||
|
||||
<h2>References</h2>
|
||||
<ul>
|
||||
<li><a href="https://github.com/tobi/qmd">tobi/qmd</a> — architecture, fusion weights, and bench design borrowed from here</li>
|
||||
<li>qmd score fusion detail: RRF k=60, top-rank bonus +0.05/+0.02, position-aware blend 75/60/40% retrieval weight</li>
|
||||
<li>Reranker model: <code>hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF</code> (~640MB)</li>
|
||||
</ul>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,52 @@
|
||||
FROM ubuntu:24.04
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
python3.12 python3.12-venv python3.12-dev python3-pip \
|
||||
libpoppler-cpp-dev poppler-utils \
|
||||
libgl1 libglib2.0-0 \
|
||||
build-essential curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install CPU torch first, on its own, from the CPU index.
|
||||
#
|
||||
# Order matters: anything that depends on torch (sentence-transformers) will
|
||||
# otherwise resolve the default CUDA build and pull ~2.7GB of nvidia-* wheels.
|
||||
# Reinstalling torch afterwards replaces torch but leaves those wheels behind,
|
||||
# orphaned and unused — which is how the CPU image ended up larger than the
|
||||
# CUDA one. Installing CPU torch up front means nothing ever requests CUDA.
|
||||
#
|
||||
# Keeping it in its own layer also bounds the blob size: the registry drops
|
||||
# uploads that take longer than 60s, so no single layer should approach ~3GB.
|
||||
# Placing it before the source COPYs keeps this expensive layer cached when
|
||||
# only application code changes.
|
||||
RUN uv venv .venv && \
|
||||
. .venv/bin/activate && \
|
||||
UV_HTTP_TIMEOUT=600 uv pip install torch torchvision \
|
||||
--index-url https://download.pytorch.org/whl/cpu
|
||||
|
||||
COPY pyproject.toml ./
|
||||
COPY kb/ kb/
|
||||
COPY main.py ./
|
||||
COPY VERSION ./
|
||||
|
||||
# Remaining dependencies resolve against the CPU torch already present.
|
||||
RUN . .venv/bin/activate && \
|
||||
uv pip install "sentence-transformers[onnx]" && \
|
||||
uv pip install -e .
|
||||
|
||||
ENV PATH="/app/.venv/bin:$PATH"
|
||||
ENV VIRTUAL_ENV="/app/.venv"
|
||||
ENV KB_DEVICE=cpu
|
||||
ENV KB_INGEST_DEVICE=cpu
|
||||
ENV KB_DATA_DIR=/data
|
||||
|
||||
EXPOSE 8000
|
||||
VOLUME ["/data"]
|
||||
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
@@ -13,15 +13,24 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install CUDA torch on its own, before the source COPYs.
|
||||
#
|
||||
# This is the bulk of the image (~2.8GiB compressed). Splitting it from the
|
||||
# application install keeps it cached when only code changes, and keeps the
|
||||
# app layer small. The registry drops any blob upload that takes longer than
|
||||
# 60s, so this layer is deliberately the only large one.
|
||||
RUN uv venv .venv && \
|
||||
. .venv/bin/activate && \
|
||||
UV_HTTP_TIMEOUT=600 uv pip install torch torchvision \
|
||||
--index-url https://download.pytorch.org/whl/cu130
|
||||
|
||||
COPY pyproject.toml ./
|
||||
COPY kb/ kb/
|
||||
COPY main.py ./
|
||||
COPY VERSION ./
|
||||
|
||||
RUN uv venv .venv && \
|
||||
. .venv/bin/activate && \
|
||||
uv pip install -e . && \
|
||||
uv pip install --no-deps onnxruntime-gpu
|
||||
RUN . .venv/bin/activate && \
|
||||
uv pip install -e .
|
||||
|
||||
ENV PATH="/app/.venv/bin:$PATH"
|
||||
ENV VIRTUAL_ENV="/app/.venv"
|
||||
|
||||
@@ -1,68 +0,0 @@
|
||||
# Stage 1: Build — install Python deps with dev tools available
|
||||
FROM rocm/dev-ubuntu-24.04:6.4-complete AS builder
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
python3.12 python3.12-venv python3.12-dev python3-pip \
|
||||
libpoppler-cpp-dev poppler-utils \
|
||||
build-essential curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY pyproject.toml ./
|
||||
COPY kb/ kb/
|
||||
COPY main.py ./
|
||||
COPY VERSION ./
|
||||
|
||||
RUN uv venv .venv && \
|
||||
. .venv/bin/activate && \
|
||||
uv pip install -e . && \
|
||||
uv pip install --no-deps onnxruntime-rocm
|
||||
|
||||
# Stage 2: Runtime — minimal ROCm runtime libs only
|
||||
FROM ubuntu:24.04
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Add ROCm apt repository
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ca-certificates curl gnupg \
|
||||
&& mkdir -p /etc/apt/keyrings \
|
||||
&& curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key \
|
||||
| gpg --dearmor -o /etc/apt/keyrings/rocm.gpg \
|
||||
&& echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.4.1 noble main" \
|
||||
> /etc/apt/sources.list.d/rocm.list \
|
||||
&& printf 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600\n' \
|
||||
> /etc/apt/preferences.d/rocm-pin-600 \
|
||||
&& apt-get update && apt-get install -y --no-install-recommends \
|
||||
python3.12 python3.12-venv \
|
||||
libpoppler-cpp0t64 poppler-utils \
|
||||
libgl1 libglib2.0-0 \
|
||||
rocm-hip-runtime \
|
||||
rocm-hip-libraries \
|
||||
miopen-hip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy built venv and application from builder
|
||||
COPY --from=builder /app/.venv .venv
|
||||
COPY --from=builder /app/kb kb
|
||||
COPY --from=builder /app/main.py .
|
||||
COPY --from=builder /app/pyproject.toml .
|
||||
COPY --from=builder /app/VERSION .
|
||||
|
||||
ENV PATH="/app/.venv/bin:$PATH"
|
||||
ENV VIRTUAL_ENV="/app/.venv"
|
||||
ENV KB_DEVICE=auto
|
||||
ENV KB_INGEST_DEVICE=auto
|
||||
ENV KB_DATA_DIR=/data
|
||||
|
||||
EXPOSE 8000
|
||||
VOLUME ["/data"]
|
||||
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
+1
-1
@@ -1 +1 @@
|
||||
2.1.0
|
||||
3.3.0
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
services:
|
||||
kb-engine:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.cpu
|
||||
ports:
|
||||
- "${KB_PORT:-8000}:8000"
|
||||
volumes:
|
||||
- ${KB_DATA_PATH:-./data}:/data
|
||||
environment:
|
||||
- KB_MODEL=${KB_MODEL:-all-MiniLM-L6-v2}
|
||||
- KB_DEVICE=cpu
|
||||
- KB_INGEST_DEVICE=cpu
|
||||
- KB_API_KEY=${KB_API_KEY:-}
|
||||
- KB_SEARCH_THRESHOLD=${KB_SEARCH_THRESHOLD:-0.01}
|
||||
- KB_MIN_CHUNK_ALNUM=${KB_MIN_CHUNK_ALNUM:-3}
|
||||
- HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-}
|
||||
restart: unless-stopped
|
||||
|
||||
kb-mcp:
|
||||
build:
|
||||
context: ../mcp
|
||||
dockerfile: Dockerfile
|
||||
ports:
|
||||
- "${KB_MCP_PORT:-3000}:3000"
|
||||
environment:
|
||||
- KB_ENGINE_URL=http://kb-engine:8000
|
||||
- KB_API_KEY=${KB_API_KEY:-}
|
||||
- KB_MCP_API_KEY=${KB_MCP_API_KEY:-}
|
||||
# Comma-separated IPs/FQDNs allowed to connect remotely (e.g. 192.168.1.50,kb.example.com)
|
||||
- KB_MCP_ALLOWED_HOSTS=${KB_MCP_ALLOWED_HOSTS:-}
|
||||
depends_on:
|
||||
- kb-engine
|
||||
restart: unless-stopped
|
||||
@@ -21,5 +21,25 @@ services:
|
||||
- KB_INGEST_DEVICE=${KB_INGEST_DEVICE:-auto}
|
||||
- KB_API_KEY=${KB_API_KEY:-}
|
||||
- KB_SEARCH_THRESHOLD=${KB_SEARCH_THRESHOLD:-0.01}
|
||||
- KB_MIN_CHUNK_ALNUM=${KB_MIN_CHUNK_ALNUM:-3}
|
||||
- KB_RERANK_ENABLED=${KB_RERANK_ENABLED:-true}
|
||||
- KB_RERANKER_MODEL=${KB_RERANKER_MODEL:-BAAI/bge-reranker-v2-m3}
|
||||
- KB_RERANK_CANDIDATES=${KB_RERANK_CANDIDATES:-40}
|
||||
- HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-}
|
||||
restart: unless-stopped
|
||||
|
||||
kb-mcp:
|
||||
build:
|
||||
context: ../mcp
|
||||
dockerfile: Dockerfile
|
||||
ports:
|
||||
- "${KB_MCP_PORT:-3000}:3000"
|
||||
environment:
|
||||
- KB_ENGINE_URL=http://kb-engine:8000
|
||||
- KB_API_KEY=${KB_API_KEY:-}
|
||||
- KB_MCP_API_KEY=${KB_MCP_API_KEY:-}
|
||||
# Comma-separated IPs/FQDNs allowed to connect remotely (e.g. 192.168.1.50,kb.example.com)
|
||||
- KB_MCP_ALLOWED_HOSTS=${KB_MCP_ALLOWED_HOSTS:-}
|
||||
depends_on:
|
||||
- kb-engine
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
services:
|
||||
kb-engine:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.rocm
|
||||
devices:
|
||||
- "/dev/kfd"
|
||||
- "/dev/dri"
|
||||
group_add:
|
||||
- "video"
|
||||
ports:
|
||||
- "${KB_PORT:-8000}:8000"
|
||||
volumes:
|
||||
- ${KB_DATA_PATH:-./data}:/data
|
||||
environment:
|
||||
- KB_MODEL=${KB_MODEL:-all-MiniLM-L6-v2}
|
||||
- KB_DEVICE=${KB_DEVICE:-auto}
|
||||
- KB_INGEST_DEVICE=${KB_INGEST_DEVICE:-auto}
|
||||
- KB_API_KEY=${KB_API_KEY:-}
|
||||
- KB_SEARCH_THRESHOLD=${KB_SEARCH_THRESHOLD:-0.01}
|
||||
- HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-}
|
||||
restart: unless-stopped
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/bin/bash
|
||||
|
||||
docker stop engine-kb-engine-1
|
||||
KB_MODEL=BAAI/bge-base-en-v1.5 KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d --build
|
||||
@@ -20,6 +20,11 @@ class Config:
|
||||
self.ingest_device = os.environ.get("KB_INGEST_DEVICE", "auto")
|
||||
self.api_key = os.environ.get("KB_API_KEY") or None
|
||||
self.search_threshold = float(os.environ.get("KB_SEARCH_THRESHOLD", "0.01"))
|
||||
self.rerank_enabled = os.environ.get("KB_RERANK_ENABLED", "false").lower() in ("1", "true", "yes")
|
||||
self.reranker_model = os.environ.get("KB_RERANKER_MODEL", "BAAI/bge-reranker-v2-m3")
|
||||
self.rerank_candidates = int(os.environ.get("KB_RERANK_CANDIDATES", "40"))
|
||||
self.bulk_safety_percent = int(os.environ.get("KB_BULK_SAFETY_PERCENT", "70"))
|
||||
self.min_chunk_alnum = int(os.environ.get("KB_MIN_CHUNK_ALNUM", "3"))
|
||||
self.host = os.environ.get("KB_HOST", "0.0.0.0")
|
||||
self.port = int(os.environ.get("KB_PORT", "8000"))
|
||||
|
||||
|
||||
@@ -74,6 +74,7 @@ def get_connection(db_path: str) -> sqlite3.Connection:
|
||||
conn.enable_load_extension(False)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA busy_timeout=5000")
|
||||
conn.execute("PRAGMA foreign_keys=ON")
|
||||
return conn
|
||||
|
||||
@@ -185,6 +186,20 @@ def init_schema(conn: sqlite3.Connection, embedding_dim: int) -> None:
|
||||
_backfill_enriched_text(conn)
|
||||
_rebuild_fts(conn)
|
||||
|
||||
# Migrate: add updated_at to documents if missing (v3.0.0)
|
||||
if "updated_at" not in doc_cols:
|
||||
conn.execute("ALTER TABLE documents ADD COLUMN updated_at TEXT")
|
||||
|
||||
# Migrate: add job_type to jobs if missing (bulk operations)
|
||||
job_cols = {row[1] for row in conn.execute("PRAGMA table_info(jobs)").fetchall()}
|
||||
if "job_type" not in job_cols:
|
||||
conn.execute("ALTER TABLE jobs ADD COLUMN job_type TEXT DEFAULT 'ingest'")
|
||||
|
||||
# Migrate: add description to tags if missing (tag contexts, v3.3.0)
|
||||
tag_cols = {row[1] for row in conn.execute("PRAGMA table_info(tags)").fetchall()}
|
||||
if "description" not in tag_cols:
|
||||
conn.execute("ALTER TABLE tags ADD COLUMN description TEXT")
|
||||
|
||||
conn.commit()
|
||||
|
||||
|
||||
@@ -325,6 +340,92 @@ def untag_document(conn: sqlite3.Connection, document_id: int, tag_names: list[s
|
||||
conn.commit()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bulk operation helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def resolve_bulk_selection(
|
||||
conn: sqlite3.Connection,
|
||||
document_ids: list[int] | None = None,
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
from_id: int | None = None,
|
||||
to_id: int | None = None,
|
||||
) -> list[int]:
|
||||
"""Return document IDs matching the bulk selection filter.
|
||||
|
||||
Filters combine with AND logic. At least one filter must be provided.
|
||||
"""
|
||||
sql = "SELECT DISTINCT d.id FROM documents d"
|
||||
joins: list[str] = []
|
||||
where: list[str] = []
|
||||
params: list = []
|
||||
|
||||
if tags:
|
||||
for i, tag in enumerate(tags):
|
||||
joins.append(f"JOIN document_tags dt{i} ON d.id = dt{i}.document_id")
|
||||
joins.append(f"JOIN tags t{i} ON dt{i}.tag_id = t{i}.id")
|
||||
where.append(f"t{i}.name = ?")
|
||||
params.append(tag)
|
||||
|
||||
if doc_type:
|
||||
where.append("d.doc_type = ?")
|
||||
params.append(doc_type)
|
||||
|
||||
if document_ids:
|
||||
placeholders = ",".join("?" for _ in document_ids)
|
||||
where.append(f"d.id IN ({placeholders})")
|
||||
params.extend(document_ids)
|
||||
|
||||
if from_id is not None:
|
||||
where.append("d.id >= ?")
|
||||
params.append(from_id)
|
||||
|
||||
if to_id is not None:
|
||||
where.append("d.id <= ?")
|
||||
params.append(to_id)
|
||||
|
||||
if joins:
|
||||
sql += " " + " ".join(joins)
|
||||
if where:
|
||||
sql += " WHERE " + " AND ".join(where)
|
||||
|
||||
rows = conn.execute(sql, params).fetchall()
|
||||
return [row["id"] for row in rows]
|
||||
|
||||
|
||||
def create_bulk_job(
|
||||
conn: sqlite3.Connection,
|
||||
job_type: str,
|
||||
filters_json: str,
|
||||
matched: int,
|
||||
succeeded: int,
|
||||
failed: int,
|
||||
errors_json: str = "[]",
|
||||
) -> int:
|
||||
"""Create an audit log entry for a bulk operation and return its id."""
|
||||
cur = conn.execute(
|
||||
"""INSERT INTO jobs(filename, status, job_type, document_id, chunk_count, error, completed_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, current_timestamp)""",
|
||||
(
|
||||
filters_json,
|
||||
"done" if failed == 0 else "partial_failure",
|
||||
job_type,
|
||||
matched,
|
||||
succeeded,
|
||||
errors_json if failed > 0 else None,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
return cur.lastrowid
|
||||
|
||||
|
||||
def count_documents(conn: sqlite3.Connection) -> int:
|
||||
"""Return total number of documents in the database."""
|
||||
row = conn.execute("SELECT COUNT(*) AS cnt FROM documents").fetchone()
|
||||
return row["cnt"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Vec table management
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Chunking pipeline for structured data files (JSON, YAML, TOML).
|
||||
|
||||
Data files are ingested as plain text. Minified JSON is pretty-printed
|
||||
first so chunk boundaries fall on structural lines rather than mid-object.
|
||||
Malformed input never fails ingestion — it is chunked as-is.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from kb.ingest.code import _fixed_token_chunks
|
||||
|
||||
|
||||
def chunk_data(
|
||||
text: str,
|
||||
language: str | None,
|
||||
max_tokens: int = 1024,
|
||||
) -> list[dict]:
|
||||
"""Split a data file into chunks.
|
||||
|
||||
Returns a list of chunk dicts, each containing:
|
||||
text, chunk_index, metadata
|
||||
"""
|
||||
if language == "json":
|
||||
try:
|
||||
text = json.dumps(json.loads(text), indent=2, ensure_ascii=False)
|
||||
except (ValueError, TypeError):
|
||||
pass # not valid JSON — ingest the raw text unchanged
|
||||
|
||||
chunks: list[dict] = []
|
||||
for piece in _fixed_token_chunks(text, max_tokens):
|
||||
piece = piece.strip()
|
||||
if piece:
|
||||
chunks.append({
|
||||
"text": piece,
|
||||
"chunk_index": len(chunks),
|
||||
"metadata": {},
|
||||
})
|
||||
return chunks
|
||||
@@ -11,6 +11,10 @@ SUPPORTED_EXTENSIONS: dict[str, tuple[str, str | None]] = {
|
||||
".py": ("code", "python"),
|
||||
".sh": ("code", "bash"),
|
||||
".go": ("code", "go"),
|
||||
".json": ("data", "json"),
|
||||
".yaml": ("data", "yaml"),
|
||||
".yml": ("data", "yaml"),
|
||||
".toml": ("data", "toml"),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -25,6 +25,7 @@ from docling.document_converter import DocumentConverter, PdfFormatOption # noq
|
||||
from docling_core.transforms.chunker.hierarchical_chunker import ( # noqa: E402
|
||||
HierarchicalChunker,
|
||||
)
|
||||
from kb.ingest.quality import has_minimum_content
|
||||
|
||||
|
||||
def _fixed_size_chunks(text: str, max_chars: int = 2000) -> list[str]:
|
||||
@@ -40,6 +41,7 @@ def _fixed_size_chunks(text: str, max_chars: int = 2000) -> list[str]:
|
||||
def chunk_document(
|
||||
file_path: Path,
|
||||
ingest_device: str = "cpu",
|
||||
min_chunk_alnum: int = 3,
|
||||
) -> list[dict]:
|
||||
"""Convert and chunk a PDF/DOCX/HTML document using Docling.
|
||||
|
||||
@@ -71,7 +73,7 @@ def chunk_document(
|
||||
chunks: list[dict] = []
|
||||
for idx, chunk in enumerate(raw_chunks):
|
||||
text = chunk.text.strip() if hasattr(chunk, "text") else str(chunk).strip()
|
||||
if not text:
|
||||
if not text or not has_minimum_content(text, min_chunk_alnum):
|
||||
continue
|
||||
|
||||
metadata: dict = {}
|
||||
@@ -98,6 +100,8 @@ def chunk_document(
|
||||
if not full_text and hasattr(doc, "text"):
|
||||
full_text = doc.text
|
||||
for idx, piece in enumerate(_fixed_size_chunks(full_text)):
|
||||
if not has_minimum_content(piece, min_chunk_alnum):
|
||||
continue
|
||||
chunks.append({
|
||||
"text": piece,
|
||||
"chunk_index": idx,
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
"""Small, conservative ingestion-quality checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def has_minimum_content(text: str, min_alnum: int = 3) -> bool:
|
||||
"""Return whether text contains enough letters/numbers to be searchable.
|
||||
|
||||
Counting alphanumeric characters avoids indexing OCR fragments consisting
|
||||
only of punctuation or one-character labels while retaining short IDs.
|
||||
"""
|
||||
if min_alnum <= 0:
|
||||
return True
|
||||
return sum(character.isalnum() for character in text) >= min_alnum
|
||||
@@ -0,0 +1 @@
|
||||
"""Explicit maintenance commands for kb-engine data."""
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Repair notes whose title was generated from the old ``note`` fallback.
|
||||
|
||||
Preview changes by default::
|
||||
|
||||
python -m kb.maintenance.backfill_note_titles
|
||||
|
||||
Apply them, including refreshed FTS text and embeddings::
|
||||
|
||||
python -m kb.maintenance.backfill_note_titles --apply
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import struct
|
||||
|
||||
from kb import database, embeddings
|
||||
from kb.config import cfg
|
||||
from kb.ingest.note import auto_title
|
||||
|
||||
|
||||
_SYNTHETIC_NOTE = re.compile(
|
||||
r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}_note\.note$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def find_repairs(conn) -> list[dict]:
|
||||
"""Return unambiguous synthetic note titles and their derived replacements."""
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT d.id, d.title, d.original_filename, c.text, c.metadata
|
||||
FROM documents d
|
||||
JOIN chunks c ON c.document_id = d.id AND c.chunk_index = 0
|
||||
WHERE d.doc_type = 'note'
|
||||
ORDER BY d.id
|
||||
"""
|
||||
).fetchall()
|
||||
repairs = []
|
||||
for row in rows:
|
||||
if not _SYNTHETIC_NOTE.fullmatch(row["title"] or ""):
|
||||
continue
|
||||
title = auto_title(row["text"] or "")
|
||||
if not title:
|
||||
continue
|
||||
repairs.append({
|
||||
"document_id": row["id"],
|
||||
"old_title": row["title"],
|
||||
"new_title": title,
|
||||
"original_filename": row["original_filename"],
|
||||
})
|
||||
return repairs
|
||||
|
||||
|
||||
def apply_repairs(conn, repairs: list[dict]) -> None:
|
||||
"""Update titles, enriched text, FTS, and vectors for selected notes."""
|
||||
for repair in repairs:
|
||||
doc_id = repair["document_id"]
|
||||
title = repair["new_title"]
|
||||
chunks = conn.execute(
|
||||
"SELECT id, text, metadata FROM chunks WHERE document_id = ? ORDER BY chunk_index",
|
||||
(doc_id,),
|
||||
).fetchall()
|
||||
enriched = []
|
||||
for chunk in chunks:
|
||||
metadata = json.loads(chunk["metadata"] or "{}")
|
||||
enriched.append(database.build_enriched_text(title, chunk["text"], metadata))
|
||||
vectors = embeddings.embed_texts(enriched)
|
||||
|
||||
conn.execute(
|
||||
"UPDATE documents SET title = ?, updated_at = current_timestamp WHERE id = ?",
|
||||
(title, doc_id),
|
||||
)
|
||||
for chunk, text, vector in zip(chunks, enriched, vectors):
|
||||
conn.execute(
|
||||
"UPDATE chunks SET enriched_text = ? WHERE id = ?", (text, chunk["id"])
|
||||
)
|
||||
conn.execute("DELETE FROM chunks_vec WHERE chunk_id = ?", (chunk["id"],))
|
||||
blob = struct.pack(f"{len(vector)}f", *vector)
|
||||
conn.execute(
|
||||
"INSERT INTO chunks_vec(embedding, chunk_id) VALUES (?, ?)",
|
||||
(blob, chunk["id"]),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--apply", action="store_true", help="apply repairs (the default is preview only)"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
conn = database.get_connection(cfg.db_path)
|
||||
try:
|
||||
repairs = find_repairs(conn)
|
||||
for repair in repairs:
|
||||
print(
|
||||
f'{repair["document_id"]}: {repair["old_title"]!r} -> '
|
||||
f'{repair["new_title"]!r}'
|
||||
)
|
||||
if not args.apply:
|
||||
print(f"Previewed {len(repairs)} repair(s); rerun with --apply to update them.")
|
||||
return
|
||||
embeddings.load_model(cfg.model, cfg.device)
|
||||
apply_repairs(conn, repairs)
|
||||
print(f"Repaired {len(repairs)} note title(s).")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Cross-encoder reranker management.
|
||||
|
||||
Mirrors the embeddings module: one module-level model, loaded eagerly at
|
||||
startup when reranking is enabled. Reranking is strictly optional — search
|
||||
degrades gracefully to plain hybrid retrieval when the model is absent.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("kb.reranker")
|
||||
|
||||
_reranker: Optional[object] = None
|
||||
_model_name: Optional[str] = None
|
||||
|
||||
|
||||
def load_reranker(model_name: str, device: str = "cpu") -> None:
|
||||
"""Load a cross-encoder reranking model.
|
||||
|
||||
Args:
|
||||
model_name: HuggingFace model name or local path. Must have a
|
||||
sequence-classification head (e.g. BAAI/bge-reranker-v2-m3).
|
||||
device: Target device — "cpu", "cuda", or "auto".
|
||||
"""
|
||||
global _reranker, _model_name
|
||||
|
||||
from sentence_transformers import CrossEncoder
|
||||
|
||||
from kb.embeddings import _resolve_device
|
||||
|
||||
resolved_device = _resolve_device(device)
|
||||
logger.info("Loading reranker '%s' on device '%s'", model_name, resolved_device)
|
||||
|
||||
_reranker = CrossEncoder(model_name, device=resolved_device)
|
||||
_model_name = model_name
|
||||
|
||||
logger.info("Reranker loaded: %s", model_name)
|
||||
|
||||
|
||||
def is_available() -> bool:
|
||||
"""Return True if a reranker model is loaded and usable."""
|
||||
return _reranker is not None
|
||||
|
||||
|
||||
def rerank_scores(query: str, texts: list[str]) -> list[float]:
|
||||
"""Score (query, text) pairs with the cross-encoder.
|
||||
|
||||
Returns:
|
||||
One relevance score per text, sigmoid-normalised to 0-1.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If no reranker has been loaded.
|
||||
"""
|
||||
if _reranker is None:
|
||||
raise RuntimeError("Reranker not loaded. Call load_reranker() first.")
|
||||
|
||||
import numpy as np
|
||||
|
||||
scores = _reranker.predict([(query, t) for t in texts], convert_to_numpy=True)
|
||||
# CrossEncoder heads may emit raw logits; squash to 0-1 so scores blend
|
||||
# predictably with normalised retrieval scores. Sigmoid is monotonic, so
|
||||
# ordering is unaffected for models that already output probabilities.
|
||||
return (1.0 / (1.0 + np.exp(-np.asarray(scores, dtype="float64")))).tolist()
|
||||
@@ -1 +1 @@
|
||||
from kb.routes import health, search, jobs, documents, tags, status, reindex, auth
|
||||
from kb.routes import health, search, jobs, documents, tags, status, reindex, auth, notes
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
"""Bulk operation endpoints — delete, tag, and set-tags on multiple documents."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import HTTPException
|
||||
from pydantic import BaseModel, model_validator
|
||||
|
||||
from main import app
|
||||
from kb.config import cfg
|
||||
from kb.database import (
|
||||
get_connection,
|
||||
resolve_bulk_selection,
|
||||
count_documents,
|
||||
create_bulk_job,
|
||||
tag_document,
|
||||
untag_document,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("kb.routes.bulk")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Request models
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class BulkSelectionRequest(BaseModel):
|
||||
document_ids: Optional[list[int]] = None
|
||||
tags: Optional[list[str]] = None
|
||||
doc_type: Optional[str] = None
|
||||
from_id: Optional[int] = None
|
||||
to_id: Optional[int] = None
|
||||
force: bool = False
|
||||
|
||||
@model_validator(mode="after")
|
||||
def require_at_least_one_filter(self):
|
||||
if not any([self.document_ids, self.tags, self.doc_type,
|
||||
self.from_id is not None, self.to_id is not None]):
|
||||
raise ValueError("At least one selection filter is required")
|
||||
return self
|
||||
|
||||
|
||||
class BulkDeleteRequest(BulkSelectionRequest):
|
||||
pass
|
||||
|
||||
|
||||
class BulkTagsRequest(BulkSelectionRequest):
|
||||
add: Optional[list[str]] = None
|
||||
remove: Optional[list[str]] = None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def require_add_or_remove(self):
|
||||
if not self.add and not self.remove:
|
||||
raise ValueError("At least one of 'add' or 'remove' is required")
|
||||
return self
|
||||
|
||||
|
||||
class BulkSetTagsRequest(BulkSelectionRequest):
|
||||
new_tags: list[str]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _check_safety_threshold(matched: int, total: int, force: bool) -> None:
|
||||
"""Raise 409 if the operation would affect too many documents."""
|
||||
threshold = cfg.bulk_safety_percent
|
||||
if threshold <= 0 or force or total == 0:
|
||||
return
|
||||
percent = (matched / total) * 100
|
||||
if percent > threshold:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"error": "safety_threshold_exceeded",
|
||||
"message": (
|
||||
f"Operation would affect {matched} of {total} documents "
|
||||
f"({percent:.1f}%). Exceeds safety threshold of {threshold}%. "
|
||||
f"Use force: true to proceed."
|
||||
),
|
||||
"matched": matched,
|
||||
"total": total,
|
||||
"percent": round(percent, 1),
|
||||
"threshold": threshold,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _filters_dict(req: BulkSelectionRequest) -> str:
|
||||
"""Build a JSON string of the selection filter for audit logging."""
|
||||
d = {}
|
||||
if req.document_ids:
|
||||
d["document_ids"] = req.document_ids
|
||||
if req.tags:
|
||||
d["tags"] = req.tags
|
||||
if req.doc_type:
|
||||
d["doc_type"] = req.doc_type
|
||||
if req.from_id is not None:
|
||||
d["from_id"] = req.from_id
|
||||
if req.to_id is not None:
|
||||
d["to_id"] = req.to_id
|
||||
return json.dumps(d)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.post("/api/v1/bulk/delete")
|
||||
async def bulk_delete(req: BulkDeleteRequest):
|
||||
conn = get_connection(cfg.db_path)
|
||||
try:
|
||||
doc_ids = resolve_bulk_selection(
|
||||
conn, req.document_ids, req.tags, req.doc_type, req.from_id, req.to_id,
|
||||
)
|
||||
total = count_documents(conn)
|
||||
_check_safety_threshold(len(doc_ids), total, req.force)
|
||||
|
||||
succeeded = 0
|
||||
failed = 0
|
||||
errors = []
|
||||
stored_files: list[str] = []
|
||||
|
||||
for doc_id in doc_ids:
|
||||
try:
|
||||
doc = conn.execute(
|
||||
"SELECT id, stored_path FROM documents WHERE id = ?", (doc_id,)
|
||||
).fetchone()
|
||||
if not doc:
|
||||
failed += 1
|
||||
errors.append({"document_id": doc_id, "error": "not found"})
|
||||
continue
|
||||
|
||||
if doc["stored_path"]:
|
||||
stored_files.append(doc["stored_path"])
|
||||
|
||||
# Delete embeddings
|
||||
chunk_ids = conn.execute(
|
||||
"SELECT id FROM chunks WHERE document_id = ?", (doc_id,)
|
||||
).fetchall()
|
||||
for row in chunk_ids:
|
||||
conn.execute("DELETE FROM chunks_vec WHERE chunk_id = ?", (row["id"],))
|
||||
|
||||
# Delete document (cascades to chunks, document_tags)
|
||||
conn.execute("DELETE FROM documents WHERE id = ?", (doc_id,))
|
||||
succeeded += 1
|
||||
except Exception as exc:
|
||||
failed += 1
|
||||
errors.append({"document_id": doc_id, "error": str(exc)})
|
||||
|
||||
conn.commit()
|
||||
|
||||
# Best-effort file cleanup after commit
|
||||
for path in stored_files:
|
||||
try:
|
||||
f = Path(path)
|
||||
if f.exists():
|
||||
f.unlink()
|
||||
except OSError as exc:
|
||||
logger.warning("Failed to delete stored file %s: %s", path, exc)
|
||||
|
||||
errors_json = json.dumps(errors) if errors else "[]"
|
||||
job_id = create_bulk_job(
|
||||
conn, "bulk_delete", _filters_dict(req),
|
||||
len(doc_ids), succeeded, failed, errors_json,
|
||||
)
|
||||
|
||||
return {
|
||||
"job_id": job_id,
|
||||
"status": "done" if failed == 0 else "partial_failure",
|
||||
"matched": len(doc_ids),
|
||||
"succeeded": succeeded,
|
||||
"failed": failed,
|
||||
"errors": errors,
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
@app.post("/api/v1/bulk/tags")
|
||||
async def bulk_tags(req: BulkTagsRequest):
|
||||
conn = get_connection(cfg.db_path)
|
||||
try:
|
||||
doc_ids = resolve_bulk_selection(
|
||||
conn, req.document_ids, req.tags, req.doc_type, req.from_id, req.to_id,
|
||||
)
|
||||
total = count_documents(conn)
|
||||
_check_safety_threshold(len(doc_ids), total, req.force)
|
||||
|
||||
succeeded = 0
|
||||
failed = 0
|
||||
errors = []
|
||||
|
||||
for doc_id in doc_ids:
|
||||
try:
|
||||
if req.add:
|
||||
tag_document(conn, doc_id, req.add)
|
||||
if req.remove:
|
||||
untag_document(conn, doc_id, req.remove)
|
||||
conn.execute(
|
||||
"UPDATE documents SET updated_at = current_timestamp WHERE id = ?",
|
||||
(doc_id,),
|
||||
)
|
||||
succeeded += 1
|
||||
except Exception as exc:
|
||||
failed += 1
|
||||
errors.append({"document_id": doc_id, "error": str(exc)})
|
||||
|
||||
conn.commit()
|
||||
|
||||
errors_json = json.dumps(errors) if errors else "[]"
|
||||
job_id = create_bulk_job(
|
||||
conn, "bulk_tags", _filters_dict(req),
|
||||
len(doc_ids), succeeded, failed, errors_json,
|
||||
)
|
||||
|
||||
return {
|
||||
"job_id": job_id,
|
||||
"status": "done" if failed == 0 else "partial_failure",
|
||||
"matched": len(doc_ids),
|
||||
"succeeded": succeeded,
|
||||
"failed": failed,
|
||||
"errors": errors,
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
@app.post("/api/v1/bulk/set-tags")
|
||||
async def bulk_set_tags(req: BulkSetTagsRequest):
|
||||
conn = get_connection(cfg.db_path)
|
||||
try:
|
||||
doc_ids = resolve_bulk_selection(
|
||||
conn, req.document_ids, req.tags, req.doc_type, req.from_id, req.to_id,
|
||||
)
|
||||
total = count_documents(conn)
|
||||
_check_safety_threshold(len(doc_ids), total, req.force)
|
||||
|
||||
succeeded = 0
|
||||
failed = 0
|
||||
errors = []
|
||||
|
||||
for doc_id in doc_ids:
|
||||
try:
|
||||
# Remove all existing tags
|
||||
conn.execute(
|
||||
"DELETE FROM document_tags WHERE document_id = ?", (doc_id,)
|
||||
)
|
||||
# Apply new tag set
|
||||
if req.new_tags:
|
||||
tag_document(conn, doc_id, req.new_tags)
|
||||
conn.execute(
|
||||
"UPDATE documents SET updated_at = current_timestamp WHERE id = ?",
|
||||
(doc_id,),
|
||||
)
|
||||
succeeded += 1
|
||||
except Exception as exc:
|
||||
failed += 1
|
||||
errors.append({"document_id": doc_id, "error": str(exc)})
|
||||
|
||||
conn.commit()
|
||||
|
||||
errors_json = json.dumps(errors) if errors else "[]"
|
||||
job_id = create_bulk_job(
|
||||
conn, "bulk_set_tags", _filters_dict(req),
|
||||
len(doc_ids), succeeded, failed, errors_json,
|
||||
)
|
||||
|
||||
return {
|
||||
"job_id": job_id,
|
||||
"status": "done" if failed == 0 else "partial_failure",
|
||||
"matched": len(doc_ids),
|
||||
"succeeded": succeeded,
|
||||
"failed": failed,
|
||||
"errors": errors,
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
@@ -7,11 +7,13 @@ from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import HTTPException, Query
|
||||
from pydantic import BaseModel, Field
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from main import app
|
||||
from kb.config import cfg
|
||||
from kb.database import get_connection
|
||||
from kb.search import hybrid_search
|
||||
|
||||
logger = logging.getLogger("kb.routes.documents")
|
||||
|
||||
@@ -20,13 +22,15 @@ logger = logging.getLogger("kb.routes.documents")
|
||||
async def list_documents(
|
||||
type: Optional[str] = Query(None),
|
||||
tags: Optional[str] = Query(None),
|
||||
title: Optional[str] = Query(None),
|
||||
filename: Optional[str] = Query(None),
|
||||
):
|
||||
conn = get_connection(cfg.db_path)
|
||||
try:
|
||||
sql = """
|
||||
SELECT d.id, d.title, d.doc_type,
|
||||
SELECT d.id, d.title, d.original_filename, d.source_path, d.doc_type,
|
||||
(SELECT COUNT(*) FROM chunks c WHERE c.document_id = d.id) AS chunk_count,
|
||||
d.created_at
|
||||
d.created_at, d.updated_at
|
||||
FROM documents d
|
||||
"""
|
||||
joins: list[str] = []
|
||||
@@ -37,6 +41,14 @@ async def list_documents(
|
||||
where.append("d.doc_type = ?")
|
||||
params.append(type)
|
||||
|
||||
if title:
|
||||
where.append("d.title LIKE ? COLLATE NOCASE")
|
||||
params.append(f"%{title}%")
|
||||
|
||||
if filename:
|
||||
where.append("d.original_filename LIKE ? COLLATE NOCASE")
|
||||
params.append(f"%{filename}%")
|
||||
|
||||
if tags:
|
||||
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
|
||||
for i, tag in enumerate(tag_list):
|
||||
@@ -50,7 +62,7 @@ async def list_documents(
|
||||
if where:
|
||||
sql += " WHERE " + " AND ".join(where)
|
||||
|
||||
sql += " ORDER BY d.created_at DESC"
|
||||
sql += " ORDER BY COALESCE(d.updated_at, d.created_at) DESC"
|
||||
|
||||
rows = conn.execute(sql, params).fetchall()
|
||||
|
||||
@@ -70,10 +82,13 @@ async def list_documents(
|
||||
results.append({
|
||||
"id": row["id"],
|
||||
"title": row["title"],
|
||||
"original_filename": row["original_filename"],
|
||||
"source_path": row["source_path"],
|
||||
"doc_type": row["doc_type"],
|
||||
"tags": [t["name"] for t in tag_rows],
|
||||
"chunk_count": row["chunk_count"],
|
||||
"created_at": row["created_at"],
|
||||
"updated_at": row["updated_at"],
|
||||
})
|
||||
|
||||
return results
|
||||
@@ -81,8 +96,49 @@ async def list_documents(
|
||||
conn.close()
|
||||
|
||||
|
||||
class DocumentFindRequest(BaseModel):
|
||||
query: str
|
||||
top: int = Field(default=10, ge=1, le=100)
|
||||
tags: Optional[list[str]] = None
|
||||
doc_type: Optional[str] = None
|
||||
|
||||
|
||||
@app.post("/api/v1/documents/find")
|
||||
async def find_documents(req: DocumentFindRequest):
|
||||
"""Return document-level results aggregated from hybrid chunk search."""
|
||||
conn = get_connection(cfg.db_path)
|
||||
try:
|
||||
search_result = hybrid_search(
|
||||
conn,
|
||||
req.query,
|
||||
cfg,
|
||||
top=max(req.top * 10, 50),
|
||||
tags=req.tags,
|
||||
doc_type=req.doc_type,
|
||||
)
|
||||
documents: dict[int, dict] = {}
|
||||
for result in search_result["results"]:
|
||||
doc_id = result["document_id"]
|
||||
if doc_id not in documents:
|
||||
documents[doc_id] = {
|
||||
"document_id": doc_id,
|
||||
"title": result["title"],
|
||||
"doc_type": result["doc_type"],
|
||||
"source_path": result["source_path"],
|
||||
"original_filename": result["original_filename"],
|
||||
"tags": result["tags"],
|
||||
"score": result["score"],
|
||||
"hit_count": 0,
|
||||
"top_chunk": result["text"],
|
||||
}
|
||||
documents[doc_id]["hit_count"] += 1
|
||||
return list(documents.values())[: req.top]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
@app.get("/api/v1/documents/{doc_id}")
|
||||
async def get_document(doc_id: int):
|
||||
async def get_document(doc_id: int, include_chunks: bool = Query(True)):
|
||||
conn = get_connection(cfg.db_path)
|
||||
try:
|
||||
doc = conn.execute(
|
||||
@@ -91,6 +147,11 @@ async def get_document(doc_id: int):
|
||||
if not doc:
|
||||
raise HTTPException(status_code=404, detail="Document not found.")
|
||||
|
||||
chunk_count = conn.execute(
|
||||
"SELECT COUNT(*) AS n FROM chunks WHERE document_id = ?", (doc_id,)
|
||||
).fetchone()["n"]
|
||||
chunks = []
|
||||
if include_chunks:
|
||||
chunks = conn.execute(
|
||||
"SELECT * FROM chunks WHERE document_id = ? ORDER BY chunk_index",
|
||||
(doc_id,),
|
||||
@@ -113,6 +174,7 @@ async def get_document(doc_id: int):
|
||||
**dict(doc),
|
||||
"has_file": has_file,
|
||||
"tags": [t["name"] for t in tag_rows],
|
||||
"chunk_count": chunk_count,
|
||||
"chunks": [dict(c) for c in chunks],
|
||||
}
|
||||
finally:
|
||||
|
||||
@@ -10,6 +10,7 @@ from fastapi.responses import JSONResponse
|
||||
from main import app
|
||||
from kb.config import cfg
|
||||
from kb.database import get_connection, create_job, get_job, list_jobs, get_document_by_hash
|
||||
from kb.ingest.note import auto_title
|
||||
from kb.staging import stage_file, stage_note
|
||||
|
||||
|
||||
@@ -32,6 +33,7 @@ async def submit_job(
|
||||
content_hash = hashlib.sha256(content).hexdigest()
|
||||
filename = file.filename
|
||||
else:
|
||||
title = title or auto_title(note) or "note"
|
||||
content = note.encode("utf-8")
|
||||
content_hash = hashlib.sha256(content).hexdigest()
|
||||
filename = None
|
||||
@@ -48,7 +50,7 @@ async def submit_job(
|
||||
if file:
|
||||
staging_path = stage_file(cfg.staging_dir, file.filename, content)
|
||||
else:
|
||||
staging_path = stage_note(cfg.staging_dir, title or "note", note)
|
||||
staging_path = stage_note(cfg.staging_dir, title, note)
|
||||
filename = staging_path.name
|
||||
|
||||
tags_list = [t.strip() for t in tags.split(",") if t.strip()] if tags else []
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Note mutation endpoint — update existing notes in place."""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
|
||||
from fastapi import HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from main import app
|
||||
from kb.config import cfg
|
||||
from kb.database import (
|
||||
get_connection,
|
||||
build_enriched_text,
|
||||
insert_chunk,
|
||||
insert_embedding,
|
||||
)
|
||||
from kb.embeddings import embed_texts
|
||||
from kb.ingest.note import chunk_note
|
||||
|
||||
logger = logging.getLogger("kb.routes.notes")
|
||||
|
||||
|
||||
class NoteUpdateRequest(BaseModel):
|
||||
text: str
|
||||
|
||||
|
||||
@app.patch("/api/v1/notes/{doc_id}")
|
||||
async def update_note(doc_id: int, req: NoteUpdateRequest):
|
||||
conn = get_connection(cfg.db_path)
|
||||
try:
|
||||
doc = conn.execute(
|
||||
"SELECT id, title, doc_type FROM documents WHERE id = ?", (doc_id,)
|
||||
).fetchone()
|
||||
if not doc:
|
||||
raise HTTPException(status_code=404, detail="Document not found.")
|
||||
if doc["doc_type"] != "note":
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail="Only notes can be updated via this endpoint.",
|
||||
)
|
||||
|
||||
title = doc["title"]
|
||||
|
||||
# Delete existing chunks and their embeddings
|
||||
chunk_ids = conn.execute(
|
||||
"SELECT id FROM chunks WHERE document_id = ?", (doc_id,)
|
||||
).fetchall()
|
||||
for row in chunk_ids:
|
||||
conn.execute("DELETE FROM chunks_vec WHERE chunk_id = ?", (row["id"],))
|
||||
conn.execute("DELETE FROM chunks WHERE document_id = ?", (doc_id,))
|
||||
|
||||
# Run note chunking pipeline on new text
|
||||
chunks = chunk_note(req.text)
|
||||
chunk_texts = [c["text"] for c in chunks]
|
||||
chunk_metas = [
|
||||
{k: v for k, v in c.items() if k != "text"} or None for c in chunks
|
||||
]
|
||||
|
||||
enriched_texts = [
|
||||
build_enriched_text(title, ct, cm)
|
||||
for ct, cm in zip(chunk_texts, chunk_metas)
|
||||
]
|
||||
|
||||
# Embed — if this fails, the transaction rolls back
|
||||
vectors = embed_texts(enriched_texts)
|
||||
|
||||
for idx, (chunk_text, enriched, vector) in enumerate(
|
||||
zip(chunk_texts, enriched_texts, vectors)
|
||||
):
|
||||
chunk_id = insert_chunk(
|
||||
conn,
|
||||
document_id=doc_id,
|
||||
chunk_index=idx,
|
||||
text=chunk_text,
|
||||
enriched_text=enriched,
|
||||
metadata=chunk_metas[idx],
|
||||
)
|
||||
insert_embedding(conn, chunk_id, vector)
|
||||
|
||||
# Update content_hash and updated_at
|
||||
content_hash = hashlib.sha256(req.text.encode("utf-8")).hexdigest()
|
||||
conn.execute(
|
||||
"UPDATE documents SET content_hash = ?, updated_at = current_timestamp WHERE id = ?",
|
||||
(content_hash, doc_id),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
# Return updated document
|
||||
updated_doc = conn.execute(
|
||||
"SELECT * FROM documents WHERE id = ?", (doc_id,)
|
||||
).fetchone()
|
||||
|
||||
new_chunks = conn.execute(
|
||||
"SELECT * FROM chunks WHERE document_id = ? ORDER BY chunk_index",
|
||||
(doc_id,),
|
||||
).fetchall()
|
||||
|
||||
tag_rows = conn.execute(
|
||||
"""
|
||||
SELECT t.name FROM tags t
|
||||
JOIN document_tags dt ON t.id = dt.tag_id
|
||||
WHERE dt.document_id = ?
|
||||
ORDER BY t.name
|
||||
""",
|
||||
(doc_id,),
|
||||
).fetchall()
|
||||
|
||||
return {
|
||||
**dict(updated_doc),
|
||||
"tags": [t["name"] for t in tag_rows],
|
||||
"chunks": [dict(c) for c in new_chunks],
|
||||
}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception:
|
||||
conn.rollback()
|
||||
logger.exception("Failed to update note %d", doc_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to update note.")
|
||||
finally:
|
||||
conn.close()
|
||||
@@ -19,6 +19,8 @@ class SearchRequest(BaseModel):
|
||||
fts_only: bool = False
|
||||
vec_only: bool = False
|
||||
threshold: Optional[float] = None
|
||||
explain: bool = False
|
||||
rerank: Optional[bool] = None
|
||||
|
||||
|
||||
@app.post("/api/v1/search")
|
||||
@@ -35,6 +37,8 @@ async def search(req: SearchRequest):
|
||||
fts_only=req.fts_only,
|
||||
vec_only=req.vec_only,
|
||||
threshold=req.threshold,
|
||||
explain=req.explain,
|
||||
rerank=req.rerank,
|
||||
)
|
||||
return result
|
||||
except Exception as exc:
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import os
|
||||
|
||||
from main import app, __version__
|
||||
from kb import reranker
|
||||
from kb.config import cfg
|
||||
from kb.database import get_connection
|
||||
from kb.embeddings import get_model_dim
|
||||
@@ -62,6 +63,12 @@ async def status():
|
||||
"queued": queue_stats.get("queued", 0),
|
||||
"processing": queue_stats.get("processing", 0),
|
||||
},
|
||||
"rerank": {
|
||||
"enabled": cfg.rerank_enabled,
|
||||
"model": cfg.reranker_model,
|
||||
"loaded": reranker.is_available(),
|
||||
"candidates": cfg.rerank_candidates,
|
||||
},
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
@@ -16,14 +16,45 @@ async def list_tags():
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT t.name, COUNT(dt.document_id) AS count
|
||||
SELECT t.name, t.description, COUNT(dt.document_id) AS count
|
||||
FROM tags t
|
||||
LEFT JOIN document_tags dt ON t.id = dt.tag_id
|
||||
GROUP BY t.id, t.name
|
||||
ORDER BY t.name
|
||||
"""
|
||||
).fetchall()
|
||||
return [{"name": row["name"], "count": row["count"]} for row in rows]
|
||||
return [
|
||||
{"name": row["name"], "count": row["count"], "description": row["description"]}
|
||||
for row in rows
|
||||
]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
class TagDescriptionRequest(BaseModel):
|
||||
description: Optional[str] = None
|
||||
|
||||
|
||||
@app.put("/api/v1/tags/{name}/description")
|
||||
async def set_tag_description(name: str, req: TagDescriptionRequest):
|
||||
"""Set or clear a one-line context description on a tag.
|
||||
|
||||
Descriptions are returned as ``tag_contexts`` with every search result on
|
||||
a document carrying the tag, helping consumers judge relevance.
|
||||
"""
|
||||
conn = get_connection(cfg.db_path)
|
||||
try:
|
||||
# name matching is case-insensitive (tags.name is COLLATE NOCASE)
|
||||
tag = conn.execute("SELECT id FROM tags WHERE name = ?", (name,)).fetchone()
|
||||
if not tag:
|
||||
raise HTTPException(status_code=404, detail=f"Tag '{name}' not found.")
|
||||
|
||||
description = (req.description or "").strip() or None
|
||||
conn.execute(
|
||||
"UPDATE tags SET description = ? WHERE id = ?", (description, tag["id"])
|
||||
)
|
||||
conn.commit()
|
||||
return {"name": name, "description": description}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
@@ -48,6 +79,13 @@ async def update_document_tags(doc_id: int, req: TagUpdateRequest):
|
||||
if req.remove:
|
||||
untag_document(conn, doc_id, req.remove)
|
||||
|
||||
if req.add or req.remove:
|
||||
conn.execute(
|
||||
"UPDATE documents SET updated_at = current_timestamp WHERE id = ?",
|
||||
(doc_id,),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
tag_rows = conn.execute(
|
||||
"""
|
||||
SELECT t.name FROM tags t
|
||||
|
||||
+168
-17
@@ -18,6 +18,8 @@ def hybrid_search(
|
||||
fts_only: bool = False,
|
||||
vec_only: bool = False,
|
||||
threshold: float | None = None,
|
||||
explain: bool = False,
|
||||
rerank: bool | None = None,
|
||||
) -> dict:
|
||||
"""Run hybrid search and return merged, enriched results.
|
||||
|
||||
@@ -31,11 +33,29 @@ def hybrid_search(
|
||||
fts_only: Only use FTS5 (skip vector search).
|
||||
vec_only: Only use vector search (skip FTS5).
|
||||
threshold: Optional minimum score; results below are dropped.
|
||||
explain: Attach a per-result score breakdown (arm scores, ranks,
|
||||
RRF contributions, rerank blend) under an ``explain`` key.
|
||||
rerank: Cross-encoder rerank the top candidates. None uses the
|
||||
engine default (cfg.rerank_enabled); True still requires a
|
||||
loaded reranker and degrades silently to plain retrieval
|
||||
otherwise. Never applied to fts_only / vec_only searches.
|
||||
|
||||
Returns:
|
||||
Dict with keys: query, results, total_matches, returned.
|
||||
Dict with keys: query, results, total_matches, returned, reranked.
|
||||
"""
|
||||
from kb import reranker
|
||||
|
||||
want_rerank = cfg.rerank_enabled if rerank is None else rerank
|
||||
do_rerank = (
|
||||
want_rerank
|
||||
and not fts_only
|
||||
and not vec_only
|
||||
and reranker.is_available()
|
||||
)
|
||||
|
||||
candidate_count = top * 3
|
||||
if do_rerank:
|
||||
candidate_count = max(candidate_count, cfg.rerank_candidates)
|
||||
|
||||
fts_results: dict[int, float] = {}
|
||||
vec_results: dict[int, float] = {}
|
||||
@@ -49,10 +69,12 @@ def hybrid_search(
|
||||
# --- merge ---------------------------------------------------------------
|
||||
if fts_only:
|
||||
merged = sorted(fts_results.items(), key=lambda x: x[1], reverse=True)
|
||||
details = _single_arm_details("fts", fts_results)
|
||||
elif vec_only:
|
||||
merged = sorted(vec_results.items(), key=lambda x: x[1], reverse=True)
|
||||
details = _single_arm_details("vec", vec_results)
|
||||
else:
|
||||
merged = _rrf_merge(fts_results, vec_results)
|
||||
merged, details = _rrf_merge(fts_results, vec_results)
|
||||
|
||||
# Apply threshold filter — use config default if not specified per-query
|
||||
effective_threshold = threshold if threshold is not None else cfg.search_threshold
|
||||
@@ -60,16 +82,30 @@ def hybrid_search(
|
||||
merged = [(cid, score) for cid, score in merged if score >= effective_threshold]
|
||||
|
||||
total_matches = len(merged)
|
||||
|
||||
# --- rerank --------------------------------------------------------------
|
||||
# Blended scores are 0-1 normalised, a different scale from RRF scores;
|
||||
# the threshold above was applied to RRF scores and is NOT re-applied.
|
||||
reranked = False
|
||||
if do_rerank and merged:
|
||||
candidates = merged[: cfg.rerank_candidates]
|
||||
rr_scores = reranker.rerank_scores(
|
||||
query, _fetch_chunk_texts(conn, [cid for cid, _ in candidates])
|
||||
)
|
||||
merged = _blend_rerank(candidates, rr_scores, details)
|
||||
reranked = True
|
||||
|
||||
merged = merged[:top]
|
||||
|
||||
# --- enrich --------------------------------------------------------------
|
||||
results = _enrich(conn, merged)
|
||||
results = _enrich(conn, merged, details if explain else None)
|
||||
|
||||
return {
|
||||
"query": query,
|
||||
"results": results,
|
||||
"total_matches": total_matches,
|
||||
"returned": len(results),
|
||||
"reranked": reranked,
|
||||
}
|
||||
|
||||
|
||||
@@ -232,31 +268,135 @@ def _rrf_merge(
|
||||
fts_results: dict[int, float],
|
||||
vec_results: dict[int, float],
|
||||
k: int = 60,
|
||||
) -> list[tuple[int, float]]:
|
||||
) -> tuple[list[tuple[int, float]], dict[int, dict]]:
|
||||
"""Reciprocal Rank Fusion over two scored result sets.
|
||||
|
||||
Each set is ranked independently (highest score first, rank starts at 1).
|
||||
RRF score for a document = sum of 1/(k + rank) across sets it appears in.
|
||||
RRF score for a document = sum of 1/(k + rank) across sets it appears in,
|
||||
plus a top-rank bonus per arm: +0.05 for rank 1, +0.02 for ranks 2-3.
|
||||
The bonus preserves exact matches — a chunk at the top of either arm is
|
||||
nearly impossible to displace via mid-rank RRF accumulation alone.
|
||||
|
||||
Score scale: base RRF maxes at 2/(k+1) ≈ 0.033 (k=60); with bonuses the
|
||||
ceiling is ≈ 0.133. Bonuses only raise scores, so the threshold filter
|
||||
(default 0.01) can never drop a result that base RRF would have kept.
|
||||
|
||||
Returns:
|
||||
Sorted list of (chunk_id, rrf_score), highest first.
|
||||
(scores, details) — scores is a sorted list of (chunk_id, rrf_score),
|
||||
highest first; details maps chunk_id to a per-arm score breakdown
|
||||
suitable for the ``explain`` response field.
|
||||
"""
|
||||
fts_ranked = _rank_by_score(fts_results)
|
||||
vec_ranked = _rank_by_score(vec_results)
|
||||
|
||||
all_ids = set(fts_ranked) | set(vec_ranked)
|
||||
scores: list[tuple[int, float]] = []
|
||||
details: dict[int, dict] = {}
|
||||
|
||||
for chunk_id in all_ids:
|
||||
rrf = 0.0
|
||||
if chunk_id in fts_ranked:
|
||||
rrf += 1.0 / (k + fts_ranked[chunk_id])
|
||||
if chunk_id in vec_ranked:
|
||||
rrf += 1.0 / (k + vec_ranked[chunk_id])
|
||||
fts_rank = fts_ranked.get(chunk_id)
|
||||
vec_rank = vec_ranked.get(chunk_id)
|
||||
rrf_fts = 1.0 / (k + fts_rank) if fts_rank is not None else None
|
||||
rrf_vec = 1.0 / (k + vec_rank) if vec_rank is not None else None
|
||||
bonus = _top_rank_bonus(fts_rank) + _top_rank_bonus(vec_rank)
|
||||
rrf = (rrf_fts or 0.0) + (rrf_vec or 0.0) + bonus
|
||||
|
||||
details[chunk_id] = {
|
||||
"fts_score": _round6(fts_results.get(chunk_id)),
|
||||
"fts_rank": fts_rank,
|
||||
"vec_score": _round6(vec_results.get(chunk_id)),
|
||||
"vec_rank": vec_rank,
|
||||
"rrf_fts": _round6(rrf_fts),
|
||||
"rrf_vec": _round6(rrf_vec),
|
||||
"bonus": bonus,
|
||||
"final_score": _round6(rrf),
|
||||
}
|
||||
scores.append((chunk_id, rrf))
|
||||
|
||||
scores.sort(key=lambda x: x[1], reverse=True)
|
||||
return scores
|
||||
return scores, details
|
||||
|
||||
|
||||
def _top_rank_bonus(rank: int | None) -> float:
|
||||
"""Bonus for appearing at the top of one arm's ranking."""
|
||||
if rank == 1:
|
||||
return 0.05
|
||||
if rank in (2, 3):
|
||||
return 0.02
|
||||
return 0.0
|
||||
|
||||
|
||||
def _single_arm_details(arm: str, results: dict[int, float]) -> dict[int, dict]:
|
||||
"""Explain details for fts_only / vec_only searches (raw arm scores)."""
|
||||
ranked = _rank_by_score(results)
|
||||
return {
|
||||
chunk_id: {
|
||||
f"{arm}_score": _round6(score),
|
||||
f"{arm}_rank": ranked[chunk_id],
|
||||
"final_score": _round6(score),
|
||||
}
|
||||
for chunk_id, score in results.items()
|
||||
}
|
||||
|
||||
|
||||
def _round6(value: float | None) -> float | None:
|
||||
return round(value, 6) if value is not None else None
|
||||
|
||||
|
||||
def _fetch_chunk_texts(conn: sqlite3.Connection, chunk_ids: list[int]) -> list[str]:
|
||||
"""Fetch chunk texts in the same order as *chunk_ids*."""
|
||||
placeholders = ",".join("?" * len(chunk_ids))
|
||||
rows = conn.execute(
|
||||
f"SELECT id, text FROM chunks WHERE id IN ({placeholders})", chunk_ids
|
||||
).fetchall()
|
||||
by_id = {row[0]: row[1] for row in rows}
|
||||
return [by_id.get(cid, "") for cid in chunk_ids]
|
||||
|
||||
|
||||
def _blend_rerank(
|
||||
candidates: list[tuple[int, float]],
|
||||
rr_scores: list[float],
|
||||
details: dict[int, dict],
|
||||
) -> list[tuple[int, float]]:
|
||||
"""Blend retrieval and cross-encoder scores, position-aware.
|
||||
|
||||
Retrieval scores are min-max normalised within the candidate set; rerank
|
||||
scores are already 0-1. The retrieval weight depends on pre-rerank rank —
|
||||
75% for ranks 1-3, 60% for 4-10, 40% for 11+ — so the reranker can rescue
|
||||
mid-ranked semantic matches without destroying top exact-match hits.
|
||||
|
||||
*candidates* must be in retrieval order; *rr_scores* aligned with it.
|
||||
Mutates *details* with the blend breakdown. Returns (chunk_id, blended)
|
||||
sorted highest first.
|
||||
"""
|
||||
retrieval = [score for _, score in candidates]
|
||||
lo, hi = min(retrieval), max(retrieval)
|
||||
span = hi - lo
|
||||
|
||||
blended: list[tuple[int, float]] = []
|
||||
for i, ((chunk_id, score), rr) in enumerate(zip(candidates, rr_scores)):
|
||||
rank = i + 1
|
||||
norm = (score - lo) / span if span > 0 else 1.0
|
||||
if rank <= 3:
|
||||
weight = 0.75
|
||||
elif rank <= 10:
|
||||
weight = 0.60
|
||||
else:
|
||||
weight = 0.40
|
||||
final = weight * norm + (1.0 - weight) * rr
|
||||
|
||||
if chunk_id in details:
|
||||
details[chunk_id].update({
|
||||
"pre_rerank_rank": rank,
|
||||
"retrieval_norm": _round6(norm),
|
||||
"rerank_score": _round6(rr),
|
||||
"blend_weight": weight,
|
||||
"final_score": _round6(final),
|
||||
})
|
||||
blended.append((chunk_id, final))
|
||||
|
||||
blended.sort(key=lambda x: x[1], reverse=True)
|
||||
return blended
|
||||
|
||||
|
||||
def _rank_by_score(results: dict[int, float]) -> dict[int, int]:
|
||||
@@ -268,8 +408,13 @@ def _rank_by_score(results: dict[int, float]) -> dict[int, int]:
|
||||
def _enrich(
|
||||
conn: sqlite3.Connection,
|
||||
merged: list[tuple[int, float]],
|
||||
details: dict[int, dict] | None = None,
|
||||
) -> list[dict]:
|
||||
"""Fetch chunk text, document metadata, chunk metadata, and tags."""
|
||||
"""Fetch chunk text, document metadata, chunk metadata, and tags.
|
||||
|
||||
When *details* is given, each result gains an ``explain`` key with its
|
||||
score breakdown.
|
||||
"""
|
||||
results: list[dict] = []
|
||||
|
||||
for chunk_id, score in merged:
|
||||
@@ -277,7 +422,7 @@ def _enrich(
|
||||
"""
|
||||
SELECT c.id, c.text, c.chunk_index, c.metadata AS chunk_meta,
|
||||
d.id AS doc_id, d.title, d.doc_type, d.source_path,
|
||||
d.created_at
|
||||
d.created_at, d.original_filename
|
||||
FROM chunks c
|
||||
JOIN documents d ON c.document_id = d.id
|
||||
WHERE c.id = ?
|
||||
@@ -292,7 +437,7 @@ def _enrich(
|
||||
|
||||
tag_rows = conn.execute(
|
||||
"""
|
||||
SELECT t.name FROM tags t
|
||||
SELECT t.name, t.description FROM tags t
|
||||
JOIN document_tags dt ON t.id = dt.tag_id
|
||||
WHERE dt.document_id = ?
|
||||
ORDER BY t.name
|
||||
@@ -300,8 +445,9 @@ def _enrich(
|
||||
(row[4],), # doc_id
|
||||
).fetchall()
|
||||
|
||||
results.append({
|
||||
result = {
|
||||
"chunk_id": row[0],
|
||||
"document_id": row[4],
|
||||
"score": round(score, 6),
|
||||
"text": row[1],
|
||||
"chunk_index": row[2],
|
||||
@@ -310,7 +456,12 @@ def _enrich(
|
||||
"doc_type": row[6],
|
||||
"source_path": row[7],
|
||||
"created_at": row[8],
|
||||
"original_filename": row[9],
|
||||
"tags": [t[0] for t in tag_rows],
|
||||
})
|
||||
"tag_contexts": {t[0]: t[1] for t in tag_rows if t[1]},
|
||||
}
|
||||
if details is not None and row[0] in details:
|
||||
result["explain"] = details[row[0]]
|
||||
results.append(result)
|
||||
|
||||
return results
|
||||
|
||||
@@ -16,7 +16,8 @@ def stage_file(staging_dir: Path, filename: str, content: bytes) -> Path:
|
||||
The path to the newly created staged file.
|
||||
"""
|
||||
staging_dir.mkdir(parents=True, exist_ok=True)
|
||||
dest = staging_dir / f"{uuid.uuid4()}_{filename}"
|
||||
safe_filename = filename.replace("/", "_").replace("\\", "_")
|
||||
dest = staging_dir / f"{uuid.uuid4()}_{safe_filename}"
|
||||
dest.write_bytes(content)
|
||||
logger.debug("Staged file: %s (%d bytes)", dest, len(content))
|
||||
return dest
|
||||
@@ -31,7 +32,8 @@ def stage_note(staging_dir: Path, title: str, text: str) -> Path:
|
||||
The path to the newly created staged note file.
|
||||
"""
|
||||
staging_dir.mkdir(parents=True, exist_ok=True)
|
||||
dest = staging_dir / f"{uuid.uuid4()}_{title}.note"
|
||||
safe_title = title.replace("/", "_").replace("\\", "_")
|
||||
dest = staging_dir / f"{uuid.uuid4()}_{safe_title}.note"
|
||||
dest.write_text(text, encoding="utf-8")
|
||||
logger.debug("Staged note: %s (%d chars)", dest, len(text))
|
||||
return dest
|
||||
|
||||
+9
-1
@@ -113,7 +113,9 @@ def _process_job(job_row) -> tuple[str, int | None, int]:
|
||||
chunks = chunk_note(text)
|
||||
elif doc_type == "pdf":
|
||||
from kb.ingest.docling_pipeline import chunk_document
|
||||
chunks = chunk_document(staged_path, cfg.ingest_device)
|
||||
chunks = chunk_document(
|
||||
staged_path, cfg.ingest_device, cfg.min_chunk_alnum
|
||||
)
|
||||
elif doc_type == "markdown":
|
||||
text = staged_path.read_text(encoding="utf-8")
|
||||
from kb.ingest.markdown import chunk_markdown
|
||||
@@ -124,6 +126,12 @@ def _process_job(job_row) -> tuple[str, int | None, int]:
|
||||
_, language = detector.detect_type(Path(filename))
|
||||
from kb.ingest.code import chunk_code
|
||||
chunks = chunk_code(text, language)
|
||||
elif doc_type == "data":
|
||||
text = staged_path.read_text(encoding="utf-8")
|
||||
if not language:
|
||||
_, language = detector.detect_type(Path(filename))
|
||||
from kb.ingest.data import chunk_data
|
||||
chunks = chunk_data(text, language)
|
||||
else:
|
||||
raise ValueError(f"Unsupported doc_type: {doc_type}")
|
||||
|
||||
|
||||
+13
-1
@@ -40,6 +40,18 @@ async def lifespan(app: FastAPI):
|
||||
init_schema(conn, model_dim)
|
||||
conn.close()
|
||||
|
||||
# Optional reranker — search degrades gracefully if this fails
|
||||
if cfg.rerank_enabled:
|
||||
from kb.reranker import load_reranker
|
||||
try:
|
||||
load_reranker(cfg.reranker_model, cfg.device)
|
||||
except Exception:
|
||||
log.warning(
|
||||
"Failed to load reranker '%s' — searches will not be reranked",
|
||||
cfg.reranker_model,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
# Start background ingestion worker
|
||||
worker_task = asyncio.create_task(ingestion_worker())
|
||||
|
||||
@@ -62,7 +74,7 @@ async def lifespan(app: FastAPI):
|
||||
app = FastAPI(title="kb-engine", version=__version__, lifespan=lifespan)
|
||||
|
||||
# Import routes after app is created
|
||||
from kb.routes import health, search, jobs, documents, tags, status, reindex, auth # noqa: E402, F401
|
||||
from kb.routes import health, search, jobs, documents, tags, status, reindex, auth, notes, bulk # noqa: E402, F401
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Shared test setup — make the engine root importable (for ``import kb``)."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Focused tests for document metadata lookup and aggregation."""
|
||||
|
||||
import pytest
|
||||
|
||||
from kb import database
|
||||
from kb.routes import documents
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def document_db(tmp_path, monkeypatch):
|
||||
db_path = tmp_path / "kb.db"
|
||||
conn = database.get_connection(db_path)
|
||||
database.init_schema(conn, 3)
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO documents(title, source_path, content_hash, doc_type, original_filename)
|
||||
VALUES ('Vehicle Guide', '/data/staging/random.pdf', 'hash', 'pdf', 'M38T_manual.pdf')
|
||||
"""
|
||||
)
|
||||
doc_id = conn.execute("SELECT id FROM documents").fetchone()["id"]
|
||||
conn.execute(
|
||||
"INSERT INTO chunks(document_id, chunk_index, text, enriched_text) VALUES (?, 0, 'body', 'body')",
|
||||
(doc_id,),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
monkeypatch.setattr(documents.cfg, "data_dir", tmp_path)
|
||||
return doc_id
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_list_filters_title_and_original_filename(document_db):
|
||||
by_title = await documents.list_documents(
|
||||
type=None, tags=None, title="vehicle", filename=None
|
||||
)
|
||||
by_filename = await documents.list_documents(
|
||||
type=None, tags=None, title=None, filename="m38t"
|
||||
)
|
||||
|
||||
assert [item["id"] for item in by_title] == [document_db]
|
||||
assert by_filename[0]["original_filename"] == "M38T_manual.pdf"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_info_can_omit_chunks_without_losing_count(document_db):
|
||||
result = await documents.get_document(document_db, include_chunks=False)
|
||||
|
||||
assert result["chunk_count"] == 1
|
||||
assert result["chunks"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_find_aggregates_chunk_hits_by_document(monkeypatch):
|
||||
class Connection:
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
monkeypatch.setattr(documents, "get_connection", lambda _path: Connection())
|
||||
monkeypatch.setattr(
|
||||
documents,
|
||||
"hybrid_search",
|
||||
lambda *_args, **_kwargs: {
|
||||
"results": [
|
||||
{
|
||||
"document_id": 7,
|
||||
"title": "Guide",
|
||||
"doc_type": "pdf",
|
||||
"source_path": "/staging/file",
|
||||
"original_filename": "guide.pdf",
|
||||
"tags": [],
|
||||
"score": 0.8,
|
||||
"text": "best hit",
|
||||
},
|
||||
{
|
||||
"document_id": 7,
|
||||
"title": "Guide",
|
||||
"doc_type": "pdf",
|
||||
"source_path": "/staging/file",
|
||||
"original_filename": "guide.pdf",
|
||||
"tags": [],
|
||||
"score": 0.7,
|
||||
"text": "second hit",
|
||||
},
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
result = await documents.find_documents(
|
||||
documents.DocumentFindRequest(query="guide")
|
||||
)
|
||||
|
||||
assert result == [{
|
||||
"document_id": 7,
|
||||
"title": "Guide",
|
||||
"doc_type": "pdf",
|
||||
"source_path": "/staging/file",
|
||||
"original_filename": "guide.pdf",
|
||||
"tags": [],
|
||||
"score": 0.8,
|
||||
"hit_count": 2,
|
||||
"top_chunk": "best hit",
|
||||
}]
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Tests for structured-data (.json/.yaml/.toml) ingestion."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from kb.ingest.data import chunk_data
|
||||
from kb.ingest.detector import detect_type, is_supported
|
||||
|
||||
|
||||
def test_detector_accepts_data_extensions():
|
||||
assert detect_type(Path("config.json")) == ("data", "json")
|
||||
assert detect_type(Path("stack.yaml")) == ("data", "yaml")
|
||||
assert detect_type(Path("stack.yml")) == ("data", "yaml")
|
||||
assert detect_type(Path("pyproject.toml")) == ("data", "toml")
|
||||
for name in ("a.json", "b.yaml", "c.yml", "d.toml"):
|
||||
assert is_supported(Path(name))
|
||||
|
||||
|
||||
def test_minified_json_is_pretty_printed():
|
||||
minified = json.dumps({"hosts": [{"name": "web1", "ip": "10.0.0.1"}]})
|
||||
assert "\n" not in minified
|
||||
chunks = chunk_data(minified, "json")
|
||||
assert len(chunks) == 1
|
||||
text = chunks[0]["text"]
|
||||
assert "\n" in text, "expected pretty-printed multi-line JSON"
|
||||
assert '"name": "web1"' in text
|
||||
|
||||
|
||||
def test_large_minified_json_multi_chunks():
|
||||
big = json.dumps([{"id": i, "payload": "x" * 100} for i in range(200)])
|
||||
chunks = chunk_data(big, "json", max_tokens=256)
|
||||
assert len(chunks) > 1, "large JSON must split into multiple chunks"
|
||||
# Pretty-printing means chunks break on lines, not mid-token blobs.
|
||||
for c in chunks:
|
||||
assert c["text"].strip()
|
||||
assert [c["chunk_index"] for c in chunks] == list(range(len(chunks)))
|
||||
|
||||
|
||||
def test_malformed_json_ingests_raw():
|
||||
broken = '{"unterminated": [1, 2'
|
||||
chunks = chunk_data(broken, "json")
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0]["text"] == broken
|
||||
|
||||
|
||||
def test_yaml_passes_through_unchanged():
|
||||
yaml_text = "services:\n web:\n image: nginx\n"
|
||||
chunks = chunk_data(yaml_text, "yaml")
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0]["text"] == yaml_text.strip()
|
||||
|
||||
|
||||
def test_toml_passes_through():
|
||||
toml_text = '[tool.example]\nname = "kb"\n'
|
||||
chunks = chunk_data(toml_text, "toml")
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0]["text"] == toml_text.strip()
|
||||
|
||||
|
||||
def test_empty_file_yields_no_chunks():
|
||||
assert chunk_data("", "json") == []
|
||||
assert chunk_data(" \n ", "yaml") == []
|
||||
@@ -0,0 +1,19 @@
|
||||
"""Tests for filtering noisy OCR fragments."""
|
||||
|
||||
from kb.ingest.quality import has_minimum_content
|
||||
|
||||
|
||||
def test_short_ocr_fragments_are_rejected():
|
||||
assert not has_minimum_content('"')
|
||||
assert not has_minimum_content("B")
|
||||
assert not has_minimum_content("12")
|
||||
|
||||
|
||||
def test_short_identifiers_and_real_text_are_retained():
|
||||
assert has_minimum_content("BID")
|
||||
assert has_minimum_content("00:15")
|
||||
assert has_minimum_content("Useful text")
|
||||
|
||||
|
||||
def test_filter_can_be_disabled():
|
||||
assert has_minimum_content("B", min_alnum=0)
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Tests for automatic and backfilled note titles."""
|
||||
|
||||
import sqlite3
|
||||
|
||||
from kb.ingest.note import auto_title
|
||||
from kb.maintenance import backfill_note_titles
|
||||
from kb.maintenance.backfill_note_titles import find_repairs
|
||||
|
||||
|
||||
def test_auto_title_strips_markdown_and_limits_length():
|
||||
assert auto_title("## Useful heading\nBody") == "Useful heading"
|
||||
assert auto_title("x" * 100) == "x" * 80
|
||||
|
||||
|
||||
def test_find_repairs_only_selects_synthetic_note_titles():
|
||||
conn = sqlite3.connect(":memory:")
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript(
|
||||
"""
|
||||
CREATE TABLE documents (
|
||||
id INTEGER PRIMARY KEY, title TEXT, original_filename TEXT, doc_type TEXT
|
||||
);
|
||||
CREATE TABLE chunks (
|
||||
id INTEGER PRIMARY KEY, document_id INTEGER, chunk_index INTEGER,
|
||||
text TEXT, metadata TEXT
|
||||
);
|
||||
INSERT INTO documents VALUES
|
||||
(1, '7dec828a-1234-4567-89ab-123456789abc_note.note',
|
||||
'7dec828a-1234-4567-89ab-123456789abc_note.note', 'note'),
|
||||
(2, 'A deliberate title', 'named.note', 'note');
|
||||
INSERT INTO chunks VALUES
|
||||
(1, 1, 0, '# Derived title\nBody', '{}'),
|
||||
(2, 2, 0, 'Must not replace', '{}');
|
||||
"""
|
||||
)
|
||||
|
||||
assert find_repairs(conn) == [{
|
||||
"document_id": 1,
|
||||
"old_title": "7dec828a-1234-4567-89ab-123456789abc_note.note",
|
||||
"new_title": "Derived title",
|
||||
"original_filename": "7dec828a-1234-4567-89ab-123456789abc_note.note",
|
||||
}]
|
||||
|
||||
|
||||
def test_apply_repairs_refreshes_title_search_text_and_vector(monkeypatch):
|
||||
conn = sqlite3.connect(":memory:")
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript(
|
||||
"""
|
||||
CREATE TABLE documents (id INTEGER PRIMARY KEY, title TEXT, updated_at TEXT);
|
||||
CREATE TABLE chunks (
|
||||
id INTEGER PRIMARY KEY, document_id INTEGER, chunk_index INTEGER,
|
||||
text TEXT, metadata TEXT, enriched_text TEXT
|
||||
);
|
||||
CREATE TABLE chunks_vec (embedding BLOB, chunk_id INTEGER);
|
||||
INSERT INTO documents VALUES (1, 'old', NULL);
|
||||
INSERT INTO chunks VALUES (3, 1, 0, 'New title\nBody', '{}', 'old text');
|
||||
INSERT INTO chunks_vec VALUES (X'00', 3);
|
||||
"""
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
backfill_note_titles.embeddings,
|
||||
"embed_texts",
|
||||
lambda texts: [[1.0, 2.0, 3.0] for _ in texts],
|
||||
)
|
||||
|
||||
backfill_note_titles.apply_repairs(conn, [{
|
||||
"document_id": 1, "new_title": "New title"
|
||||
}])
|
||||
|
||||
assert conn.execute("SELECT title FROM documents").fetchone()[0] == "New title"
|
||||
assert conn.execute("SELECT enriched_text FROM chunks").fetchone()[0] == (
|
||||
"New title\n\nNew title\nBody"
|
||||
)
|
||||
assert len(conn.execute("SELECT embedding FROM chunks_vec").fetchone()[0]) == 12
|
||||
@@ -0,0 +1,273 @@
|
||||
"""Tests for hybrid search: explain traces, document_id, and RRF merging."""
|
||||
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from kb.database import (
|
||||
get_connection,
|
||||
init_schema,
|
||||
insert_chunk,
|
||||
insert_document,
|
||||
insert_embedding,
|
||||
tag_document,
|
||||
)
|
||||
from kb.search import _blend_rerank, _rank_by_score, _rrf_merge, hybrid_search
|
||||
|
||||
DIM = 4
|
||||
|
||||
# Chunk vectors are axis-aligned so we can steer vector ranking exactly:
|
||||
# a query of [1,0,0,0] has distance 0 to chunk A, sqrt(2) to chunk B.
|
||||
VEC_A = [1.0, 0.0, 0.0, 0.0]
|
||||
VEC_B = [0.0, 1.0, 0.0, 0.0]
|
||||
QUERY_VEC = [1.0, 0.0, 0.0, 0.0]
|
||||
|
||||
|
||||
class _Cfg:
|
||||
search_threshold = 0.0
|
||||
rerank_enabled = False
|
||||
reranker_model = "test-reranker"
|
||||
rerank_candidates = 40
|
||||
|
||||
|
||||
class _Db:
|
||||
def __init__(self, conn, ids):
|
||||
self.conn = conn
|
||||
self.ids = ids
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db(tmp_path, monkeypatch):
|
||||
"""Real schema (FTS5 + sqlite-vec) with two docs and stubbed embeddings."""
|
||||
fake = types.ModuleType("kb.embeddings")
|
||||
fake.embed_texts = lambda texts: [QUERY_VEC for _ in texts]
|
||||
monkeypatch.setitem(sys.modules, "kb.embeddings", fake)
|
||||
|
||||
conn = get_connection(str(tmp_path / "kb.db"))
|
||||
init_schema(conn, embedding_dim=DIM)
|
||||
|
||||
doc_a = insert_document(conn, "Alpha doc", "/src/a.md", "hash-a", "markdown")
|
||||
doc_b = insert_document(conn, "Bravo doc", "/src/b.md", "hash-b", "markdown")
|
||||
chunk_a = insert_chunk(conn, doc_a, 0, "alpha network switch configuration")
|
||||
chunk_b = insert_chunk(conn, doc_b, 0, "bravo unrelated cooking recipe")
|
||||
insert_embedding(conn, chunk_a, VEC_A)
|
||||
insert_embedding(conn, chunk_b, VEC_B)
|
||||
tag_document(conn, doc_a, ["ops"])
|
||||
|
||||
ids = {"doc_a": doc_a, "doc_b": doc_b, "chunk_a": chunk_a, "chunk_b": chunk_b}
|
||||
yield _Db(conn, ids)
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_results_include_document_id(db):
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg())
|
||||
assert result["results"], "expected at least one hit"
|
||||
top_hit = result["results"][0]
|
||||
assert top_hit["document_id"] == db.ids["doc_a"]
|
||||
assert top_hit["chunk_id"] == db.ids["chunk_a"]
|
||||
|
||||
|
||||
def test_explain_absent_by_default(db):
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg())
|
||||
assert all("explain" not in r for r in result["results"])
|
||||
|
||||
|
||||
def test_explain_hybrid_breakdown(db):
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), explain=True)
|
||||
top_hit = result["results"][0]
|
||||
exp = top_hit["explain"]
|
||||
|
||||
# Chunk A is rank 1 in both arms: FTS matches "alpha"/"switch", vector
|
||||
# distance is 0 (similarity 1.0).
|
||||
assert exp["fts_rank"] == 1
|
||||
assert exp["vec_rank"] == 1
|
||||
assert exp["fts_score"] > 0
|
||||
assert exp["vec_score"] == pytest.approx(1.0)
|
||||
assert exp["rrf_fts"] == pytest.approx(1.0 / 61, abs=1e-6)
|
||||
assert exp["rrf_vec"] == pytest.approx(1.0 / 61, abs=1e-6)
|
||||
assert exp["final_score"] == pytest.approx(
|
||||
exp["rrf_fts"] + exp["rrf_vec"] + exp["bonus"], abs=1e-5
|
||||
)
|
||||
assert exp["final_score"] == pytest.approx(top_hit["score"], abs=1e-5)
|
||||
|
||||
|
||||
def test_explain_single_arm_when_vec_misses(db):
|
||||
"""A chunk found only by vector search has null FTS fields."""
|
||||
result = hybrid_search(db.conn, "zzz-no-fts-match", _Cfg(), explain=True)
|
||||
for r in result["results"]:
|
||||
exp = r["explain"]
|
||||
assert exp["fts_score"] is None
|
||||
assert exp["fts_rank"] is None
|
||||
assert exp["rrf_fts"] is None
|
||||
assert exp["vec_rank"] is not None
|
||||
|
||||
|
||||
def test_explain_fts_only_shape(db):
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), fts_only=True, explain=True)
|
||||
top_hit = result["results"][0]
|
||||
exp = top_hit["explain"]
|
||||
assert exp["fts_rank"] == 1
|
||||
assert exp["final_score"] == pytest.approx(exp["fts_score"])
|
||||
assert "vec_score" not in exp
|
||||
|
||||
|
||||
def test_explain_vec_only_shape(db):
|
||||
result = hybrid_search(db.conn, "anything", _Cfg(), vec_only=True, explain=True)
|
||||
top_hit = result["results"][0]
|
||||
exp = top_hit["explain"]
|
||||
assert exp["vec_rank"] == 1
|
||||
assert exp["final_score"] == pytest.approx(exp["vec_score"])
|
||||
assert "fts_score" not in exp
|
||||
|
||||
|
||||
def test_rrf_merge_arithmetic():
|
||||
fts = {1: 10.0, 2: 5.0}
|
||||
vec = {2: 0.9, 3: 0.8}
|
||||
scores, details = _rrf_merge(fts, vec)
|
||||
by_id = dict(scores)
|
||||
|
||||
# fts rank 1 → 1/61 + 0.05 bonus
|
||||
assert by_id[1] == pytest.approx(1 / 61 + 0.05)
|
||||
# fts rank 2 (+0.02) and vec rank 1 (+0.05) — bonuses stack across arms
|
||||
assert by_id[2] == pytest.approx(1 / 62 + 1 / 61 + 0.07)
|
||||
# vec rank 2 → 1/62 + 0.02
|
||||
assert by_id[3] == pytest.approx(1 / 62 + 0.02)
|
||||
# Chunk 2 appears in both arms, so it must win.
|
||||
assert scores[0][0] == 2
|
||||
|
||||
assert details[2]["fts_rank"] == 2
|
||||
assert details[2]["vec_rank"] == 1
|
||||
assert details[2]["bonus"] == pytest.approx(0.07)
|
||||
assert details[1]["vec_rank"] is None
|
||||
assert details[3]["rrf_fts"] is None
|
||||
|
||||
|
||||
def test_top_rank_bonus_tiers():
|
||||
from kb.search import _top_rank_bonus
|
||||
|
||||
assert _top_rank_bonus(1) == 0.05
|
||||
assert _top_rank_bonus(2) == 0.02
|
||||
assert _top_rank_bonus(3) == 0.02
|
||||
assert _top_rank_bonus(4) == 0.0
|
||||
assert _top_rank_bonus(None) == 0.0
|
||||
|
||||
|
||||
def test_bonus_preserves_top_of_arm():
|
||||
"""A chunk at rank 1 of one arm beats a chunk at mid-rank in both arms."""
|
||||
fts = {10: 100.0, 11: 90.0, 12: 80.0, 13: 70.0, 14: 60.0}
|
||||
vec = {20: 0.9, 11: 0.8, 12: 0.7, 13: 0.6, 14: 0.5}
|
||||
scores, _ = _rrf_merge(fts, vec)
|
||||
order = [cid for cid, _ in scores]
|
||||
# Chunk 10 (fts #1, absent from vec): 1/61 + 0.05 ≈ 0.0664.
|
||||
# Chunk 13 (rank 4 in fts, rank 4 in vec): 2/64 ≈ 0.031, no bonus.
|
||||
assert order.index(10) < order.index(13)
|
||||
|
||||
|
||||
def test_rank_by_score():
|
||||
assert _rank_by_score({7: 0.5, 8: 0.9, 9: 0.1}) == {8: 1, 7: 2, 9: 3}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reranking
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _enable_fake_reranker(monkeypatch, score_fn):
|
||||
from kb import reranker
|
||||
|
||||
monkeypatch.setattr(reranker, "is_available", lambda: True)
|
||||
monkeypatch.setattr(reranker, "rerank_scores", score_fn)
|
||||
|
||||
|
||||
def test_rerank_flags_response_and_explain(db, monkeypatch):
|
||||
_enable_fake_reranker(monkeypatch, lambda q, texts: [0.9] * len(texts))
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), explain=True, rerank=True)
|
||||
|
||||
assert result["reranked"] is True
|
||||
top_hit = result["results"][0]
|
||||
exp = top_hit["explain"]
|
||||
assert exp["rerank_score"] == pytest.approx(0.9)
|
||||
assert exp["pre_rerank_rank"] == 1
|
||||
assert exp["blend_weight"] == 0.75
|
||||
assert exp["final_score"] == pytest.approx(top_hit["score"], abs=1e-5)
|
||||
# rank 1: 75% retrieval (norm=1.0 for the top candidate) + 25% rerank
|
||||
assert top_hit["score"] == pytest.approx(0.75 * 1.0 + 0.25 * 0.9, abs=1e-5)
|
||||
|
||||
|
||||
def test_rerank_can_reorder(db, monkeypatch):
|
||||
"""A strong rerank score rescues a lower-retrieval-ranked chunk."""
|
||||
|
||||
def favour_chunk_b(query, texts):
|
||||
return [1.0 if "cooking" in t else 0.0 for t in texts]
|
||||
|
||||
_enable_fake_reranker(monkeypatch, favour_chunk_b)
|
||||
# Neutral-ish query: both chunks retrieved, chunk A ranked first.
|
||||
result = hybrid_search(db.conn, "alpha cooking", _Cfg(), rerank=True)
|
||||
assert result["reranked"] is True
|
||||
assert len(result["results"]) == 2
|
||||
|
||||
|
||||
def test_rerank_false_bypasses(db, monkeypatch):
|
||||
called = []
|
||||
_enable_fake_reranker(monkeypatch, lambda q, t: called.append(1) or [0.5] * len(t))
|
||||
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), rerank=False)
|
||||
assert result["reranked"] is False
|
||||
assert not called
|
||||
|
||||
|
||||
def test_rerank_unavailable_degrades_gracefully(db):
|
||||
# No reranker loaded: rerank=True must not error.
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), rerank=True, explain=True)
|
||||
assert result["reranked"] is False
|
||||
assert "rerank_score" not in result["results"][0]["explain"]
|
||||
|
||||
|
||||
def test_rerank_skipped_for_single_arm(db, monkeypatch):
|
||||
called = []
|
||||
_enable_fake_reranker(monkeypatch, lambda q, t: called.append(1) or [0.5] * len(t))
|
||||
|
||||
result = hybrid_search(db.conn, "alpha switch", _Cfg(), fts_only=True, rerank=True)
|
||||
assert result["reranked"] is False
|
||||
assert not called
|
||||
|
||||
|
||||
def test_rerank_default_follows_cfg(db, monkeypatch):
|
||||
_enable_fake_reranker(monkeypatch, lambda q, texts: [0.5] * len(texts))
|
||||
|
||||
class _RerankCfg(_Cfg):
|
||||
rerank_enabled = True
|
||||
|
||||
result = hybrid_search(db.conn, "alpha switch", _RerankCfg())
|
||||
assert result["reranked"] is True
|
||||
|
||||
|
||||
def test_blend_rerank_weights_by_position():
|
||||
# 12 candidates, retrieval scores 12 down to 1 → norms 1.0 down to 0.0.
|
||||
candidates = [(cid, float(12 - i)) for i, cid in enumerate(range(100, 112))]
|
||||
details = {cid: {} for cid, _ in candidates}
|
||||
rr = [1.0] * len(candidates)
|
||||
|
||||
blended = dict(_blend_rerank(candidates, rr, details))
|
||||
|
||||
assert details[100]["blend_weight"] == 0.75 # rank 1
|
||||
assert details[102]["blend_weight"] == 0.75 # rank 3
|
||||
assert details[103]["blend_weight"] == 0.60 # rank 4
|
||||
assert details[109]["blend_weight"] == 0.60 # rank 10
|
||||
assert details[110]["blend_weight"] == 0.40 # rank 11
|
||||
|
||||
# rank 1: norm 1.0 → 0.75*1.0 + 0.25*1.0 = 1.0
|
||||
assert blended[100] == pytest.approx(1.0)
|
||||
# rank 4: norm 8/11 → 0.6*(8/11) + 0.4*1.0
|
||||
assert blended[103] == pytest.approx(0.6 * (8 / 11) + 0.4)
|
||||
# rank 11: norm 1/11 → 0.4*(1/11) + 0.6*1.0
|
||||
assert blended[110] == pytest.approx(0.4 * (1 / 11) + 0.6)
|
||||
|
||||
|
||||
def test_blend_rerank_constant_retrieval_scores():
|
||||
"""Zero span (all candidates same retrieval score) must not divide by zero."""
|
||||
candidates = [(1, 0.5), (2, 0.5)]
|
||||
details = {1: {}, 2: {}}
|
||||
blended = dict(_blend_rerank(candidates, [0.2, 0.8], details))
|
||||
assert blended[1] == pytest.approx(0.75 * 1.0 + 0.25 * 0.2)
|
||||
assert blended[2] == pytest.approx(0.75 * 1.0 + 0.25 * 0.8)
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Tests for tag context descriptions."""
|
||||
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from kb.database import (
|
||||
get_connection,
|
||||
init_schema,
|
||||
insert_chunk,
|
||||
insert_document,
|
||||
insert_embedding,
|
||||
tag_document,
|
||||
)
|
||||
from kb.search import hybrid_search
|
||||
|
||||
DIM = 4
|
||||
|
||||
|
||||
class _Cfg:
|
||||
search_threshold = 0.0
|
||||
rerank_enabled = False
|
||||
reranker_model = "test-reranker"
|
||||
rerank_candidates = 40
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def conn(tmp_path, monkeypatch):
|
||||
fake = types.ModuleType("kb.embeddings")
|
||||
fake.embed_texts = lambda texts: [[1.0, 0.0, 0.0, 0.0] for _ in texts]
|
||||
monkeypatch.setitem(sys.modules, "kb.embeddings", fake)
|
||||
|
||||
conn = get_connection(str(tmp_path / "kb.db"))
|
||||
init_schema(conn, embedding_dim=DIM)
|
||||
yield conn
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_migration_is_idempotent(conn):
|
||||
# Running init_schema again (fresh start on an existing DB) must not fail.
|
||||
init_schema(conn, embedding_dim=DIM)
|
||||
cols = {row[1] for row in conn.execute("PRAGMA table_info(tags)").fetchall()}
|
||||
assert "description" in cols
|
||||
|
||||
|
||||
def test_search_results_carry_tag_contexts(conn):
|
||||
doc = insert_document(conn, "Runbook", "/notes/runbook.md", "h1", "markdown")
|
||||
chunk = insert_chunk(conn, doc, 0, "restart the proxy after cert renewal")
|
||||
insert_embedding(conn, chunk, [1.0, 0.0, 0.0, 0.0])
|
||||
tag_document(conn, doc, ["ops", "draft"])
|
||||
conn.execute(
|
||||
"UPDATE tags SET description = ? WHERE name = ?",
|
||||
("Lab operations runbooks", "ops"),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
result = hybrid_search(conn, "restart proxy", _Cfg())
|
||||
hit = result["results"][0]
|
||||
assert hit["tags"] == ["draft", "ops"]
|
||||
# Only described tags appear in tag_contexts.
|
||||
assert hit["tag_contexts"] == {"ops": "Lab operations runbooks"}
|
||||
|
||||
|
||||
def test_tag_contexts_empty_when_no_descriptions(conn):
|
||||
doc = insert_document(conn, "Plain", "/notes/plain.md", "h2", "markdown")
|
||||
chunk = insert_chunk(conn, doc, 0, "some plain text about switches")
|
||||
insert_embedding(conn, chunk, [1.0, 0.0, 0.0, 0.0])
|
||||
tag_document(conn, doc, ["misc"])
|
||||
|
||||
result = hybrid_search(conn, "switches", _Cfg())
|
||||
hit = result["results"][0]
|
||||
assert hit["tag_contexts"] == {}
|
||||
@@ -0,0 +1,17 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt ./
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY *.py ./
|
||||
|
||||
ENV KB_ENGINE_URL=http://engine:8000
|
||||
ENV KB_API_KEY=
|
||||
ENV KB_MCP_API_KEY=
|
||||
ENV KB_MCP_PORT=3000
|
||||
|
||||
EXPOSE 3000
|
||||
|
||||
CMD ["python", "server.py"]
|
||||
@@ -0,0 +1,17 @@
|
||||
"""Configuration from environment variables."""
|
||||
|
||||
import os
|
||||
|
||||
|
||||
KB_ENGINE_URL = os.environ.get("KB_ENGINE_URL", "http://localhost:8000")
|
||||
KB_API_KEY = os.environ.get("KB_API_KEY", "")
|
||||
KB_MCP_API_KEY = os.environ.get("KB_MCP_API_KEY", "")
|
||||
KB_MCP_PORT = int(os.environ.get("KB_MCP_PORT", "3000"))
|
||||
KB_MCP_ALLOWED_HOSTS = os.environ.get("KB_MCP_ALLOWED_HOSTS", "")
|
||||
|
||||
|
||||
def parse_allowed_hosts() -> list[str]:
|
||||
"""Parse KB_MCP_ALLOWED_HOSTS into a list of host strings."""
|
||||
if not KB_MCP_ALLOWED_HOSTS:
|
||||
return []
|
||||
return [h.strip() for h in KB_MCP_ALLOWED_HOSTS.split(",") if h.strip()]
|
||||
+213
@@ -0,0 +1,213 @@
|
||||
"""HTTP client for the kb engine API."""
|
||||
|
||||
import httpx
|
||||
|
||||
from config import KB_ENGINE_URL, KB_API_KEY
|
||||
|
||||
|
||||
def _auth_headers() -> dict[str, str]:
|
||||
h: dict[str, str] = {}
|
||||
if KB_API_KEY:
|
||||
h["Authorization"] = f"Bearer {KB_API_KEY}"
|
||||
return h
|
||||
|
||||
|
||||
def _client() -> httpx.Client:
|
||||
return httpx.Client(base_url=KB_ENGINE_URL, headers=_auth_headers(), timeout=60.0)
|
||||
|
||||
|
||||
def search(query: str, top: int = 10, tags: list[str] | None = None,
|
||||
doc_type: str | None = None, fts_only: bool = False,
|
||||
vec_only: bool = False, threshold: float | None = None,
|
||||
explain: bool = False, rerank: bool | None = None) -> dict:
|
||||
body: dict = {"query": query, "top": top}
|
||||
if tags:
|
||||
body["tags"] = tags
|
||||
if doc_type:
|
||||
body["doc_type"] = doc_type
|
||||
if fts_only:
|
||||
body["fts_only"] = True
|
||||
if vec_only:
|
||||
body["vec_only"] = True
|
||||
if threshold is not None:
|
||||
body["threshold"] = threshold
|
||||
if explain:
|
||||
body["explain"] = True
|
||||
if rerank is not None:
|
||||
body["rerank"] = rerank
|
||||
with _client() as c:
|
||||
r = c.post("/api/v1/search", json=body)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def add_note(text: str, tags: list[str] | None = None,
|
||||
title: str | None = None) -> dict:
|
||||
fields = {"note": text}
|
||||
if tags:
|
||||
fields["tags"] = ",".join(tags)
|
||||
if title:
|
||||
fields["title"] = title
|
||||
with _client() as c:
|
||||
r = c.post("/api/v1/jobs", data=fields)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def update_note(doc_id: int, text: str) -> dict:
|
||||
with _client() as c:
|
||||
r = c.patch(f"/api/v1/notes/{doc_id}", json={"text": text})
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def get_document(doc_id: int) -> dict:
|
||||
with _client() as c:
|
||||
r = c.get(f"/api/v1/documents/{doc_id}")
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def list_documents(doc_type: str | None = None,
|
||||
tags: str | None = None) -> list[dict]:
|
||||
params: dict = {}
|
||||
if doc_type:
|
||||
params["type"] = doc_type
|
||||
if tags:
|
||||
params["tags"] = tags
|
||||
with _client() as c:
|
||||
r = c.get("/api/v1/documents", params=params)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def get_status() -> dict:
|
||||
with _client() as c:
|
||||
r = c.get("/api/v1/status")
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def list_jobs(status: str | None = None) -> list[dict]:
|
||||
params: dict = {}
|
||||
if status:
|
||||
params["status"] = status
|
||||
with _client() as c:
|
||||
r = c.get("/api/v1/jobs", params=params)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def update_tags(doc_id: int, add: list[str] | None = None,
|
||||
remove: list[str] | None = None) -> dict:
|
||||
body: dict = {}
|
||||
if add:
|
||||
body["add"] = add
|
||||
if remove:
|
||||
body["remove"] = remove
|
||||
with _client() as c:
|
||||
r = c.put(f"/api/v1/documents/{doc_id}/tags", json=body)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def delete_document(doc_id: int) -> dict:
|
||||
with _client() as c:
|
||||
r = c.delete(f"/api/v1/documents/{doc_id}")
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def _bulk_body(
|
||||
document_ids: list[int] | None = None,
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
from_id: int | None = None,
|
||||
to_id: int | None = None,
|
||||
force: bool = False,
|
||||
**extra,
|
||||
) -> dict:
|
||||
body: dict = {}
|
||||
if document_ids:
|
||||
body["document_ids"] = document_ids
|
||||
if tags:
|
||||
body["tags"] = tags
|
||||
if doc_type:
|
||||
body["doc_type"] = doc_type
|
||||
if from_id is not None:
|
||||
body["from_id"] = from_id
|
||||
if to_id is not None:
|
||||
body["to_id"] = to_id
|
||||
if force:
|
||||
body["force"] = True
|
||||
body.update(extra)
|
||||
return body
|
||||
|
||||
|
||||
def bulk_delete(
|
||||
document_ids: list[int] | None = None,
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
from_id: int | None = None,
|
||||
to_id: int | None = None,
|
||||
force: bool = False,
|
||||
) -> dict:
|
||||
body = _bulk_body(document_ids, tags, doc_type, from_id, to_id, force)
|
||||
with _client() as c:
|
||||
r = c.post("/api/v1/bulk/delete", json=body)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def bulk_tags(
|
||||
document_ids: list[int] | None = None,
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
from_id: int | None = None,
|
||||
to_id: int | None = None,
|
||||
add: list[str] | None = None,
|
||||
remove: list[str] | None = None,
|
||||
force: bool = False,
|
||||
) -> dict:
|
||||
extra = {}
|
||||
if add:
|
||||
extra["add"] = add
|
||||
if remove:
|
||||
extra["remove"] = remove
|
||||
body = _bulk_body(document_ids, tags, doc_type, from_id, to_id, force, **extra)
|
||||
with _client() as c:
|
||||
r = c.post("/api/v1/bulk/tags", json=body)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def bulk_set_tags(
|
||||
document_ids: list[int] | None = None,
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
from_id: int | None = None,
|
||||
to_id: int | None = None,
|
||||
new_tags: list[str] | None = None,
|
||||
force: bool = False,
|
||||
) -> dict:
|
||||
extra = {"new_tags": new_tags or []}
|
||||
body = _bulk_body(document_ids, tags, doc_type, from_id, to_id, force, **extra)
|
||||
with _client() as c:
|
||||
r = c.post("/api/v1/bulk/set-tags", json=body)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
def upload_file(filename: str, file_bytes: bytes,
|
||||
tags: list[str] | None = None) -> dict:
|
||||
fields: dict = {}
|
||||
if tags:
|
||||
fields["tags"] = ",".join(tags)
|
||||
with _client() as c:
|
||||
r = c.post(
|
||||
"/api/v1/jobs",
|
||||
data=fields,
|
||||
files={"file": (filename, file_bytes)},
|
||||
)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
@@ -0,0 +1,4 @@
|
||||
mcp>=1.9.0
|
||||
httpx>=0.27
|
||||
uvicorn>=0.30
|
||||
starlette>=0.38
|
||||
+478
@@ -0,0 +1,478 @@
|
||||
"""kb MCP server — exposes knowledge base operations as MCP tools."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
from mcp.server.transport_security import TransportSecuritySettings
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.routing import Mount
|
||||
|
||||
import config
|
||||
import engine
|
||||
import uploads
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
logger = logging.getLogger("kb.mcp")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Transport security — DNS rebinding protection with configurable allowed hosts
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_LOCALHOST_HOSTS = ["127.0.0.1:*", "localhost:*", "[::1]:*"]
|
||||
_LOCALHOST_ORIGINS = ["http://127.0.0.1:*", "http://localhost:*", "http://[::1]:*"]
|
||||
|
||||
_extra_hosts = config.parse_allowed_hosts()
|
||||
_allowed_hosts = _LOCALHOST_HOSTS + [f"{h}:*" for h in _extra_hosts]
|
||||
_allowed_origins = _LOCALHOST_ORIGINS + [f"http://{h}:*" for h in _extra_hosts]
|
||||
|
||||
_transport_security = TransportSecuritySettings(
|
||||
enable_dns_rebinding_protection=True,
|
||||
allowed_hosts=_allowed_hosts,
|
||||
allowed_origins=_allowed_origins,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# FastMCP server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
mcp = FastMCP(
|
||||
"kb",
|
||||
instructions=(
|
||||
"Knowledge base MCP server with hybrid semantic + full-text search. "
|
||||
"kb_search uses dense vector embeddings (semantic similarity) fused with "
|
||||
"BM25 full-text ranking, so it finds conceptually related content even "
|
||||
"when the exact words don't match — agents can ask natural-language "
|
||||
"questions rather than guessing keywords. When the engine has a "
|
||||
"cross-encoder reranker enabled, results are reranked server-side by "
|
||||
"default (pass rerank=False for lower latency). Also provides tools for "
|
||||
"adding notes, uploading files, and managing documents and tags. Use tags "
|
||||
"to organise and filter documents (e.g. tag notes with 'agent:mybot' and "
|
||||
"filter searches by that tag). This server requires Bearer token "
|
||||
"authentication — all requests are authenticated via the Authorization "
|
||||
"header at the HTTP transport layer."
|
||||
),
|
||||
transport_security=_transport_security,
|
||||
)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_search(
|
||||
query: str,
|
||||
top: int = 10,
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
fts_only: bool = False,
|
||||
explain: bool = False,
|
||||
rerank: bool | None = None,
|
||||
) -> str:
|
||||
"""Hybrid semantic (vector) + full-text search over the knowledge base.
|
||||
|
||||
Combines dense vector embeddings (semantic similarity — finds conceptually
|
||||
related content even when the wording differs) with BM25 keyword ranking,
|
||||
fused via reciprocal rank fusion. Because the search is semantic, you can
|
||||
ask natural-language questions ("what did we decide about X?") rather than
|
||||
guessing the exact keywords used in the source documents.
|
||||
|
||||
When the engine has a cross-encoder reranker enabled, the top candidates
|
||||
are reranked server-side by default — you normally do NOT need to rerank
|
||||
results yourself. Check kb_status's "rerank" block to see whether it is
|
||||
active.
|
||||
|
||||
Returns ranked chunks matching the query, with text content, relevance
|
||||
scores, and document metadata.
|
||||
|
||||
Args:
|
||||
query: The search query — a natural language question or keywords.
|
||||
top: Maximum number of results to return (default 10).
|
||||
tags: Filter results to documents with ALL of these tags.
|
||||
doc_type: Filter by document type (e.g. "note", "pdf", "markdown",
|
||||
"code", "data").
|
||||
fts_only: Disable the vector/semantic component and use only BM25
|
||||
keyword matching. Default false (hybrid mode). Set true only when
|
||||
you need exact-string matching (e.g. an error code, identifier).
|
||||
explain: Include a per-result score breakdown (BM25 score/rank, vector
|
||||
similarity/rank, rank-fusion contributions, rerank blend) under an
|
||||
"explain" key. Useful for diagnosing why a result ranked where it did.
|
||||
rerank: Set false to skip server-side reranking for lower latency.
|
||||
Default (None) uses the engine's configured behaviour.
|
||||
|
||||
Tips for complex queries:
|
||||
- Consider expanding into 2-3 variant phrasings and calling this tool multiple
|
||||
times, then deduplicating results by chunk_id. For example, search for both
|
||||
"pension revaluation rules" and "how are pensions revalued" to cast a wider net.
|
||||
- If the engine's reranker is disabled, you can still rerank the returned
|
||||
results yourself using your own judgement of relevance to the question.
|
||||
- Call kb_status to see which embedding model is in use and whether
|
||||
server-side reranking is active.
|
||||
"""
|
||||
result = engine.search(
|
||||
query=query,
|
||||
top=top,
|
||||
tags=tags or None,
|
||||
doc_type=doc_type,
|
||||
fts_only=fts_only,
|
||||
explain=explain,
|
||||
rerank=rerank,
|
||||
)
|
||||
|
||||
results_list = result if isinstance(result, list) else result.get("results", [])
|
||||
return json.dumps(results_list, indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_addnote(
|
||||
text: str,
|
||||
tags: list[str] | None = None,
|
||||
title: str | None = None,
|
||||
) -> str:
|
||||
"""Add a text note to the knowledge base for indexing and search.
|
||||
|
||||
The note is queued for ingestion — it will be chunked, embedded, and made
|
||||
searchable. Use kb_jobs to check ingestion status.
|
||||
|
||||
Args:
|
||||
text: The note text content.
|
||||
tags: Tags to apply to the note.
|
||||
title: Optional title (auto-derived from first line if omitted).
|
||||
"""
|
||||
result = engine.add_note(text=text, tags=tags or None, title=title)
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_update_note(
|
||||
document_id: int,
|
||||
text: str,
|
||||
) -> str:
|
||||
"""Update an existing note's content in place.
|
||||
|
||||
Replaces the note text, re-chunks, and re-embeds while preserving the
|
||||
document ID, creation timestamp, and tags. Only works on documents with
|
||||
doc_type "note".
|
||||
|
||||
Args:
|
||||
document_id: The ID of the note document to update.
|
||||
text: The new text content for the note.
|
||||
"""
|
||||
result = engine.update_note(document_id, text)
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_get(
|
||||
document_id: int | None = None,
|
||||
source_path: str | None = None,
|
||||
) -> str:
|
||||
"""Retrieve document details from the knowledge base.
|
||||
|
||||
Look up a document by its ID or source path. Returns full document metadata,
|
||||
tags, and chunk contents.
|
||||
|
||||
Args:
|
||||
document_id: The numeric document ID.
|
||||
source_path: The document's source path (alternative to document_id).
|
||||
"""
|
||||
if document_id is not None:
|
||||
result = engine.get_document(document_id)
|
||||
return json.dumps(result, indent=2)
|
||||
elif source_path is not None:
|
||||
docs = engine.list_documents()
|
||||
matches = [d for d in docs if d.get("source_path") == source_path]
|
||||
if not matches:
|
||||
return json.dumps({"error": "No document found with that source_path"})
|
||||
doc = engine.get_document(matches[0]["id"])
|
||||
return json.dumps(doc, indent=2)
|
||||
else:
|
||||
return json.dumps({"error": "Provide either document_id or source_path"})
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_status() -> str:
|
||||
"""Get knowledge base engine status.
|
||||
|
||||
Returns engine version, embedding model info, device info, document counts,
|
||||
database size, and ingestion queue state.
|
||||
"""
|
||||
result = engine.get_status()
|
||||
result["authenticated"] = bool(config.KB_MCP_API_KEY)
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_jobs(
|
||||
status: str | None = None,
|
||||
) -> str:
|
||||
"""List ingestion jobs and their status.
|
||||
|
||||
Returns recent jobs showing what has been queued, is processing, completed,
|
||||
or failed.
|
||||
|
||||
Args:
|
||||
status: Filter by job status ("queued", "processing", "done", "failed", "skipped").
|
||||
"""
|
||||
result = engine.list_jobs(status=status)
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_delete(
|
||||
document_id: int,
|
||||
) -> str:
|
||||
"""Permanently delete a document from the knowledge base.
|
||||
|
||||
Removes the document and all associated data (chunks, embeddings, tags,
|
||||
stored files). This action cannot be undone.
|
||||
|
||||
Args:
|
||||
document_id: The ID of the document to delete.
|
||||
"""
|
||||
result = engine.delete_document(document_id)
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_upload_start(
|
||||
filename: str,
|
||||
total_size: int,
|
||||
tags: list[str] | None = None,
|
||||
) -> str:
|
||||
"""Start a chunked file upload to the knowledge base.
|
||||
|
||||
Use this for uploading files from a remote agent. The upload process is:
|
||||
1. Call kb_upload_start to get an upload_id
|
||||
2. Call kb_upload_chunk repeatedly with base64-encoded file chunks (recommended ~1MB each)
|
||||
3. Call kb_upload_finish to submit the file for ingestion
|
||||
|
||||
Example for a 3MB file:
|
||||
upload = kb_upload_start(filename="report.pdf", total_size=3145728, tags=["project:x"])
|
||||
kb_upload_chunk(upload_id=upload["upload_id"], data="<base64 chunk 0>", chunk_index=0)
|
||||
kb_upload_chunk(upload_id=upload["upload_id"], data="<base64 chunk 1>", chunk_index=1)
|
||||
kb_upload_chunk(upload_id=upload["upload_id"], data="<base64 chunk 2>", chunk_index=2)
|
||||
result = kb_upload_finish(upload_id=upload["upload_id"])
|
||||
|
||||
Args:
|
||||
filename: Original filename (used for type detection).
|
||||
total_size: Total file size in bytes.
|
||||
tags: Tags to apply to the uploaded document.
|
||||
"""
|
||||
upload_id = uploads.start_upload(filename, total_size, tags or [])
|
||||
return json.dumps({"upload_id": upload_id})
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_upload_chunk(
|
||||
upload_id: str,
|
||||
data: str,
|
||||
chunk_index: int,
|
||||
) -> str:
|
||||
"""Upload a base64-encoded chunk of a file.
|
||||
|
||||
Part of the chunked upload flow started by kb_upload_start.
|
||||
|
||||
Args:
|
||||
upload_id: The upload ID from kb_upload_start.
|
||||
data: Base64-encoded file data for this chunk.
|
||||
chunk_index: Zero-based index of this chunk.
|
||||
"""
|
||||
try:
|
||||
uploads.add_chunk(upload_id, data, chunk_index)
|
||||
return json.dumps({"status": "ok", "chunk_index": chunk_index})
|
||||
except KeyError as e:
|
||||
return json.dumps({"error": str(e)})
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_upload_finish(
|
||||
upload_id: str,
|
||||
) -> str:
|
||||
"""Finish a chunked upload and submit the file for ingestion.
|
||||
|
||||
Reassembles all uploaded chunks and forwards the complete file to the
|
||||
engine for processing. Returns the ingestion job ID.
|
||||
|
||||
Args:
|
||||
upload_id: The upload ID from kb_upload_start.
|
||||
"""
|
||||
try:
|
||||
filename, file_bytes, tags = uploads.finish_upload(upload_id)
|
||||
result = engine.upload_file(filename, file_bytes, tags)
|
||||
return json.dumps(result, indent=2)
|
||||
except KeyError as e:
|
||||
return json.dumps({"error": str(e)})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bulk operation tools
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_bulk_delete(
|
||||
document_ids: list[int] | None = None,
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
from_id: int | None = None,
|
||||
to_id: int | None = None,
|
||||
force: bool = False,
|
||||
) -> str:
|
||||
"""Permanently delete multiple documents matching a filter.
|
||||
|
||||
Removes matched documents and all associated data (chunks, embeddings, tags,
|
||||
stored files). This action cannot be undone.
|
||||
|
||||
Selection filters combine with AND logic — at least one is required.
|
||||
|
||||
A safety threshold applies: if the operation would affect more than 70% of
|
||||
all documents, it is rejected unless force=true.
|
||||
|
||||
Args:
|
||||
document_ids: Delete documents with these specific IDs.
|
||||
tags: Delete documents that have ALL of these tags (selection filter).
|
||||
doc_type: Delete documents of this type (e.g. "note", "pdf").
|
||||
from_id: Delete documents with id >= this value.
|
||||
to_id: Delete documents with id <= this value.
|
||||
force: Override the safety threshold if it would block the operation.
|
||||
"""
|
||||
result = engine.bulk_delete(
|
||||
document_ids=document_ids, tags=tags, doc_type=doc_type,
|
||||
from_id=from_id, to_id=to_id, force=force,
|
||||
)
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_bulk_tags(
|
||||
document_ids: list[int] | None = None,
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
from_id: int | None = None,
|
||||
to_id: int | None = None,
|
||||
add: list[str] | None = None,
|
||||
remove: list[str] | None = None,
|
||||
force: bool = False,
|
||||
) -> str:
|
||||
"""Add and/or remove tags on multiple documents matching a filter.
|
||||
|
||||
Selection filters combine with AND logic — at least one is required.
|
||||
Note: the 'tags' parameter is a SELECTION FILTER (which documents to target),
|
||||
while 'add' and 'remove' specify the TAG CHANGES to apply to those documents.
|
||||
|
||||
Args:
|
||||
document_ids: Target documents with these specific IDs.
|
||||
tags: Target documents that have ALL of these tags (selection filter).
|
||||
doc_type: Target documents of this type.
|
||||
from_id: Target documents with id >= this value.
|
||||
to_id: Target documents with id <= this value.
|
||||
add: Tags to add to matched documents.
|
||||
remove: Tags to remove from matched documents.
|
||||
force: Override the safety threshold if it would block the operation.
|
||||
"""
|
||||
result = engine.bulk_tags(
|
||||
document_ids=document_ids, tags=tags, doc_type=doc_type,
|
||||
from_id=from_id, to_id=to_id, add=add, remove=remove, force=force,
|
||||
)
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def kb_bulk_set_tags(
|
||||
document_ids: list[int] | None = None,
|
||||
tags: list[str] | None = None,
|
||||
doc_type: str | None = None,
|
||||
from_id: int | None = None,
|
||||
to_id: int | None = None,
|
||||
new_tags: list[str] | None = None,
|
||||
force: bool = False,
|
||||
) -> str:
|
||||
"""Replace all tags on multiple documents with a new set.
|
||||
|
||||
Removes ALL existing tags from matched documents, then applies the new tag set.
|
||||
Selection filters combine with AND logic — at least one is required.
|
||||
Note: the 'tags' parameter is a SELECTION FILTER (which documents to target),
|
||||
while 'new_tags' is the REPLACEMENT tag set to apply.
|
||||
|
||||
Args:
|
||||
document_ids: Target documents with these specific IDs.
|
||||
tags: Target documents that have ALL of these tags (selection filter).
|
||||
doc_type: Target documents of this type.
|
||||
from_id: Target documents with id >= this value.
|
||||
to_id: Target documents with id <= this value.
|
||||
new_tags: The replacement tag set to apply to all matched documents.
|
||||
force: Override the safety threshold if it would block the operation.
|
||||
"""
|
||||
result = engine.bulk_set_tags(
|
||||
document_ids=document_ids, tags=tags, doc_type=doc_type,
|
||||
from_id=from_id, to_id=to_id, new_tags=new_tags, force=force,
|
||||
)
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auth middleware
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class BearerAuthMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next):
|
||||
if not config.KB_MCP_API_KEY:
|
||||
return await call_next(request)
|
||||
|
||||
auth_header = request.headers.get("authorization", "")
|
||||
if auth_header.startswith("Bearer ") and auth_header[7:] == config.KB_MCP_API_KEY:
|
||||
return await call_next(request)
|
||||
|
||||
return JSONResponse(
|
||||
status_code=401,
|
||||
content={"error": "Unauthorized"},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ASGI app assembly
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def create_app():
|
||||
"""Create the ASGI app with auth middleware wrapping the MCP server."""
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
mcp_app = mcp.streamable_http_app()
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
uploads.start_cleanup_task()
|
||||
logger.info("Upload cleanup task started")
|
||||
# Delegate to the MCP app's lifespan if it has one
|
||||
if hasattr(mcp_app, 'router') and hasattr(mcp_app.router, 'lifespan_context'):
|
||||
async with mcp_app.router.lifespan_context(app):
|
||||
yield
|
||||
else:
|
||||
yield
|
||||
|
||||
app = Starlette(
|
||||
routes=[Mount("/", app=mcp_app)],
|
||||
middleware=[Middleware(BearerAuthMiddleware)],
|
||||
lifespan=lifespan,
|
||||
)
|
||||
return app
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
logger.info(
|
||||
"Starting kb MCP server on port %d, engine=%s",
|
||||
config.KB_MCP_PORT,
|
||||
config.KB_ENGINE_URL,
|
||||
)
|
||||
|
||||
app = create_app()
|
||||
uvicorn.run(app, host="0.0.0.0", port=config.KB_MCP_PORT)
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Chunked upload staging management."""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import logging
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger("kb.mcp.uploads")
|
||||
|
||||
UPLOAD_TIMEOUT_SECONDS = 600 # 10 minutes
|
||||
|
||||
|
||||
@dataclass
|
||||
class StagedUpload:
|
||||
upload_id: str
|
||||
filename: str
|
||||
total_size: int
|
||||
tags: list[str]
|
||||
staging_dir: Path
|
||||
created_at: float = field(default_factory=time.time)
|
||||
chunks: dict[int, Path] = field(default_factory=dict)
|
||||
|
||||
|
||||
_uploads: dict[str, StagedUpload] = {}
|
||||
_cleanup_task: asyncio.Task | None = None
|
||||
|
||||
|
||||
def start_upload(filename: str, total_size: int, tags: list[str]) -> str:
|
||||
upload_id = str(uuid.uuid4())
|
||||
staging_dir = Path(tempfile.mkdtemp(prefix=f"kb_upload_{upload_id[:8]}_"))
|
||||
_uploads[upload_id] = StagedUpload(
|
||||
upload_id=upload_id,
|
||||
filename=filename,
|
||||
total_size=total_size,
|
||||
tags=tags,
|
||||
staging_dir=staging_dir,
|
||||
)
|
||||
logger.info("Started upload %s for %s (%d bytes)", upload_id, filename, total_size)
|
||||
return upload_id
|
||||
|
||||
|
||||
def add_chunk(upload_id: str, data_b64: str, chunk_index: int) -> None:
|
||||
upload = _uploads.get(upload_id)
|
||||
if upload is None:
|
||||
raise KeyError(f"Upload ID not found: {upload_id}")
|
||||
chunk_bytes = base64.b64decode(data_b64)
|
||||
chunk_path = upload.staging_dir / f"chunk_{chunk_index:06d}"
|
||||
chunk_path.write_bytes(chunk_bytes)
|
||||
upload.chunks[chunk_index] = chunk_path
|
||||
logger.info("Added chunk %d to upload %s (%d bytes)", chunk_index, upload_id, len(chunk_bytes))
|
||||
|
||||
|
||||
def finish_upload(upload_id: str) -> tuple[str, bytes, list[str]]:
|
||||
"""Reassemble chunks and return (filename, file_bytes, tags)."""
|
||||
upload = _uploads.get(upload_id)
|
||||
if upload is None:
|
||||
raise KeyError(f"Upload ID not found: {upload_id}")
|
||||
try:
|
||||
parts = []
|
||||
for idx in sorted(upload.chunks.keys()):
|
||||
parts.append(upload.chunks[idx].read_bytes())
|
||||
file_bytes = b"".join(parts)
|
||||
return upload.filename, file_bytes, upload.tags
|
||||
finally:
|
||||
_cleanup_upload(upload_id)
|
||||
|
||||
|
||||
def _cleanup_upload(upload_id: str) -> None:
|
||||
upload = _uploads.pop(upload_id, None)
|
||||
if upload and upload.staging_dir.exists():
|
||||
shutil.rmtree(upload.staging_dir, ignore_errors=True)
|
||||
|
||||
|
||||
async def cleanup_abandoned_uploads() -> None:
|
||||
"""Background task that removes uploads older than the timeout."""
|
||||
while True:
|
||||
await asyncio.sleep(60)
|
||||
now = time.time()
|
||||
expired = [
|
||||
uid for uid, u in _uploads.items()
|
||||
if now - u.created_at > UPLOAD_TIMEOUT_SECONDS
|
||||
]
|
||||
for uid in expired:
|
||||
logger.warning("Cleaning up abandoned upload %s", uid)
|
||||
_cleanup_upload(uid)
|
||||
|
||||
|
||||
def start_cleanup_task() -> None:
|
||||
global _cleanup_task
|
||||
if _cleanup_task is None or _cleanup_task.done():
|
||||
_cleanup_task = asyncio.create_task(cleanup_abandoned_uploads())
|
||||
@@ -0,0 +1,63 @@
|
||||
# kb — Next Steps
|
||||
|
||||
> Implementation status (2026-08-22): `document_id`, metadata-only info,
|
||||
> title/filename filters, document-level `find`, and short Docling chunk
|
||||
> filtering are implemented on `feature/tasks-5-15-completion`. OCR is already
|
||||
> enabled through RapidOCR; further OCR tuning remains measurement-gated.
|
||||
|
||||
UX improvements to make documents easier to find and inspect, prompted by a session where searching for an uploaded PDF (`M38T_PHEV_RHD_OM_EN_UK_20251209.pdf`, doc id 2077, 1801 chunks) surfaced lots of chunk hits but no obvious path back to the original document.
|
||||
|
||||
## Problems observed
|
||||
|
||||
### 1. `kb list` silently ignores positional arguments
|
||||
|
||||
```
|
||||
kb list --type pdf "M38T_PHEV_RHD_OM_EN_UK_20251209"
|
||||
```
|
||||
|
||||
The quoted term is dropped without warning; user gets the default newest-first listing and assumes the document is missing. `kb list` currently only supports `--tags` and `--type` filters.
|
||||
|
||||
### 2. `kb search` returns chunks with no `document_id`
|
||||
|
||||
Result objects expose `chunk_id`, `title`, `source_path`, `tags` — but not `document_id`. To get from a search hit back to the owning document you have to title-match against `kb list` output or call an undocumented endpoint. The skill docs even claim a `source.document_id` field that isn't actually present in the CLI output.
|
||||
|
||||
### 3. `kb info` dumps every chunk with no summary mode
|
||||
|
||||
`kb info 2077` returns ~1801 chunk objects. The document-level metadata (`id`, `title`, `original_filename`, `source_path`, `stored_path`, `doc_type`, `language`, `content_hash`, `has_file`, `tags`, `created_at`, `updated_at`) **is** present at the top level of the JSON, but in practice it's invisible — human format presumably dumps the chunk list and the user sees only chunks.
|
||||
|
||||
There's no way to ask for "just tell me about this document."
|
||||
|
||||
### 4. Search hits can look like noise on image-heavy PDFs
|
||||
|
||||
Top chunks for the M38T search were single characters (`"1"`, `"B"`, `"\""`). Almost certainly an FTS artefact on short tokens from a scan/image-heavy PDF — but it makes the result set look broken. Worth considering a minimum-text-length filter on indexed chunks, or down-weighting very short chunks in ranking.
|
||||
|
||||
## Proposed changes
|
||||
|
||||
### Small / high-value
|
||||
|
||||
- **`kb info --no-chunks`** (or make `--chunks` opt-in): default to metadata + chunk count, only include chunks when asked. Human format should always lead with the metadata block.
|
||||
- **`kb list --title <substring>`** (or accept a positional query) for filename / title search. At minimum, error or warn when positional args are passed and ignored.
|
||||
- **Include `document_id` in `kb search` result objects.** Either at the top of each result or under `source.document_id` (matching the skill docs).
|
||||
|
||||
### Medium
|
||||
|
||||
- **`kb find <query>`** as a doc-level search that aggregates chunk hits per document and returns ranked *documents* (with hit count, top chunk preview). This is what users usually want when they say "find my PDF about X."
|
||||
- **Update the `kb` skill docs** to match actual CLI output shape, and to steer users toward `kb list | jq` for filename lookups until proper filtering lands.
|
||||
|
||||
### Larger
|
||||
|
||||
- **Quality filter for short chunks** during ingestion (e.g. drop chunks with < N alphanumeric chars, or fold them into neighbours). Stops scanned/image-heavy PDFs from polluting search.
|
||||
- **OCR path for scan-heavy PDFs.** The M38T manual extracted enough real text to be useful, but other "scan" docs likely don't. Detect low text density per page and route through OCR.
|
||||
|
||||
## Quick reference (current workarounds)
|
||||
|
||||
```bash
|
||||
# Find a doc by filename
|
||||
kb list --type pdf --format json | jq '.[] | select(.title | contains("M38T"))'
|
||||
|
||||
# Get just metadata for a doc
|
||||
kb info 2077 --format json | jq 'del(.chunks)'
|
||||
|
||||
# Download the original
|
||||
kb export 2077 -o manual.pdf
|
||||
```
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-03-29
|
||||
@@ -0,0 +1,69 @@
|
||||
## Context
|
||||
|
||||
When a document is ingested, the worker chunks its content and stores each chunk's text in the `chunks` table. FTS5 triggers index that text, and the embedding model embeds it. The document title is stored only in `documents.title` — it never participates in search. This means short documents (or documents whose content lacks the title keywords) are invisible to queries that match the title.
|
||||
|
||||
The reindex endpoint (`POST /api/v1/reindex`) currently reads `chunks.text` and re-embeds it. Any fix must apply consistently at both ingestion and reindex time.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Document titles are searchable via both FTS5 and vector search
|
||||
- Section header breadcrumbs (when present in chunk metadata) are also searchable
|
||||
- Search results continue to return the original chunk text (no title prefix in the `text` field returned to clients)
|
||||
- Existing documents become searchable by title after a `kb reindex`
|
||||
- No schema-breaking migration — additive column only
|
||||
|
||||
**Non-Goals:**
|
||||
- Changing the chunking strategies themselves (note, markdown, code, docling)
|
||||
- Adding a separate title-search endpoint or client-side title filtering
|
||||
- Changing the search result JSON structure
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Add an `enriched_text` column to the `chunks` table
|
||||
|
||||
Store the title-prefixed text in a new `chunks.enriched_text` column alongside the existing `chunks.text`. The `text` column remains the raw chunk content (used for display in search results). The `enriched_text` column holds `"{title}\n\n{section_header}\n\n{text}"` (with section_header omitted when absent).
|
||||
|
||||
**Why not just modify `chunks.text`?** The title would then appear in every search result's text field, which is redundant (title is already a separate field) and would confuse consumers that display results.
|
||||
|
||||
**Why not reconstruct enriched text on-the-fly at search time?** FTS5 uses an external content table and triggers — it needs a real column to index. Reconstructing via JOIN at FTS query time would defeat the purpose of the FTS index.
|
||||
|
||||
### 2. Point FTS5 at `enriched_text` instead of `text`
|
||||
|
||||
Update the FTS5 virtual table definition and its sync triggers to index `enriched_text` rather than `text`. This is the core change that makes titles searchable via keyword search.
|
||||
|
||||
Since FTS5 external content tables cannot be ALTERed, existing databases require a rebuild: drop and recreate `chunks_fts` and its triggers, then repopulate. This is handled as a schema migration in `init_schema`.
|
||||
|
||||
### 3. Embed `enriched_text` instead of `text`
|
||||
|
||||
At ingestion time, pass `enriched_text` values to `embed_texts()` instead of raw chunk text. At reindex time, read `enriched_text` from the database. This makes titles searchable via vector similarity too.
|
||||
|
||||
### 4. Build enriched text in the worker, not in the ingest modules
|
||||
|
||||
The enrichment format is: `"{title}\n\n{chunk_text}"` or `"{title} > {section_header}\n\n{chunk_text}"` when a section header exists in chunk metadata.
|
||||
|
||||
This happens in `worker._process_job()` after chunking and before embedding/insertion. The ingest modules remain unchanged — they continue to return raw chunk text and metadata.
|
||||
|
||||
### 5. Schema migration adds `enriched_text` and rebuilds FTS
|
||||
|
||||
The `init_schema` function will:
|
||||
1. Add `enriched_text TEXT` column to `chunks` if missing
|
||||
2. Backfill `enriched_text` from existing data (join with `documents.title` and chunk metadata)
|
||||
3. Drop and recreate `chunks_fts` to index `enriched_text` instead of `text`
|
||||
4. Recreate the FTS sync triggers
|
||||
|
||||
This is safe because the migration only runs when the column is missing (first startup after upgrade). The backfill uses a single UPDATE...FROM query.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
**Slightly larger database** — Each chunk stores the title string twice (once in `enriched_text`, once via the document FK). For a typical KB with short titles this is negligible (< 1% size increase).
|
||||
→ Acceptable for the search quality improvement.
|
||||
|
||||
**FTS rebuild on upgrade** — First startup after upgrade will rebuild the FTS index, which takes a few seconds for large KBs.
|
||||
→ This is a one-time cost and happens automatically.
|
||||
|
||||
**Embedding drift** — Existing vector embeddings won't include title context until `kb reindex` is run. The FTS backfill happens automatically, but vectors require an explicit reindex.
|
||||
→ Document this in release notes. The FTS improvement alone is a significant win even without reindexing vectors.
|
||||
|
||||
**Title changes not propagated** — If a document's title were ever updated, `enriched_text` would be stale. Currently the engine has no title-update endpoint, so this is not a concern.
|
||||
→ No mitigation needed now. If title editing is added later, it should update enriched_text.
|
||||
@@ -0,0 +1,28 @@
|
||||
## Why
|
||||
|
||||
Short documents and notes are unsearchable when the user's query matches the document title but not the chunk content. For example, a document titled "Suitcase Locks" containing only "Steve = 1234 / Theresa = 4567" is invisible to both FTS and vector search for the query "suitcase locks". This is because chunk text — the only thing indexed and embedded — does not include the document title. This is a standard RAG deficiency that most pipelines solve by prepending title context to each chunk.
|
||||
|
||||
## What Changes
|
||||
|
||||
- **Prepend document title to chunk text at ingestion time**: Before embedding and FTS indexing, each chunk's text will be prefixed with the document title (e.g., `"Suitcase Locks\n\n Steve = 363..."`). This ensures the title participates in both full-text and semantic search.
|
||||
- **Include section header context in chunk text**: For chunks that have a `section_header` in their metadata, prepend the header breadcrumb too (e.g., `"DCG Lab Hardware > GRIMDAWN > motherboard\n\nMSI X870 Tomahawk..."`). This improves search for queries that reference section names.
|
||||
- **Store the raw chunk text separately from the enriched text**: The original chunk text (without title prefix) must remain accessible so that search results don't display the prepended title redundantly — the title is already returned as a separate field.
|
||||
- **Reindex command must apply the same enrichment**: When `kb reindex` re-embeds all chunks, it must reconstruct the enriched text (title + section header + chunk text) from stored metadata.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `chunk-enrichment`: Prepending document title and section context to chunk text before indexing and embedding, while preserving the original text for display.
|
||||
|
||||
### Modified Capabilities
|
||||
- `engine-api`: The search endpoint's returned `text` field must continue to show the original chunk text (without the prepended title), so no visible API change, but the internal indexing behaviour changes. The reindex endpoint must apply enrichment consistently.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Engine ingestion pipeline** (`worker.py`): The `_process_job` function must build enriched text from title + section headers + chunk text before passing to `embed_texts()` and `insert_chunk()`.
|
||||
- **Database schema** (`database.py`): Need to store both raw `text` (for display) and enriched `text` (for FTS/embedding), or reconstruct enriched text at index time. Simplest approach: store raw text in `chunks.text`, use enriched text only for FTS content and embedding vectors.
|
||||
- **FTS triggers** (`database.py`): The FTS5 external content table currently mirrors `chunks.text`. If we add an `enriched_text` column, the FTS index should be built from that instead.
|
||||
- **Reindex flow** (`worker.py` / `database.py`): Must reconstruct enriched text by joining chunk metadata with document title.
|
||||
- **Search result enrichment** (`routes/search.py`): No change needed — results already return `chunks.text` (raw) and `documents.title` separately.
|
||||
- **All four ingest modules** (`note.py`, `markdown.py`, `code.py`, `docling_pipeline.py`): No changes needed — enrichment happens after chunking, in the worker.
|
||||
- **Existing documents**: Require a `reindex` to benefit from the new enrichment. No data migration needed since the original text is preserved.
|
||||
@@ -0,0 +1,75 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Chunk text enrichment with document title
|
||||
|
||||
The engine SHALL prepend the document title to each chunk's text before FTS indexing and vector embedding. The enriched text SHALL be stored in a dedicated `enriched_text` column on the `chunks` table. The original chunk text SHALL remain in the `text` column for display purposes.
|
||||
|
||||
The enrichment format SHALL be:
|
||||
- Without section header: `"{title}\n\n{chunk_text}"`
|
||||
- With section header: `"{title} > {section_header}\n\n{chunk_text}"`
|
||||
|
||||
Where `section_header` is the value from the chunk's metadata `section_header` field, when present.
|
||||
|
||||
#### Scenario: Note ingestion with title enrichment
|
||||
- **WHEN** a note titled "Suitcase Locks" with content "Steve = 363" is ingested
|
||||
- **THEN** the `chunks.text` column SHALL contain "Steve = 363" and the `chunks.enriched_text` column SHALL contain "Suitcase Locks\n\nSteve = 363"
|
||||
|
||||
#### Scenario: Markdown chunk with section header enrichment
|
||||
- **WHEN** a markdown document titled "DCG Lab Hardware" produces a chunk with section_header "GRIMDAWN > motherboard" and text "MSI X870 Tomahawk"
|
||||
- **THEN** the `chunks.enriched_text` SHALL contain "DCG Lab Hardware > GRIMDAWN > motherboard\n\nMSI X870 Tomahawk"
|
||||
|
||||
#### Scenario: Chunk without section header
|
||||
- **WHEN** a document titled "Docker Tips" produces a chunk with no section_header in metadata and text "dbash() { docker exec -it $1 bash; }"
|
||||
- **THEN** the `chunks.enriched_text` SHALL contain "Docker Tips\n\ndbash() { docker exec -it $1 bash; }"
|
||||
|
||||
---
|
||||
|
||||
### Requirement: FTS5 indexes enriched text
|
||||
|
||||
The FTS5 virtual table `chunks_fts` SHALL index the `enriched_text` column instead of the `text` column. All FTS sync triggers (insert, update, delete) SHALL operate on `enriched_text`.
|
||||
|
||||
#### Scenario: FTS search matches document title
|
||||
- **WHEN** a user searches for "suitcase locks" and a document titled "Suitcase Locks" exists with chunk text "Steve = 363"
|
||||
- **THEN** the FTS5 search SHALL return that chunk as a match
|
||||
|
||||
#### Scenario: FTS search still matches chunk content
|
||||
- **WHEN** a user searches for "MSI X870" and a chunk contains that text in its body
|
||||
- **THEN** the FTS5 search SHALL return that chunk as a match (enrichment does not break content matching)
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Vector embeddings use enriched text
|
||||
|
||||
The embedding model SHALL receive `enriched_text` (not raw `text`) when generating vectors during both initial ingestion and reindex operations.
|
||||
|
||||
#### Scenario: Vector search matches document title
|
||||
- **WHEN** a user searches semantically for "luggage combination codes" and a document titled "Suitcase Locks" exists
|
||||
- **THEN** the vector search SHALL return that chunk with higher similarity than it would without title enrichment
|
||||
|
||||
#### Scenario: Reindex uses enriched text
|
||||
- **WHEN** `POST /api/v1/reindex` is called
|
||||
- **THEN** the engine SHALL read `enriched_text` from the chunks table and embed that (not `text`)
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Schema migration adds enriched_text column
|
||||
|
||||
On startup, `init_schema` SHALL add the `enriched_text` column to the `chunks` table if it does not exist. It SHALL then backfill `enriched_text` for all existing chunks by joining with `documents.title` and parsing chunk metadata for section headers. It SHALL rebuild the FTS5 table and triggers to index `enriched_text`.
|
||||
|
||||
#### Scenario: First startup after upgrade
|
||||
- **WHEN** the engine starts and `chunks.enriched_text` column does not exist
|
||||
- **THEN** the engine SHALL add the column, backfill all rows, drop and recreate `chunks_fts` to index `enriched_text`, and recreate the FTS sync triggers
|
||||
|
||||
#### Scenario: Subsequent startup
|
||||
- **WHEN** the engine starts and `chunks.enriched_text` column already exists
|
||||
- **THEN** the engine SHALL not perform any migration and start normally
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Search results return raw text
|
||||
|
||||
Search results SHALL continue to return the original chunk text (from `chunks.text`) in the `text` field, not the enriched text. The document title is already returned as a separate `title` field.
|
||||
|
||||
#### Scenario: Search result text field
|
||||
- **WHEN** a search returns a chunk from document "Suitcase Locks" with raw text "Steve = 363"
|
||||
- **THEN** the result `text` field SHALL be "Steve = 363" (not "Suitcase Locks\n\nSteve = 363")
|
||||
@@ -0,0 +1,31 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Background ingestion worker
|
||||
|
||||
The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.
|
||||
|
||||
#### Scenario: Successful PDF ingestion
|
||||
- **WHEN** the background worker picks up a queued PDF job
|
||||
- **THEN** it SHALL update the job status to `processing`, run Docling conversion and chunking, build enriched text for each chunk by prepending the document title, embed all chunks using enriched text, insert document and chunks into the database, move the staged file to `{data_dir}/documents/{content_hash}.pdf`, update `documents.stored_path` with the permanent path, store the original filename in `documents.original_filename`, update the job status to `done` with the resulting document_id and chunk count, and clean up the staging entry
|
||||
|
||||
#### Scenario: Ingestion failure
|
||||
- **WHEN** the background worker encounters an error during processing (e.g., corrupt PDF)
|
||||
- **THEN** it SHALL update the job status to `failed` with the error message, delete the staged file, and continue processing the next queued job
|
||||
|
||||
#### Scenario: Search during active ingestion
|
||||
- **WHEN** a search request arrives while the background worker is processing a job
|
||||
- **THEN** the search SHALL execute without blocking (SQLite WAL mode) and return results from already-ingested documents
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Engine status and reindex
|
||||
|
||||
The engine SHALL provide status information and support re-embedding all chunks. The `version` field in the status response SHALL always be present and SHALL reflect the engine's release version as read from the `VERSION` file. This field is the contract used by clients for compatibility checking.
|
||||
|
||||
#### Scenario: Get engine status
|
||||
- **WHEN** a client sends `GET /api/v1/status`
|
||||
- **THEN** the engine SHALL return JSON with `version` (string, from VERSION file), model_name, embedding_dim, GPU device info, database stats (document count by type, total chunks, DB size), and queue stats (queued/processing job count)
|
||||
|
||||
#### Scenario: Trigger reindex
|
||||
- **WHEN** a client sends `POST /api/v1/reindex`
|
||||
- **THEN** the engine SHALL re-embed all existing chunks using the `enriched_text` column and the currently loaded model, and return progress information. This operation SHALL NOT block search queries.
|
||||
@@ -0,0 +1,33 @@
|
||||
## 1. Schema Migration
|
||||
|
||||
- [x] 1.1 Add `enriched_text TEXT` column to `chunks` table in `database.py:init_schema` (with migration check for existing DBs)
|
||||
- [x] 1.2 Write backfill query: `UPDATE chunks SET enriched_text = ... FROM documents` joining title and parsing chunk metadata for section_header
|
||||
- [x] 1.3 Drop and recreate `chunks_fts` virtual table to index `enriched_text` instead of `text`
|
||||
- [x] 1.4 Update FTS sync triggers (`chunks_ai`, `chunks_ad`, `chunks_au`) to use `enriched_text`
|
||||
|
||||
## 2. Enrichment Helper
|
||||
|
||||
- [x] 2.1 Create `build_enriched_text(title: str, chunk_text: str, metadata: dict | None) -> str` helper function in `worker.py` (or a shared util) that formats `"{title} > {section_header}\n\n{chunk_text}"` or `"{title}\n\n{chunk_text}"`
|
||||
|
||||
## 3. Ingestion Pipeline
|
||||
|
||||
- [x] 3.1 Update `worker._process_job()` to build enriched text for each chunk after chunking
|
||||
- [x] 3.2 Pass enriched text to `embed_texts()` instead of raw chunk text
|
||||
- [x] 3.3 Pass enriched text to `database.insert_chunk()` as the new `enriched_text` parameter
|
||||
- [x] 3.4 Update `database.insert_chunk()` to accept and store `enriched_text`
|
||||
|
||||
## 4. Reindex
|
||||
|
||||
- [x] 4.1 Update `routes/reindex.py` to read `enriched_text` from chunks table and embed that instead of `text`
|
||||
|
||||
## 5. Search Results
|
||||
|
||||
- [x] 5.1 Verify `search.py:_enrich()` returns `chunks.text` (raw) not `enriched_text` — no change expected, but confirm
|
||||
|
||||
## 6. Testing
|
||||
|
||||
- [x] 6.1 Test: ingest a short note with a descriptive title, search by title keywords, confirm it is found
|
||||
- [x] 6.2 Test: ingest a markdown doc, search by section header, confirm chunks are found
|
||||
- [x] 6.3 Test: verify search result `text` field does not contain the prepended title
|
||||
- [x] 6.4 Test: run `reindex`, verify enriched text is used for new embeddings
|
||||
- [x] 6.5 Test: verify schema migration backfills enriched_text for pre-existing chunks on startup
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-03-31
|
||||
@@ -0,0 +1,52 @@
|
||||
## Context
|
||||
|
||||
README.md currently serves as a single documentation file for both users and developers. It contains ~290 lines mixing installation/usage instructions with build-from-source steps, release scripts, Docker image internals, and developer notes (e.g., ROCm migration plans). There is no DEVELOPER.md or CONTRIBUTING.md file.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Separate user-facing documentation (README.md) from developer-facing documentation (DEVELOPER.md)
|
||||
- README.md should answer: "What is this? How do I install it? How do I use it?"
|
||||
- DEVELOPER.md should answer: "How do I build from source? How do I release? How do I contribute?"
|
||||
- Provide a clear cross-reference link between the two files
|
||||
|
||||
**Non-Goals:**
|
||||
- Rewriting or improving documentation content itself (just moving it)
|
||||
- Creating additional docs files (CONTRIBUTING.md, architecture docs, etc.)
|
||||
- Changing any code, build scripts, or CI configuration
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Single DEVELOPER.md file (not multiple docs files)
|
||||
|
||||
All developer content goes into one top-level DEVELOPER.md rather than a `docs/` directory or separate CONTRIBUTING.md / BUILDING.md files. The total developer content is small enough (~80 lines) that splitting further would be unnecessary overhead. A single file at the repo root is immediately discoverable.
|
||||
|
||||
**Alternative considered**: `docs/` directory with multiple files. Rejected because the content volume doesn't justify the structure, and root-level DEVELOPER.md is a well-known convention.
|
||||
|
||||
### 2. Content split boundary
|
||||
|
||||
Content stays in README.md if it's needed by someone who just wants to **run** kb. Content moves to DEVELOPER.md if it's only needed by someone who wants to **build, modify, or release** kb.
|
||||
|
||||
Specifically moving to DEVELOPER.md:
|
||||
- "From source" subsections under both engine and client install
|
||||
- Entire "Building and releasing" section (release scripts, version checking, Docker image tags, registry overrides)
|
||||
- "Future: ROCm runtime migration" developer note
|
||||
|
||||
Staying in README.md:
|
||||
- Architecture overview (helps users understand what they're running)
|
||||
- Pre-built image / release install instructions
|
||||
- Client configuration
|
||||
- Usage examples
|
||||
- Engine configuration table
|
||||
- Data portability
|
||||
- API reference
|
||||
- Claude Code skill reference
|
||||
|
||||
### 3. Cross-reference approach
|
||||
|
||||
A short note in README.md's Quick Start section pointing to DEVELOPER.md for building from source. No back-link needed from DEVELOPER.md since developers will naturally find README.md first.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Stale cross-references]** If DEVELOPER.md sections are renamed, the link from README.md could break. Mitigation: link to the file, not to a specific anchor.
|
||||
- **[Discoverability]** Some users who want to build from source might miss DEVELOPER.md. Mitigation: explicit "See DEVELOPER.md" callout in the Quick Start section where "from source" instructions used to be.
|
||||
@@ -0,0 +1,28 @@
|
||||
## Why
|
||||
|
||||
README.md currently mixes user-facing content (what kb does, how to install and use it) with developer-facing content (building from source, releasing, Docker image internals, architecture deep-dives). Users looking for quick-start instructions have to scroll past release scripts and build commands. Developers looking for contribution/build info have to hunt through user docs. Splitting these into README.md (users) and DEVELOPER.md (developers/contributors) follows standard open-source convention and makes both audiences' experience cleaner.
|
||||
|
||||
## What Changes
|
||||
|
||||
- **Trim README.md** to focus on user-facing content: what kb is, how to install (from pre-built images/releases), how to configure, how to use, engine configuration reference, data portability, and API reference.
|
||||
- **Remove "from source" build instructions** from README.md (both engine and client sections).
|
||||
- **Remove "Building and releasing" section** from README.md entirely.
|
||||
- **Remove "Future: ROCm runtime migration"** developer note from README.md.
|
||||
- **Create DEVELOPER.md** containing: building engine from source, building client from source, release process (client and engine), Docker image details, version checking, ROCm migration notes, and any other contributor-oriented content.
|
||||
- **Add a link** from README.md to DEVELOPER.md for developers who want to build from source or contribute.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `developer-docs`: Developer-facing documentation covering building from source, releasing, and contributing.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
(none - no spec-level behavior changes, this is a documentation restructuring)
|
||||
|
||||
## Impact
|
||||
|
||||
- **Files modified**: `README.md` (trimmed)
|
||||
- **Files created**: `DEVELOPER.md` (new)
|
||||
- **No code changes**: purely documentation restructuring
|
||||
- **No API changes**: no functional impact
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: DEVELOPER.md exists at repo root
|
||||
The repository SHALL have a `DEVELOPER.md` file at the project root containing all developer-facing documentation.
|
||||
|
||||
#### Scenario: File exists
|
||||
- **WHEN** a developer navigates to the repository root
|
||||
- **THEN** a `DEVELOPER.md` file SHALL be present
|
||||
|
||||
### Requirement: DEVELOPER.md contains build-from-source instructions
|
||||
DEVELOPER.md SHALL contain instructions for building both the engine and client from source.
|
||||
|
||||
#### Scenario: Engine build from source
|
||||
- **WHEN** a developer reads DEVELOPER.md
|
||||
- **THEN** it SHALL include instructions for starting the engine from source using compose files (both NVIDIA and ROCm)
|
||||
|
||||
#### Scenario: Client build from source
|
||||
- **WHEN** a developer reads DEVELOPER.md
|
||||
- **THEN** it SHALL include instructions for building the client binary from source using `make build` and `make all`
|
||||
|
||||
### Requirement: DEVELOPER.md contains release process
|
||||
DEVELOPER.md SHALL document the release process for both client and engine, including release scripts, version bumping, and Docker image tagging.
|
||||
|
||||
#### Scenario: Client release documentation
|
||||
- **WHEN** a developer reads DEVELOPER.md
|
||||
- **THEN** it SHALL include `release-client.sh` usage with flag options (--gitea, --github, --minor, --no-increment, --dry-run)
|
||||
|
||||
#### Scenario: Engine release documentation
|
||||
- **WHEN** a developer reads DEVELOPER.md
|
||||
- **THEN** it SHALL include `release-engine.sh` usage with flag options and Docker image tag conventions
|
||||
|
||||
#### Scenario: Version checking
|
||||
- **WHEN** a developer reads DEVELOPER.md
|
||||
- **THEN** it SHALL include how to check client and engine versions
|
||||
|
||||
### Requirement: DEVELOPER.md contains developer notes
|
||||
DEVELOPER.md SHALL include any forward-looking developer notes such as migration plans or technical debt items.
|
||||
|
||||
#### Scenario: ROCm migration note
|
||||
- **WHEN** a developer reads DEVELOPER.md
|
||||
- **THEN** it SHALL include the ROCm runtime migration note about onnxruntime and MIGraphX
|
||||
|
||||
### Requirement: README.md excludes developer-only content
|
||||
README.md SHALL NOT contain build-from-source instructions, release processes, or developer-only notes.
|
||||
|
||||
#### Scenario: No from-source build steps in README
|
||||
- **WHEN** a user reads README.md
|
||||
- **THEN** there SHALL be no "From source" subsections under engine or client installation
|
||||
|
||||
#### Scenario: No release section in README
|
||||
- **WHEN** a user reads README.md
|
||||
- **THEN** there SHALL be no "Building and releasing" section
|
||||
|
||||
#### Scenario: No developer notes in README
|
||||
- **WHEN** a user reads README.md
|
||||
- **THEN** there SHALL be no "Future: ROCm runtime migration" section
|
||||
|
||||
### Requirement: README.md cross-references DEVELOPER.md
|
||||
README.md SHALL include a link to DEVELOPER.md for users who want to build from source or contribute.
|
||||
|
||||
#### Scenario: Developer link in quick start
|
||||
- **WHEN** a user reads the Quick Start section of README.md
|
||||
- **THEN** there SHALL be a note pointing to DEVELOPER.md for building from source
|
||||
@@ -0,0 +1,17 @@
|
||||
## 1. Create DEVELOPER.md
|
||||
|
||||
- [x] 1.1 Create DEVELOPER.md at repo root with engine build-from-source instructions (compose.nvidia.yaml and compose.rocm.yaml)
|
||||
- [x] 1.2 Add client build-from-source instructions (make build, make all)
|
||||
- [x] 1.3 Add "Building and releasing" section: release-client.sh and release-engine.sh usage with all flag options
|
||||
- [x] 1.4 Add version checking instructions (kb --version, curl status endpoint)
|
||||
- [x] 1.5 Add Docker image tag conventions and registry override documentation
|
||||
- [x] 1.6 Add "Future: ROCm runtime migration" developer note
|
||||
|
||||
## 2. Trim README.md
|
||||
|
||||
- [x] 2.1 Remove "From source (for development)" subsection under engine quick start
|
||||
- [x] 2.2 Remove "From source (for development)" subsection under client installation
|
||||
- [x] 2.3 Remove entire "Building and releasing" section
|
||||
- [x] 2.4 Remove "Future: ROCm runtime migration" section
|
||||
- [x] 2.5 Add cross-reference note to DEVELOPER.md in the Quick Start section for building from source
|
||||
- [x] 2.6 Move API reference section from README.md to DEVELOPER.md
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-03-31
|
||||
@@ -0,0 +1,51 @@
|
||||
## Context
|
||||
|
||||
The kb client currently overloads the root Cobra command to handle both command dispatch and implicit note ingestion. Any unrecognized multi-word input is silently submitted as a note via `POST /api/v1/jobs`. This was introduced to reduce friction for note-taking but has proven error-prone — typos in commands create unwanted notes. A single-word guard was added but multi-word typos still slip through.
|
||||
|
||||
The root command has: custom `ArbitraryArgs` validation, a `RunE` with arg-count branching, a `--tags` flag for the note shorthand, a custom usage template with `isRootCmd` template function, and `submitNote()` living in `add.go`.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Eliminate accidental note creation from mistyped commands
|
||||
- Provide a clean, explicit `addnote` command that pairs with existing `addfile`
|
||||
- Revert root command to standard Cobra behaviour (no custom args, no custom template)
|
||||
- Keep the same API contract — `POST /api/v1/jobs` with `note` field unchanged
|
||||
|
||||
**Non-Goals:**
|
||||
- Changing the engine API
|
||||
- Modifying `addfile` behaviour
|
||||
- Adding new content types (url, bookmark, etc.)
|
||||
- Backward compatibility shim for `kb "text"` syntax
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. New `addnote` command in its own file
|
||||
|
||||
Create `client/cmd/addnote.go` with a `cobra.Command` that takes `ExactArgs(1)` — a single quoted string. This mirrors `addfile` which also takes `ExactArgs(1)`.
|
||||
|
||||
**Rationale**: Keeps each command in its own file (consistent with the existing pattern). `ExactArgs(1)` means the user must quote multi-word notes, which is unambiguous and avoids the flag-parsing edge cases that plagued the implicit shorthand.
|
||||
|
||||
**Alternative considered**: Joining `ArbitraryArgs` like the old shorthand. Rejected — this is exactly the ambiguity we're removing.
|
||||
|
||||
### 2. Move `submitNote()` from `add.go` to `addnote.go`
|
||||
|
||||
The function is only used by the addnote command, so it belongs in the same file.
|
||||
|
||||
**Rationale**: `add.go` becomes purely about file operations (it already is, aside from hosting `submitNote()`). Clean separation.
|
||||
|
||||
### 3. Fully revert root command to Cobra defaults
|
||||
|
||||
Remove: `ArbitraryArgs`, custom `RunE` (replace with nil — Cobra shows help by default), `--tags` flag on root, custom usage template, `isRootCmd` template function.
|
||||
|
||||
**Rationale**: The root command should do one thing — dispatch to subcommands. All the custom logic was there to support the implicit shorthand which is being removed.
|
||||
|
||||
### 4. `addnote` gets its own `--tags` flag
|
||||
|
||||
The `--tags` flag moves from the root command to `addnote`, matching how `addfile` already has its own `--tags` flag.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **Breaking change for existing users** → Mitigated by clear error messaging. If someone types `kb "some text"`, Cobra will say "unknown command". The `examples` command will show the new syntax.
|
||||
- **Slightly more typing for notes** (`kb addnote "text"` vs `kb "text"`) → Acceptable trade-off for eliminating accidental ingestion. Tab-completion helps.
|
||||
- **Scripts using old syntax will break** → This is intentional. The old syntax was a foot-gun.
|
||||
@@ -0,0 +1,32 @@
|
||||
## Why
|
||||
|
||||
The implicit note shorthand (`kb "some text"`) makes it too easy to accidentally add notes when mistyping commands. Despite the single-word guard, any multi-word typo (e.g. `kb lisst --type pdf`) silently creates a note. The root command doing double-duty as both command dispatcher and note ingester undermines user trust. Reverting to explicit, structured add commands eliminates accidental ingestion and gives every content type a clear, discoverable verb.
|
||||
|
||||
## What Changes
|
||||
|
||||
- **New `addnote` command**: `kb addnote <text>` takes a single quoted positional argument and submits it as a note. Supports `--tags`. The `submitNote()` logic moves from `root.go` to a new `addnote.go` command file.
|
||||
- **Remove implicit note shorthand**: The root command reverts to standard Cobra behaviour — no `ArbitraryArgs`, no special arg-count logic, no `--tags` flag on root. Unknown input gets Cobra's default "unknown command" error.
|
||||
- **Remove custom usage template**: The root command no longer needs the `isRootCmd` template logic. Standard Cobra usage template for all commands.
|
||||
- **Update examples**: `examples.go` updated to show `kb addnote` instead of bare `kb "text"`.
|
||||
- **Update tests**: Remove implicit note shorthand tests, add `addnote` command tests.
|
||||
- **`addfile` unchanged**: Stays exactly as-is.
|
||||
- **BREAKING**: `kb "note text"` no longer works. Users must use `kb addnote "note text"`.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
_(none)_
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `go-client`: The "Implicit note shorthand" requirement is removed entirely and replaced by a new "Add note command" requirement. The "Add command (file and note ingestion)" requirement description is updated to reflect `addnote` / `addfile` as the two ingestion commands. The root command reverts to standard Cobra behaviour with no custom arg handling or usage template.
|
||||
|
||||
## Impact
|
||||
|
||||
- `client/cmd/root.go` — remove `ArbitraryArgs`, `RunE` note logic, `--tags` flag, custom usage template, `isRootCmd` template func
|
||||
- `client/cmd/add.go` — `submitNote()` function moves to new `addnote.go` (or stays in `add.go` alongside `addfile` — design decision)
|
||||
- `client/cmd/addnote.go` — new file defining the `addnote` command
|
||||
- `client/cmd/examples.go` — update example text
|
||||
- `client/cmd/root_test.go` — remove implicit note shorthand tests, add standard Cobra behaviour tests
|
||||
- No engine changes — the API contract (`POST /api/v1/jobs` with `note` field) is unchanged
|
||||
@@ -0,0 +1,87 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Add note command
|
||||
|
||||
The client SHALL provide a `kb addnote <text>` command that submits a text note to the engine for ingestion. The command SHALL take exactly one positional argument (the note text) and support a `--tags` flag for comma-separated tags. The note SHALL be submitted via `POST /api/v1/jobs` with the `note` field in a multipart request.
|
||||
|
||||
#### Scenario: Add a note
|
||||
- **WHEN** the user runs `kb addnote "remember to update DNS records"`
|
||||
- **THEN** the client SHALL submit the text as a note via `POST /api/v1/jobs` and print `Queued: note`
|
||||
|
||||
#### Scenario: Add a note with tags
|
||||
- **WHEN** the user runs `kb addnote "server room is building 3" --tags ops`
|
||||
- **THEN** the client SHALL submit the note with the specified tags
|
||||
|
||||
#### Scenario: Add a note with JSON output
|
||||
- **WHEN** the user runs `kb addnote "my note" --format json`
|
||||
- **THEN** the client SHALL output the raw JSON response from the engine
|
||||
|
||||
#### Scenario: Duplicate note detection
|
||||
- **WHEN** the user runs `kb addnote "my note"` and the engine returns HTTP 409
|
||||
- **THEN** the client SHALL display the duplicate information (document ID or job ID) and exit with code 0
|
||||
|
||||
#### Scenario: Missing argument
|
||||
- **WHEN** the user runs `kb addnote` with no arguments
|
||||
- **THEN** the client SHALL display an error indicating that the note text argument is required
|
||||
|
||||
#### Scenario: Too many arguments
|
||||
- **WHEN** the user runs `kb addnote remember to update dns` (unquoted, multiple args)
|
||||
- **THEN** the client SHALL display an error indicating that exactly one argument is required, with a hint to quote the text
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Add command (file and note ingestion)
|
||||
|
||||
The client SHALL provide a `kb addfile` command that uploads files to the engine for async ingestion. The command SHALL validate file extensions before uploading and reject unsupported types. The client SHALL handle duplicate rejection (HTTP 409) and display the existing document information. Notes are handled by the separate `addnote` command — `addfile` is exclusively for file uploads.
|
||||
|
||||
#### Scenario: Add a single file
|
||||
- **WHEN** the user runs `kb addfile report.pdf`
|
||||
- **THEN** the client SHALL validate the file extension, upload the file via `POST /api/v1/jobs` (multipart), print "Queued: report.pdf", and exit
|
||||
|
||||
#### Scenario: Add a file with tags
|
||||
- **WHEN** the user runs `kb addfile manual.pdf --tags car,maintenance`
|
||||
- **THEN** the client SHALL include the tags in the multipart upload metadata
|
||||
|
||||
#### Scenario: Add a directory recursively
|
||||
- **WHEN** the user runs `kb addfile ~/documents/ --recursive`
|
||||
- **THEN** the client SHALL discover all supported files in the directory tree, upload each one sequentially, and print "Queued: N files"
|
||||
|
||||
#### Scenario: Unsupported file extension
|
||||
- **WHEN** the user runs `kb addfile photo.jpg`
|
||||
- **THEN** the client SHALL print an error listing supported extensions and exit with a non-zero code without making any API call
|
||||
|
||||
#### Scenario: Duplicate file rejected (already ingested)
|
||||
- **WHEN** the user runs `kb addfile report.pdf` and the engine returns HTTP 409 with `{"error": "duplicate", "document_id": 42, "title": "report.pdf"}`
|
||||
- **THEN** the client SHALL print "Already imported: report.pdf (doc ID: 42)" and exit with code 0
|
||||
|
||||
#### Scenario: Duplicate file rejected (in-flight job)
|
||||
- **WHEN** the user runs `kb addfile report.pdf` and the engine returns HTTP 409 with `{"error": "duplicate", "job_id": 7, "title": "report.pdf"}`
|
||||
- **THEN** the client SHALL print "Already queued: report.pdf (job ID: 7)" and exit with code 0
|
||||
|
||||
#### Scenario: Duplicate file in recursive add
|
||||
- **WHEN** the user runs `kb addfile ~/documents/ --recursive` and some files are rejected as duplicates
|
||||
- **THEN** the client SHALL print the duplicate message for each rejected file, continue uploading remaining files, and include a summary (e.g., "Queued: 5 files, 2 duplicates skipped")
|
||||
|
||||
#### Scenario: Duplicate with JSON output
|
||||
- **WHEN** the user runs `kb addfile report.pdf --format json` and the engine returns HTTP 409
|
||||
- **THEN** the client SHALL output the raw JSON response from the engine including the document_id and title
|
||||
|
||||
#### Scenario: Add with JSON output
|
||||
- **WHEN** the user runs `kb addfile report.pdf --format json`
|
||||
- **THEN** the client SHALL output the JSON response from the engine including the job_id
|
||||
|
||||
#### Scenario: File not found
|
||||
- **WHEN** the user runs `kb addfile nonexistent.pdf`
|
||||
- **THEN** the client SHALL print an error and exit with a non-zero code without making any API call
|
||||
|
||||
#### Scenario: Upload failure
|
||||
- **WHEN** the upload fails (network error, engine returns 4xx/5xx other than 409)
|
||||
- **THEN** the client SHALL print the error and exit with a non-zero code
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Implicit note shorthand
|
||||
|
||||
**Reason**: The implicit shorthand caused accidental note creation from mistyped commands. Any unrecognized multi-word input was silently ingested as a note. Replaced by the explicit `addnote` command.
|
||||
|
||||
**Migration**: Replace `kb "note text"` with `kb addnote "note text"`. Replace `kb "note text" --tags foo` with `kb addnote "note text" --tags foo`.
|
||||
@@ -0,0 +1,29 @@
|
||||
## 1. Create addnote command
|
||||
|
||||
- [x] 1.1 Create `client/cmd/addnote.go` with `addnoteCmd` using `ExactArgs(1)`, `--tags` flag, and `RunE` calling `submitNote()`
|
||||
- [x] 1.2 Move `submitNote()` function from `client/cmd/add.go` to `client/cmd/addnote.go`
|
||||
|
||||
## 2. Revert root command to standard Cobra behaviour
|
||||
|
||||
- [x] 2.1 Remove `ArbitraryArgs`, custom `RunE` logic, and `--tags` flag from root command in `client/cmd/root.go`
|
||||
- [x] 2.2 Remove custom usage template and `isRootCmd` template function — let Cobra use its default template
|
||||
- [x] 2.3 Set root command to show help when called with no args (standard Cobra `RunE` returning `cmd.Help()` or nil)
|
||||
|
||||
## 3. Update examples and help text
|
||||
|
||||
- [x] 3.1 Update `client/cmd/examples.go` to show `kb addnote` syntax instead of `kb "text"` shorthand
|
||||
- [x] 3.2 Update root command `Long` description to remove reference to note shorthand
|
||||
|
||||
## 4. Update tests
|
||||
|
||||
- [x] 4.1 Remove implicit note shorthand tests from `client/cmd/root_test.go` (`TestRootCmd_SingleWordRejected`, `TestRootCmd_MultipleWordsNotRejected`)
|
||||
- [x] 4.2 Add test for `addnote` command (verify it wires up correctly, takes exactly one arg)
|
||||
- [x] 4.3 Add test that root command with unknown args returns an error (standard Cobra behaviour)
|
||||
- [x] 4.4 Verify `addfile` tests still pass (no changes expected)
|
||||
|
||||
## 5. Build and verify
|
||||
|
||||
- [x] 5.1 Run `go build` and verify all commands appear in `kb --help`
|
||||
- [x] 5.2 Run `go test ./...` and verify all tests pass
|
||||
- [x] 5.3 Verify `kb addnote --help` shows correct usage line and flags
|
||||
- [x] 5.4 Verify `kb addfile --help` is unchanged
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-04-02
|
||||
@@ -0,0 +1,41 @@
|
||||
## Context
|
||||
|
||||
The engine's `/api/v1/search` endpoint returns flat result objects:
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_id": 123,
|
||||
"score": 0.031,
|
||||
"text": "...",
|
||||
"chunk_index": 3,
|
||||
"chunk_metadata": {"page": 12, "section_header": "Installation"},
|
||||
"title": "Git Admin Guide",
|
||||
"doc_type": "pdf",
|
||||
"source_path": "/home/user/docs/git-admin.pdf",
|
||||
"created_at": "2026-03-15T10:30:00",
|
||||
"tags": ["git", "admin"]
|
||||
}
|
||||
```
|
||||
|
||||
The Go client's human-mode struct in `client/cmd/search.go` incorrectly expects a nested `document` object and top-level `page`/`section` fields. This causes all metadata to display as zero values.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Fix the search result struct to match the flat engine response
|
||||
- Extract `page` and `section_header` from `chunk_metadata` for human display
|
||||
- Maintain identical JSON output (already passes through raw response)
|
||||
|
||||
**Non-Goals:**
|
||||
- Changing the engine API response format
|
||||
- Adding new display fields beyond what was originally intended
|
||||
|
||||
## Decisions
|
||||
|
||||
**Flatten the struct to match API response.** The result struct will have `Title`, `DocType`, `Tags` as top-level fields (matching `title`, `doc_type`, `tags` JSON keys). `ChunkMetadata` will be decoded as `map[string]interface{}` to extract `page` and `section_header` dynamically, since its contents vary by document type.
|
||||
|
||||
**Why not a typed ChunkMetadata struct?** The metadata keys depend on the ingestion pipeline (PDFs have `page`, markdown has `section_header`, code may have others in future). A map is more resilient to engine-side additions.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Minimal risk] If the engine adds new top-level fields, the Go struct silently ignores them — this is existing behavior and acceptable for human-mode display.
|
||||
@@ -0,0 +1,24 @@
|
||||
## Why
|
||||
|
||||
The Go client's human-mode search output struct expects a nested `document` object and top-level `page`/`section` fields, but the engine API returns flat results with `title`, `doc_type`, `tags` at the result level and `page`/`section_header` inside `chunk_metadata`. This means human-mode display shows empty values for title, type, tags, page, and section.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Fix the Go client search result struct to match the flat engine API response format
|
||||
- Extract `page` and `section_header` from the `chunk_metadata` map instead of expecting them as top-level fields
|
||||
- Human-mode output will correctly display document title, type, tags, page number, and section header
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
(none)
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `go-client`: Fix search result parsing to match actual engine API response shape
|
||||
|
||||
## Impact
|
||||
|
||||
- `client/cmd/search.go` — struct definition and display logic
|
||||
- No API changes, no breaking changes — this is a bug fix aligning the client with the existing API contract
|
||||
@@ -0,0 +1,40 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Search command
|
||||
|
||||
The client SHALL provide a `kb search <query>` command that sends the query to the engine and displays results.
|
||||
|
||||
#### Scenario: Human-readable search output
|
||||
- **WHEN** the user runs `kb search "how to change oil"`
|
||||
- **THEN** the client SHALL POST to `/api/v1/search`, and display results in a human-readable format showing rank, score, document title, page/section, doc type, tags, and a text snippet
|
||||
- **THEN** the client SHALL parse search results as flat objects with top-level `title`, `doc_type`, `tags`, `score`, `text`, `chunk_index` fields
|
||||
- **THEN** the client SHALL extract `page` from `chunk_metadata` when present (PDF documents)
|
||||
- **THEN** the client SHALL extract `section_header` from `chunk_metadata` when present (markdown documents)
|
||||
|
||||
#### Scenario: JSON search output
|
||||
- **WHEN** the user runs `kb search "query" --format json`
|
||||
- **THEN** the client SHALL output the raw JSON response from the engine
|
||||
|
||||
#### Scenario: Search with filters
|
||||
- **WHEN** the user runs `kb search "brakes" --tags maintenance --type pdf --top 3`
|
||||
- **THEN** the client SHALL include the filters in the API request body
|
||||
|
||||
#### Scenario: Search mode flags
|
||||
- **WHEN** the user runs `kb search "error" --fts-only`
|
||||
- **THEN** the client SHALL set `fts_only: true` in the request body
|
||||
|
||||
#### Scenario: PDF result with page number
|
||||
- **WHEN** a search result has `chunk_metadata` containing `{"page": 12}`
|
||||
- **THEN** the human output SHALL display "Page 12" in the location line
|
||||
|
||||
#### Scenario: Markdown result with section header
|
||||
- **WHEN** a search result has `chunk_metadata` containing `{"section_header": "Installation > Prerequisites"}`
|
||||
- **THEN** the human output SHALL display "Installation > Prerequisites" in the location line
|
||||
|
||||
#### Scenario: Result with both page and section
|
||||
- **WHEN** a search result has `chunk_metadata` containing both `page` and `section_header`
|
||||
- **THEN** the human output SHALL display both separated by " / "
|
||||
|
||||
#### Scenario: Result with no location metadata
|
||||
- **WHEN** a search result has empty `chunk_metadata` or no page/section keys
|
||||
- **THEN** the human output SHALL omit the location line entirely
|
||||
@@ -0,0 +1,14 @@
|
||||
## 1. Fix search result struct
|
||||
|
||||
- [x] 1.1 Replace nested `Document` struct with flat fields (`Title`, `DocType`, `Tags`) matching engine JSON keys
|
||||
- [x] 1.2 Add `ChunkMetadata map[string]interface{}` field to capture `chunk_metadata`
|
||||
|
||||
## 2. Fix display logic
|
||||
|
||||
- [x] 2.1 Update title/type/tags references in the display loop to use the new flat fields
|
||||
- [x] 2.2 Extract `page` from `ChunkMetadata` map (replacing top-level `Page` field)
|
||||
- [x] 2.3 Extract `section_header` from `ChunkMetadata` map (replacing top-level `Section` field)
|
||||
|
||||
## 3. Verify
|
||||
|
||||
- [x] 3.1 Build the client and verify it compiles cleanly
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-04-04
|
||||
@@ -0,0 +1,194 @@
|
||||
## Context
|
||||
|
||||
The engine API (`engine/kb/routes/`) provides single-document operations for delete (`DELETE /api/v1/documents/{id}`) and tag management (`PUT /api/v1/documents/{id}/tags`). The MCP server (`mcp/server.py`) wraps these and adds a "collection" abstraction via `collection:`-prefixed tags — ~70 lines of helpers and translation logic that only the MCP layer understands.
|
||||
|
||||
The database is SQLite with WAL mode, FTS5 for full-text search, and sqlite-vec for embeddings. Foreign keys with `ON DELETE CASCADE` handle chunk cleanup when documents are deleted. Stored files on disk must be cleaned up separately.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Bulk delete, bulk tag add/remove, and bulk set-tags (replace) via engine API, MCP tools, and CLI
|
||||
- Filter-based selection: by tag, doc_type, ID list, and ID range
|
||||
- Safety threshold to prevent accidental mass operations
|
||||
- Audit trail via jobs table
|
||||
- Remove collection abstraction from MCP server
|
||||
|
||||
**Non-Goals:**
|
||||
- Async/queued bulk operations (SQLite handles thousands of rows synchronously in <1s)
|
||||
- Bulk document retrieval or bulk note creation
|
||||
- Undo/recycle bin for bulk deletes
|
||||
- Adding collection concept to engine or CLI (collections are being removed, not moved)
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Common selection filter for all bulk endpoints
|
||||
|
||||
All three bulk endpoints accept the same selection body:
|
||||
|
||||
```json
|
||||
{
|
||||
"document_ids": [1, 5, 12],
|
||||
"tags": ["agent:mybot", "draft"],
|
||||
"doc_type": "note",
|
||||
"from_id": 10,
|
||||
"to_id": 50
|
||||
}
|
||||
```
|
||||
|
||||
Filters combine with AND logic. At least one filter is required — the engine rejects requests with no selection criteria (400).
|
||||
|
||||
**Selection SQL generation**: A shared helper in `database.py` builds the WHERE clause from the filter. The `tags` filter uses the same JOIN pattern as `list_documents` (all specified tags must match). The `document_ids` filter uses `IN (?)`. The `from_id`/`to_id` filter uses `id >= ? AND id <= ?`.
|
||||
|
||||
**Alternative considered**: Separate endpoints per filter type. Rejected — combinable filters are more powerful and the SQL generation is straightforward.
|
||||
|
||||
### 2. Safety threshold with configurable percentage
|
||||
|
||||
Before executing, the engine counts matched documents and total documents. If `matched / total > threshold`, the request is rejected:
|
||||
|
||||
```
|
||||
HTTP 409 Conflict
|
||||
{
|
||||
"error": "safety_threshold_exceeded",
|
||||
"message": "Operation would affect 750 of 1000 documents (75.0%). Exceeds safety threshold of 70%. Use force: true to proceed.",
|
||||
"matched": 750,
|
||||
"total": 1000,
|
||||
"percent": 75.0,
|
||||
"threshold": 70
|
||||
}
|
||||
```
|
||||
|
||||
- Default threshold: 70% (env var `KB_BULK_SAFETY_PERCENT`, integer 0-100)
|
||||
- Override per-request: `"force": true` in the request body
|
||||
- Threshold of 0 effectively disables the safety check
|
||||
- CLI maps this to `--force` / `-f` flag
|
||||
|
||||
The check is a SELECT COUNT before the operation — minimal overhead.
|
||||
|
||||
**Alternative considered**: Dry-run mode (preview what would be affected, then confirm). Rejected — adds a two-step flow that doesn't help LLM callers (they'd just always confirm) and the safety threshold covers the dangerous case.
|
||||
|
||||
### 3. Synchronous execution with audit logging
|
||||
|
||||
Bulk operations execute synchronously and return a summary response:
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": 42,
|
||||
"status": "done",
|
||||
"matched": 750,
|
||||
"succeeded": 748,
|
||||
"failed": 2,
|
||||
"errors": [
|
||||
{"document_id": 42, "error": "file locked"},
|
||||
{"document_id": 99, "error": "not found"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A job record is created in the `jobs` table with a new `bulk_delete` / `bulk_tags` / `bulk_set_tags` status type. This requires extending the jobs table:
|
||||
|
||||
- Add `job_type` column: `"ingest"` (default, for existing jobs) or `"bulk_delete"` / `"bulk_tags"` / `"bulk_set_tags"`
|
||||
- The job's `filename` field stores a JSON summary of the selection filter for auditability
|
||||
- `document_id` field stores the count of affected documents
|
||||
- `error` field stores JSON array of individual errors if any
|
||||
|
||||
**Alternative considered**: Full async with job polling. Rejected — SQLite bulk operations are fast enough synchronously and async would require extra polling calls (defeating the purpose of reducing token usage).
|
||||
|
||||
### 4. Bulk delete implementation
|
||||
|
||||
For each matched document:
|
||||
1. Collect chunk IDs
|
||||
2. Delete embeddings from `chunks_vec`
|
||||
3. Delete the document row (cascades to chunks, document_tags)
|
||||
4. Delete stored file from disk
|
||||
|
||||
This follows the same logic as the existing `delete_document` endpoint but batched in a single transaction (except file deletion, which happens after commit). If a file deletion fails, the document is still counted as succeeded (the DB record is gone) but a warning is logged.
|
||||
|
||||
The operation processes documents within a single SQLite transaction for atomicity of the DB changes. File deletions happen post-commit and are best-effort.
|
||||
|
||||
### 5. Bulk tags implementation
|
||||
|
||||
Two distinct operations:
|
||||
|
||||
**`POST /api/v1/bulk/tags`** — Add and/or remove tags:
|
||||
```json
|
||||
{
|
||||
"add": ["reviewed", "approved"],
|
||||
"remove": ["draft"],
|
||||
...selection filters...
|
||||
}
|
||||
```
|
||||
|
||||
**`POST /api/v1/bulk/set-tags`** — Replace all tags:
|
||||
```json
|
||||
{
|
||||
"tags": ["final", "approved"],
|
||||
...selection filters...
|
||||
}
|
||||
```
|
||||
|
||||
The `set-tags` operation removes all existing tags from matched documents, then applies the new set. This is useful for cleaning up tag clutter or migrating tagging schemes.
|
||||
|
||||
Both update `updated_at` on affected documents.
|
||||
|
||||
### 6. Remove collection abstraction from MCP
|
||||
|
||||
Remove from `mcp/server.py`:
|
||||
- Constants: `COLLECTION_TAG_PREFIX`, `DEFAULT_COLLECTION`
|
||||
- Functions: `_collection_tag`, `_strip_collection_tags`, `_process_document`, `_process_search_results`, `_ensure_exclusive_collection`
|
||||
- Tool: `kb_set_collection` (entire tool removed)
|
||||
- Parameters: `collection` from `kb_search`, `kb_addnote`, `kb_upload_start`
|
||||
|
||||
The `_process_document` and `_process_search_results` calls in remaining tools are removed — documents are returned as-is from the engine, with all tags visible.
|
||||
|
||||
Users/agents that need namespace isolation use a tag convention (e.g. `agent:claude-code`) communicated via system prompt or tool instructions.
|
||||
|
||||
### 7. Engine bulk route module
|
||||
|
||||
New file: `engine/kb/routes/bulk.py`
|
||||
|
||||
Three endpoints sharing common infrastructure:
|
||||
- `_resolve_selection(conn, filters)` → list of document IDs + count
|
||||
- `_check_safety_threshold(matched, total, force)` → raises HTTPException if exceeded
|
||||
- `_log_bulk_job(conn, job_type, filters, matched, succeeded, failed, errors)` → job_id
|
||||
|
||||
### 8. MCP bulk tools
|
||||
|
||||
Three new tools in `mcp/server.py`, thin wrappers calling new `engine.py` methods:
|
||||
|
||||
- `kb_bulk_delete(document_ids?, tags?, doc_type?, from_id?, to_id?, force?)` → str (JSON)
|
||||
- `kb_bulk_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, add?, remove?, force?)` → str (JSON)
|
||||
- `kb_bulk_set_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, new_tags?, force?)` → str (JSON)
|
||||
|
||||
Note: The `tags` parameter on bulk tools serves as a **selection filter** (which documents to target), while `add`/`remove` (on bulk_tags) and `new_tags` (on bulk_set_tags) are the **operation** (what to do to the tags). Tool descriptions must make this distinction clear.
|
||||
|
||||
### 9. CLI bulk commands
|
||||
|
||||
Three new commands under `client/cmd/`:
|
||||
|
||||
```
|
||||
kb bulk-remove --tags "draft,old" --type note --force --yes
|
||||
kb bulk-tag --tags "agent:mybot" --add "reviewed" --remove "pending" --yes
|
||||
kb bulk-set-tags --ids "1,5,12" --tags "clean,final" --yes
|
||||
```
|
||||
|
||||
Filter flags (shared): `--tags`, `--type`, `--ids` (comma-separated), `--from-id`, `--to-id`, `--force`
|
||||
Confirmation: `--yes` / `-y` to skip interactive prompt.
|
||||
|
||||
Without `--yes`, the CLI first shows the match count and asks for confirmation:
|
||||
|
||||
```
|
||||
This will delete 47 documents matching: tags=[draft,old] type=note
|
||||
Proceed? [y/N]
|
||||
```
|
||||
|
||||
### 10. Engine config for safety threshold
|
||||
|
||||
New env var: `KB_BULK_SAFETY_PERCENT` (integer, default 70). Added to `engine/kb/config.py`.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Bulk delete is irreversible]** → Safety threshold mitigates accidental mass deletion. CLI requires interactive confirmation. No undo mechanism — this is deliberate to keep the system simple.
|
||||
- **[Naming collision: `tags` as filter vs operation]** → The `tags` parameter in bulk_tags selects documents, while `add`/`remove` specifies the tag changes. Clear naming and tool descriptions mitigate confusion. Engine request model uses the same field name as the existing list/search filter.
|
||||
- **[SQLite lock during large bulk ops]** → A single transaction deleting 5000 documents will hold a write lock. With WAL mode, readers are not blocked. The lock duration should be under a few seconds for typical workloads.
|
||||
- **[Breaking change: collection removal]** → Any MCP client relying on `collection` parameters will break. Since collections were only recently added and are not widely deployed, this is acceptable. Existing `collection:*` tags in the database remain as regular tags — they still work as filters, just without special treatment.
|
||||
- **[Jobs table overload]** → Bulk operations add a new job type to a table designed for ingestion jobs. The schema change is minimal (one new column) and the audit trail value outweighs the mixing of concerns.
|
||||
@@ -0,0 +1,91 @@
|
||||
## Why
|
||||
|
||||
Bulk operations on documents (delete, tag, retag) currently require one API/MCP call per document. When an LLM manages hundreds or thousands of documents, this means hundreds of tool calls — burning tokens, adding latency, and creating fragile multi-step flows that can fail partway through.
|
||||
|
||||
Additionally, the "collection" abstraction in the MCP server adds complexity without real benefit. Collections are implemented as `collection:`-prefixed tags, but this convention is only enforced in the MCP layer — the CLI and engine don't know about it. This creates inconsistency and extra code. Tags alone, with a naming convention communicated via system prompt or configuration, achieve the same namespace isolation more simply and uniformly.
|
||||
|
||||
## What Changes
|
||||
|
||||
### 1. Remove collections from MCP server
|
||||
|
||||
Strip all collection logic from `mcp/server.py`:
|
||||
- Remove `COLLECTION_TAG_PREFIX`, `DEFAULT_COLLECTION`, and all collection helper functions
|
||||
- Remove `collection` parameter from `kb_search`, `kb_addnote`, `kb_upload_start`
|
||||
- Remove `kb_set_collection` tool entirely
|
||||
- Remove `_process_document` / `_process_search_results` collection-tag stripping
|
||||
- Update MCP server instructions to explain tag-based namespace convention
|
||||
|
||||
### 2. Add bulk engine endpoints
|
||||
|
||||
Three new endpoints in the engine API:
|
||||
|
||||
- **POST /api/v1/bulk/delete** — Delete multiple documents matching a filter
|
||||
- **POST /api/v1/bulk/tags** — Add/remove tags on multiple documents matching a filter
|
||||
- **POST /api/v1/bulk/set-tags** — Replace all tags on multiple documents matching a filter
|
||||
|
||||
All accept a common **selection filter** (combinable with AND logic):
|
||||
- `document_ids` — explicit list of IDs
|
||||
- `tags` — documents matching ALL specified tags
|
||||
- `doc_type` — documents of this type
|
||||
- `from_id` / `to_id` — ID range (inclusive)
|
||||
|
||||
At least one selection criterion is required.
|
||||
|
||||
**Safety threshold**: If the operation would affect more than N% of all documents (default 70%, configurable via `KB_BULK_SAFETY_PERCENT` env var), the request is rejected with a 409 response showing what would be affected. The caller must re-send with `force: true` to proceed.
|
||||
|
||||
**Response model**: Synchronous execution with summary response. The operation is logged to the jobs table for audit trail:
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": 42,
|
||||
"status": "done",
|
||||
"matched": 750,
|
||||
"succeeded": 748,
|
||||
"failed": 2,
|
||||
"errors": [
|
||||
{"document_id": 42, "error": "file locked"},
|
||||
{"document_id": 99, "error": "not found"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Add bulk MCP tools
|
||||
|
||||
Expose the bulk engine endpoints as MCP tools:
|
||||
- `kb_bulk_delete` — bulk delete with filter selection
|
||||
- `kb_bulk_tags` — bulk add/remove tags with filter selection
|
||||
- `kb_bulk_set_tags` — bulk replace tags with filter selection
|
||||
|
||||
These are thin wrappers around the engine bulk endpoints — no collection translation, no special logic.
|
||||
|
||||
### 4. Add bulk CLI commands
|
||||
|
||||
- `kb bulk-remove` — bulk delete with `--tags`, `--type`, `--ids`, `--from-id`, `--to-id`, `--force` flags
|
||||
- `kb bulk-tag` — bulk tag/untag with `--add`, `--remove`, and the same filter flags
|
||||
- `kb bulk-set-tags` — bulk replace tags with `--tags` (new tags) and the same filter flags
|
||||
|
||||
All show a confirmation prompt with match count before executing (unless `--yes`).
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `bulk-operations`: Engine endpoints, MCP tools, and CLI commands for bulk delete, tag, and set-tags operations with filter-based selection and safety threshold.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `mcp-document-management`: Remove `kb_set_collection` tool. Remove `collection` parameter from all tools.
|
||||
|
||||
### Removed Capabilities
|
||||
|
||||
- `mcp-collections`: The collection abstraction (collection helpers, collection parameters, collection tag stripping) is removed from the MCP server entirely.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Engine API** (`engine/kb/routes/`): New `bulk.py` route module with 3 endpoints. New `bulk` job type in jobs table.
|
||||
- **Engine database** (`engine/kb/database.py`): Helper functions for bulk selection queries and bulk delete/tag operations.
|
||||
- **MCP server** (`mcp/server.py`): Remove ~70 lines of collection logic. Add 3 bulk tool definitions. Remove `collection` param from `kb_search`, `kb_addnote`, `kb_upload_start`. Remove `kb_set_collection`.
|
||||
- **MCP engine client** (`mcp/engine.py`): Add bulk operation methods. Remove no longer needed code.
|
||||
- **CLI** (`client/cmd/`): New `bulk_remove.go`, `bulk_tag.go`, `bulk_set_tags.go` command files.
|
||||
- **CLI API client** (`client/internal/api/`): Add `Post` with JSON body support if not present.
|
||||
- **Breaking changes**: `kb_set_collection` MCP tool removed. `collection` parameter removed from `kb_search`, `kb_addnote`, `kb_upload_start` MCP tools. Any MCP clients using collections will need to switch to tags.
|
||||
+230
@@ -0,0 +1,230 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Common selection filter
|
||||
|
||||
All bulk engine endpoints SHALL accept a JSON body with the following optional selection fields, combined with AND logic:
|
||||
|
||||
- `document_ids` (list of int) — match documents with these specific IDs
|
||||
- `tags` (list of str) — match documents that have ALL specified tags
|
||||
- `doc_type` (str) — match documents with this document type
|
||||
- `from_id` (int) — match documents with id >= this value
|
||||
- `to_id` (int) — match documents with id <= this value
|
||||
|
||||
At least one selection field MUST be present. If no selection fields are provided, the endpoint SHALL return 400 Bad Request.
|
||||
|
||||
#### Scenario: Filter by tags and doc_type
|
||||
|
||||
- **WHEN** a bulk endpoint receives `{"tags": ["draft"], "doc_type": "note"}`
|
||||
- **THEN** it SHALL match only documents that have the tag "draft" AND have doc_type "note"
|
||||
|
||||
#### Scenario: Filter by ID range
|
||||
|
||||
- **WHEN** a bulk endpoint receives `{"from_id": 10, "to_id": 50}`
|
||||
- **THEN** it SHALL match documents with id >= 10 AND id <= 50
|
||||
|
||||
#### Scenario: Filter by explicit IDs
|
||||
|
||||
- **WHEN** a bulk endpoint receives `{"document_ids": [1, 5, 12]}`
|
||||
- **THEN** it SHALL match only documents with those specific IDs
|
||||
|
||||
#### Scenario: Combined filters
|
||||
|
||||
- **WHEN** a bulk endpoint receives `{"tags": ["agent:mybot"], "doc_type": "note", "from_id": 100}`
|
||||
- **THEN** it SHALL match documents satisfying ALL three criteria
|
||||
|
||||
#### Scenario: No selection fields provided
|
||||
|
||||
- **WHEN** a bulk endpoint receives `{}` or `{"force": true}` with no selection fields
|
||||
- **THEN** it SHALL return 400 Bad Request
|
||||
|
||||
### Requirement: Safety threshold
|
||||
|
||||
All bulk endpoints SHALL enforce a safety threshold. Before executing, the engine SHALL count the matched documents and the total documents in the database. If `matched / total * 100` exceeds the configured threshold, the request SHALL be rejected with 409 Conflict.
|
||||
|
||||
The response SHALL include: `error` ("safety_threshold_exceeded"), `message` (human-readable), `matched` (int), `total` (int), `percent` (float), and `threshold` (int).
|
||||
|
||||
The threshold SHALL default to 70 and be configurable via the `KB_BULK_SAFETY_PERCENT` environment variable (integer 0-100). A value of 0 disables the check.
|
||||
|
||||
The caller MAY override the threshold by including `"force": true` in the request body.
|
||||
|
||||
#### Scenario: Threshold exceeded
|
||||
|
||||
- **GIVEN** 1000 total documents and `KB_BULK_SAFETY_PERCENT` is 70
|
||||
- **WHEN** a bulk endpoint matches 750 documents (75%) without `force: true`
|
||||
- **THEN** it SHALL return 409 with `matched: 750`, `total: 1000`, `percent: 75.0`, `threshold: 70`
|
||||
|
||||
#### Scenario: Threshold not exceeded
|
||||
|
||||
- **GIVEN** 1000 total documents and `KB_BULK_SAFETY_PERCENT` is 70
|
||||
- **WHEN** a bulk endpoint matches 500 documents (50%) without `force: true`
|
||||
- **THEN** the operation SHALL proceed normally
|
||||
|
||||
#### Scenario: Force override
|
||||
|
||||
- **GIVEN** 1000 total documents and a match of 900 (90%)
|
||||
- **WHEN** the request includes `"force": true`
|
||||
- **THEN** the operation SHALL proceed regardless of threshold
|
||||
|
||||
#### Scenario: Zero threshold
|
||||
|
||||
- **GIVEN** `KB_BULK_SAFETY_PERCENT` is 0
|
||||
- **THEN** the safety check SHALL be effectively disabled for all operations
|
||||
|
||||
### Requirement: Synchronous response with audit log
|
||||
|
||||
All bulk endpoints SHALL execute synchronously and return a JSON response with:
|
||||
|
||||
- `job_id` (int) — ID of the audit log entry in the jobs table
|
||||
- `status` (str) — "done" or "partial_failure"
|
||||
- `matched` (int) — number of documents that matched the selection
|
||||
- `succeeded` (int) — number of documents successfully processed
|
||||
- `failed` (int) — number of documents that failed
|
||||
- `errors` (list) — array of `{"document_id": int, "error": str}` for each failure (empty on full success)
|
||||
|
||||
A job record SHALL be created in the jobs table with `job_type` set to the operation type. The `filename` field SHALL store a JSON representation of the selection filter. The `error` field SHALL store a JSON array of individual errors if any occurred.
|
||||
|
||||
#### Scenario: Full success
|
||||
|
||||
- **WHEN** a bulk operation matches 50 documents and all succeed
|
||||
- **THEN** the response SHALL have `status: "done"`, `matched: 50`, `succeeded: 50`, `failed: 0`, `errors: []`
|
||||
|
||||
#### Scenario: Partial failure
|
||||
|
||||
- **WHEN** a bulk operation matches 50 documents but 2 fail
|
||||
- **THEN** the response SHALL have `status: "partial_failure"`, `matched: 50`, `succeeded: 48`, `failed: 2`, and `errors` listing the 2 failures
|
||||
|
||||
### Requirement: Bulk delete endpoint
|
||||
|
||||
The engine SHALL expose `POST /api/v1/bulk/delete` which permanently deletes all documents matching the selection filter. For each matched document, it SHALL delete embeddings from `chunks_vec`, delete the document row (cascading to chunks and document_tags), and delete any stored file from disk.
|
||||
|
||||
Database deletions SHALL be performed within a single transaction. File deletions SHALL occur after the transaction commits and SHALL be best-effort (failures logged but not counted as document failures).
|
||||
|
||||
#### Scenario: Bulk delete by tag
|
||||
|
||||
- **WHEN** `POST /api/v1/bulk/delete` receives `{"tags": ["old", "draft"]}`
|
||||
- **THEN** all documents with both tags "old" and "draft" SHALL be deleted
|
||||
- **AND** their chunks, embeddings, tag associations, and stored files SHALL be removed
|
||||
|
||||
#### Scenario: Bulk delete with no matches
|
||||
|
||||
- **WHEN** `POST /api/v1/bulk/delete` receives a filter that matches 0 documents
|
||||
- **THEN** the response SHALL have `matched: 0`, `succeeded: 0`, `failed: 0`
|
||||
|
||||
### Requirement: Bulk tags endpoint
|
||||
|
||||
The engine SHALL expose `POST /api/v1/bulk/tags` which adds and/or removes tags on all documents matching the selection filter. The request body SHALL include the selection filter plus:
|
||||
|
||||
- `add` (list of str, optional) — tags to add
|
||||
- `remove` (list of str, optional) — tags to remove
|
||||
|
||||
At least one of `add` or `remove` MUST be present. The endpoint SHALL return 400 if neither is provided.
|
||||
|
||||
The endpoint SHALL update `updated_at` on all affected documents.
|
||||
|
||||
#### Scenario: Add and remove tags in one call
|
||||
|
||||
- **WHEN** `POST /api/v1/bulk/tags` receives `{"tags": ["agent:mybot"], "add": ["reviewed"], "remove": ["pending"]}`
|
||||
- **THEN** all documents tagged "agent:mybot" SHALL have "reviewed" added and "pending" removed
|
||||
|
||||
### Requirement: Bulk set-tags endpoint
|
||||
|
||||
The engine SHALL expose `POST /api/v1/bulk/set-tags` which replaces all tags on matched documents with a new set. The request body SHALL include the selection filter plus:
|
||||
|
||||
- `new_tags` (list of str) — the replacement tag set
|
||||
|
||||
The endpoint SHALL remove all existing tag associations from matched documents, then apply the new set. It SHALL update `updated_at` on all affected documents.
|
||||
|
||||
#### Scenario: Replace all tags
|
||||
|
||||
- **WHEN** `POST /api/v1/bulk/set-tags` receives `{"doc_type": "note", "new_tags": ["clean", "final"]}`
|
||||
- **THEN** all notes SHALL have their existing tags removed and replaced with "clean" and "final"
|
||||
|
||||
### Requirement: Jobs table extension
|
||||
|
||||
The jobs table SHALL be extended with a `job_type` column (TEXT, default "ingest") to distinguish ingestion jobs from bulk operation audit entries. Valid values: "ingest", "bulk_delete", "bulk_tags", "bulk_set_tags".
|
||||
|
||||
Existing jobs SHALL default to `job_type = "ingest"`. The existing jobs list endpoint and CLI `kb jobs` command SHALL continue to work unchanged.
|
||||
|
||||
#### Scenario: Migration adds column
|
||||
|
||||
- **GIVEN** an existing database without the `job_type` column
|
||||
- **WHEN** the engine starts
|
||||
- **THEN** the column SHALL be added with default value "ingest"
|
||||
|
||||
### Requirement: Engine config for safety threshold
|
||||
|
||||
The engine `Config` class SHALL read `KB_BULK_SAFETY_PERCENT` from the environment as an integer (default 70, range 0-100). This value SHALL be used as the default safety threshold for all bulk endpoints.
|
||||
|
||||
### Requirement: MCP bulk delete tool
|
||||
|
||||
The MCP server SHALL expose a `kb_bulk_delete` tool with parameters: `document_ids` (optional list of int), `tags` (optional list of str), `doc_type` (optional str), `from_id` (optional int), `to_id` (optional int), `force` (optional bool).
|
||||
|
||||
The tool SHALL call `POST /api/v1/bulk/delete` on the engine via the engine client and return the JSON response.
|
||||
|
||||
The tool description SHALL clearly state that `tags` is a selection filter (which documents to delete), not tags to delete.
|
||||
|
||||
#### Scenario: MCP bulk delete by tag
|
||||
|
||||
- **WHEN** `kb_bulk_delete(tags=["old"])` is called
|
||||
- **THEN** the engine client SHALL send `POST /api/v1/bulk/delete` with `{"tags": ["old"]}`
|
||||
- **AND** the tool SHALL return the engine's JSON response
|
||||
|
||||
### Requirement: MCP bulk tags tool
|
||||
|
||||
The MCP server SHALL expose a `kb_bulk_tags` tool with parameters: `document_ids`, `tags`, `doc_type`, `from_id`, `to_id` (selection filters), plus `add` (optional list of str), `remove` (optional list of str), and `force` (optional bool).
|
||||
|
||||
The tool description SHALL clearly distinguish `tags` (selection filter) from `add`/`remove` (tag changes to apply).
|
||||
|
||||
#### Scenario: MCP bulk tag update
|
||||
|
||||
- **WHEN** `kb_bulk_tags(tags=["agent:mybot"], add=["reviewed"], remove=["draft"])` is called
|
||||
- **THEN** the engine client SHALL send the appropriate `POST /api/v1/bulk/tags` request
|
||||
|
||||
### Requirement: MCP bulk set-tags tool
|
||||
|
||||
The MCP server SHALL expose a `kb_bulk_set_tags` tool with parameters: `document_ids`, `tags`, `doc_type`, `from_id`, `to_id` (selection filters), plus `new_tags` (list of str) and `force` (optional bool).
|
||||
|
||||
#### Scenario: MCP bulk set tags
|
||||
|
||||
- **WHEN** `kb_bulk_set_tags(doc_type="note", new_tags=["clean"])` is called
|
||||
- **THEN** the engine client SHALL send `POST /api/v1/bulk/set-tags` with `{"doc_type": "note", "new_tags": ["clean"]}`
|
||||
|
||||
### Requirement: MCP engine client bulk methods
|
||||
|
||||
The MCP engine client (`mcp/engine.py`) SHALL provide three new methods:
|
||||
|
||||
- `bulk_delete(document_ids?, tags?, doc_type?, from_id?, to_id?, force?)` → dict
|
||||
- `bulk_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, add?, remove?, force?)` → dict
|
||||
- `bulk_set_tags(document_ids?, tags?, doc_type?, from_id?, to_id?, new_tags?, force?)` → dict
|
||||
|
||||
Each SHALL send a POST request to the corresponding `/api/v1/bulk/*` endpoint with the parameters as a JSON body. Each SHALL raise on non-2xx status codes, consistent with existing methods.
|
||||
|
||||
### Requirement: CLI bulk-remove command
|
||||
|
||||
The CLI SHALL expose a `kb bulk-remove` command with flags: `--tags` (comma-separated), `--type`, `--ids` (comma-separated), `--from-id`, `--to-id`, `--force`/`-f`, `--yes`/`-y`.
|
||||
|
||||
Without `--yes`, the CLI SHALL first display the match count and ask for interactive confirmation before proceeding.
|
||||
|
||||
The command SHALL call `POST /api/v1/bulk/delete` with the constructed filter.
|
||||
|
||||
#### Scenario: CLI bulk remove with confirmation
|
||||
|
||||
- **WHEN** `kb bulk-remove --tags "draft,old" --type note` is run without `--yes`
|
||||
- **THEN** the CLI SHALL display "This will delete N documents matching: tags=[draft,old] type=note" and prompt "Proceed? [y/N]"
|
||||
|
||||
#### Scenario: CLI bulk remove with --yes
|
||||
|
||||
- **WHEN** `kb bulk-remove --tags "draft" --yes` is run
|
||||
- **THEN** the CLI SHALL proceed without prompting
|
||||
|
||||
### Requirement: CLI bulk-tag command
|
||||
|
||||
The CLI SHALL expose a `kb bulk-tag` command with the same filter flags as `bulk-remove`, plus `--add` and `--remove` (comma-separated tag lists).
|
||||
|
||||
The command SHALL call `POST /api/v1/bulk/tags` with the constructed filter and tag changes.
|
||||
|
||||
### Requirement: CLI bulk-set-tags command
|
||||
|
||||
The CLI SHALL expose a `kb bulk-set-tags` command with the filter flags, plus `--set` (comma-separated list of replacement tags).
|
||||
|
||||
The command SHALL call `POST /api/v1/bulk/set-tags` with the constructed filter and `new_tags`.
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Collection abstraction in MCP server
|
||||
|
||||
The MCP server SHALL NOT maintain any collection abstraction. The following SHALL be removed:
|
||||
|
||||
- Constants: `COLLECTION_TAG_PREFIX`, `DEFAULT_COLLECTION`
|
||||
- Functions: `_collection_tag`, `_strip_collection_tags`, `_process_document`, `_process_search_results`, `_ensure_exclusive_collection`
|
||||
- Tool: `kb_set_collection` (entire tool)
|
||||
- Parameters: `collection` from `kb_search`, `kb_addnote`, `kb_upload_start`
|
||||
|
||||
Documents SHALL be returned as-is from the engine with all tags visible. No tag stripping or collection field injection SHALL occur.
|
||||
|
||||
#### Scenario: Search results show all tags
|
||||
|
||||
- **WHEN** `kb_search` is called and a result has tags `["agent:mybot", "collection:documents", "draft"]`
|
||||
- **THEN** all three tags SHALL be returned as-is — no stripping of `collection:*` tags
|
||||
|
||||
#### Scenario: kb_set_collection no longer exists
|
||||
|
||||
- **WHEN** an MCP client attempts to call `kb_set_collection`
|
||||
- **THEN** the tool SHALL not be found (removed)
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: kb_search without collection parameter
|
||||
|
||||
The `kb_search` MCP tool SHALL accept `tags` (optional list of str) for filtering but SHALL NOT accept a `collection` parameter. Callers that previously used `collection="memory"` SHALL instead use `tags=["collection:memory"]` or whatever tag convention they prefer.
|
||||
|
||||
#### Scenario: Filter by tag instead of collection
|
||||
|
||||
- **WHEN** `kb_search(query="test", tags=["agent:mybot"])` is called
|
||||
- **THEN** results SHALL be filtered to documents tagged "agent:mybot"
|
||||
- **AND** no collection field SHALL be present in the response
|
||||
|
||||
### Requirement: kb_addnote without collection parameter
|
||||
|
||||
The `kb_addnote` MCP tool SHALL accept `tags` (optional list of str) but SHALL NOT accept a `collection` parameter. The tool SHALL NOT automatically apply any default collection tag — only explicitly provided tags are applied.
|
||||
|
||||
#### Scenario: Add note with explicit tags
|
||||
|
||||
- **WHEN** `kb_addnote(text="hello", tags=["agent:mybot", "memory"])` is called
|
||||
- **THEN** the note SHALL be created with exactly those two tags — no `collection:documents` tag added
|
||||
|
||||
### Requirement: kb_upload_start without collection parameter
|
||||
|
||||
The `kb_upload_start` MCP tool SHALL accept `tags` (optional list of str) but SHALL NOT accept a `collection` parameter. The tool SHALL NOT automatically apply any default collection tag.
|
||||
|
||||
### Requirement: kb_update_note without collection processing
|
||||
|
||||
The `kb_update_note` MCP tool SHALL return the document as-is from the engine without passing it through `_process_document`. All tags SHALL be visible in the response.
|
||||
|
||||
### Requirement: kb_get without collection processing
|
||||
|
||||
The `kb_get` MCP tool SHALL return documents as-is from the engine without passing through `_process_document`. All tags SHALL be visible in the response. No `collection` field SHALL be injected.
|
||||
@@ -0,0 +1,45 @@
|
||||
## 1. Remove collections from MCP server
|
||||
|
||||
- [x] 1.1 Remove collection constants and helper functions from `mcp/server.py` (`COLLECTION_TAG_PREFIX`, `DEFAULT_COLLECTION`, `_collection_tag`, `_strip_collection_tags`, `_process_document`, `_process_search_results`, `_ensure_exclusive_collection`)
|
||||
- [x] 1.2 Remove `collection` parameter from `kb_search`, `kb_addnote`, `kb_upload_start` tools
|
||||
- [x] 1.3 Remove `kb_set_collection` tool entirely
|
||||
- [x] 1.4 Remove `_process_document` / `_process_search_results` calls from `kb_get`, `kb_update_note`, `kb_search`
|
||||
- [x] 1.5 Update MCP server instructions text to reflect tags-only approach
|
||||
|
||||
## 2. Engine bulk infrastructure
|
||||
|
||||
- [x] 2.1 Add `bulk_safety_percent` to `Config` class in `engine/kb/config.py` (env var `KB_BULK_SAFETY_PERCENT`, default 70)
|
||||
- [x] 2.2 Add `job_type` column migration to `database.py` `init_schema` (TEXT, default "ingest")
|
||||
- [x] 2.3 Add `resolve_bulk_selection(conn, document_ids, tags, doc_type, from_id, to_id)` helper to `database.py` — returns list of matching document IDs
|
||||
- [x] 2.4 Add `create_bulk_job(conn, job_type, filters_json, matched, succeeded, failed, errors_json)` helper to `database.py`
|
||||
|
||||
## 3. Engine bulk endpoints
|
||||
|
||||
- [x] 3.1 Create `engine/kb/routes/bulk.py` with shared Pydantic request model (`BulkSelectionRequest` with selection fields + `force` bool)
|
||||
- [x] 3.2 Add `_check_safety_threshold` helper that returns 409 if threshold exceeded
|
||||
- [x] 3.3 Implement `POST /api/v1/bulk/delete` — resolve selection, check threshold, delete documents in transaction, clean up files, log job, return summary
|
||||
- [x] 3.4 Implement `POST /api/v1/bulk/tags` — resolve selection, check threshold, add/remove tags on matched docs, log job, return summary
|
||||
- [x] 3.5 Implement `POST /api/v1/bulk/set-tags` — resolve selection, check threshold, clear and replace tags on matched docs, log job, return summary
|
||||
- [x] 3.6 Import bulk routes in engine app startup (add to `engine/kb/routes/__init__.py` or `main.py`)
|
||||
|
||||
## 4. MCP bulk tools
|
||||
|
||||
- [x] 4.1 Add `bulk_delete`, `bulk_tags`, `bulk_set_tags` methods to `mcp/engine.py`
|
||||
- [x] 4.2 Add `kb_bulk_delete` tool to `mcp/server.py`
|
||||
- [x] 4.3 Add `kb_bulk_tags` tool to `mcp/server.py`
|
||||
- [x] 4.4 Add `kb_bulk_set_tags` tool to `mcp/server.py`
|
||||
|
||||
## 5. CLI bulk commands
|
||||
|
||||
- [x] 5.1 Create `client/cmd/bulk_remove.go` — `kb bulk-remove` with filter flags, confirmation prompt, JSON output support
|
||||
- [x] 5.2 Create `client/cmd/bulk_tag.go` — `kb bulk-tag` with filter flags + `--add`/`--remove`, confirmation prompt
|
||||
- [x] 5.3 Create `client/cmd/bulk_set_tags.go` — `kb bulk-set-tags` with filter flags + `--set`, confirmation prompt
|
||||
|
||||
## 6. Verification
|
||||
|
||||
- [x] 6.1 Test collection removal: verify `kb_search`, `kb_addnote`, `kb_get`, `kb_update_note`, `kb_upload_start` work without collection params
|
||||
- [x] 6.2 Test bulk delete via engine API: filter by tags, by IDs, by range, safety threshold trigger and force override
|
||||
- [x] 6.3 Test bulk tags and bulk set-tags via engine API
|
||||
- [x] 6.4 Test MCP bulk tools against running engine
|
||||
- [x] 6.5 Test CLI bulk commands against running engine
|
||||
- [x] 6.6 Test audit trail: verify bulk jobs appear in `kb jobs` output
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user