140 lines
4.5 KiB
Markdown
140 lines
4.5 KiB
Markdown
# Developer Guide
|
|
|
|
Instructions for building from source, releasing, and contributing to kb.
|
|
|
|
## Building from source
|
|
|
|
### Engine
|
|
|
|
```bash
|
|
cd engine
|
|
|
|
# NVIDIA GPU
|
|
KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d
|
|
```
|
|
|
|
### Client
|
|
|
|
```bash
|
|
cd client
|
|
make build # produces ./kb binary
|
|
make all # or cross-compile: dist/kb-{os}-{arch}
|
|
```
|
|
|
|
## Running tests
|
|
|
|
### Engine
|
|
|
|
Engine tests run against SQLite (with sqlite-vec) and stub out the embedding
|
|
model, so they only need lightweight dependencies — no torch/docling install:
|
|
|
|
```bash
|
|
uv venv /tmp/kb-test-venv
|
|
uv pip install --python /tmp/kb-test-venv/bin/python pytest pytest-asyncio fastapi httpx sqlite-vec
|
|
cd engine && /tmp/kb-test-venv/bin/python -m pytest
|
|
```
|
|
|
|
### Client
|
|
|
|
```bash
|
|
cd client && go test ./...
|
|
```
|
|
|
|
## Search-quality benchmarking
|
|
|
|
`kb bench fixture.json` runs a fixture of queries with known-relevant documents
|
|
against each backend (fts, vec, hybrid, hybrid+rerank) and reports precision@k,
|
|
recall, and MRR. See `docs/bench-example.json` for the fixture format.
|
|
|
|
Run a bench before and after any ranking change (RRF weights, reranker, model
|
|
swap) and compare — keep a 20-30 query fixture against your real corpus outside
|
|
the repo.
|
|
|
|
## Building and releasing
|
|
|
|
Client and engine are versioned independently via `client/VERSION` and `engine/VERSION`. Each has its own release script and git tag prefix.
|
|
|
|
### Release client
|
|
|
|
```bash
|
|
./release-client.sh --gitea # patch bump, release via Gitea
|
|
./release-client.sh --github --minor # minor bump, release via GitHub
|
|
./release-client.sh --gitea --no-increment # release current version as-is
|
|
./release-client.sh --gitea --dry-run # preview without doing anything
|
|
```
|
|
|
|
Creates tag `client-vX.Y.Z`, builds Go binaries for all platforms, and creates a Gitea/GitHub release with binaries attached.
|
|
|
|
The client embeds a `MinEngineVersion` (from `client/MIN_ENGINE_VERSION`) and will hard-fail if the connected engine is too old.
|
|
|
|
### Release engine
|
|
|
|
```bash
|
|
./release-engine.sh --gitea # patch bump, release via Gitea
|
|
./release-engine.sh --github --minor # minor bump, release via GitHub
|
|
./release-engine.sh --gitea --no-increment # release current version as-is
|
|
./release-engine.sh --gitea --dry-run # preview without doing anything
|
|
```
|
|
|
|
Creates tag `engine-vX.Y.Z`, builds NVIDIA and CPU Docker images, creates a Gitea/GitHub release, and pushes images to the registry.
|
|
|
|
### Checking versions
|
|
|
|
```bash
|
|
# Client
|
|
kb --version
|
|
|
|
# Engine
|
|
curl http://localhost:8000/api/v1/status | jq .version
|
|
```
|
|
|
|
### Docker images
|
|
|
|
Images are pushed to `docker.dcglab.co.uk/public/kb/engine` with tags:
|
|
|
|
- `engine-v2.0.6-nvidia` / `engine-v2.0.6-cpu` — versioned
|
|
- `latest-nvidia` / `latest-cpu` — latest release
|
|
|
|
The release script authenticates to the registry using the
|
|
`DOCKER_DCGLAB_CI_USERNAME` and `DOCKER_DCGLAB_CI_PASSWORD` environment
|
|
variables.
|
|
|
|
Override the registry and org via environment variables:
|
|
|
|
```bash
|
|
REGISTRY=ghcr.io IMAGE_ORG=myorg ./release-engine.sh --github
|
|
```
|
|
|
|
Pushes are retried on transient registry failures. The engine images carry a
|
|
~5.6GB torch layer, and uploading it can fail with a 502 from the proxy in
|
|
front of the registry (or a 500 on the manifest PUT that follows), which
|
|
clears on a retry. Tune with:
|
|
|
|
```bash
|
|
PUSH_RETRIES=8 PUSH_RETRY_DELAY=20 ./release-engine.sh --gitea
|
|
```
|
|
|
|
## API reference
|
|
|
|
All endpoints are under `/api/v1/`. Requires `Authorization: Bearer <key>` header when `KB_API_KEY` is set.
|
|
|
|
| Method | Endpoint | Description |
|
|
|---|---|---|
|
|
| `GET` | `/health` | Health check (bypasses auth) |
|
|
| `POST` | `/search` | Hybrid search (JSON body) |
|
|
| `POST` | `/jobs` | Upload file/note for ingestion (multipart, returns 202 or 409 if duplicate) |
|
|
| `GET` | `/jobs` | List ingestion jobs |
|
|
| `GET` | `/jobs/{id}` | Job details |
|
|
| `GET` | `/documents` | List documents |
|
|
| `GET` | `/documents/{id}` | Document details with chunks |
|
|
| `GET` | `/documents/{id}/file` | Download original file |
|
|
| `DELETE` | `/documents/{id}` | Remove a document (and stored file) |
|
|
| `PUT` | `/documents/{id}/tags` | Add/remove tags |
|
|
| `GET` | `/tags` | List all tags (with descriptions) |
|
|
| `PUT` | `/tags/{name}/description` | Set/clear a tag context description |
|
|
| `GET` | `/status` | Engine status, GPU info, DB stats, rerank state |
|
|
| `POST` | `/reindex` | Re-embed all chunks |
|
|
| `POST` | `/bulk/delete` | Bulk delete documents by filter |
|
|
| `POST` | `/bulk/tags` | Bulk add/remove tags by filter |
|
|
| `POST` | `/bulk/set-tags` | Bulk replace tags by filter |
|