Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c5191df9c0 | |||
| afbe270181 | |||
| 9e957f1a9a | |||
| bbe6a5e909 | |||
| 743102aee4 | |||
| 0f3b3be59f | |||
| 2fa2ac1134 | |||
| b2176c36ea | |||
| 5f9946efc9 |
@@ -1,2 +1,3 @@
|
|||||||
examples/
|
examples/
|
||||||
.claude/
|
.claude/
|
||||||
|
__pycache__/
|
||||||
+100
@@ -0,0 +1,100 @@
|
|||||||
|
# Developer Guide
|
||||||
|
|
||||||
|
Instructions for building from source, releasing, and contributing to kb.
|
||||||
|
|
||||||
|
## Building from source
|
||||||
|
|
||||||
|
### Engine
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd engine
|
||||||
|
|
||||||
|
# NVIDIA GPU
|
||||||
|
KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d
|
||||||
|
|
||||||
|
# AMD GPU (ROCm)
|
||||||
|
KB_DATA_PATH=~/kb-data docker compose -f compose.rocm.yaml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### Client
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd client
|
||||||
|
make build # produces ./kb binary
|
||||||
|
make all # or cross-compile: dist/kb-{os}-{arch}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Building and releasing
|
||||||
|
|
||||||
|
Client and engine are versioned independently via `client/VERSION` and `engine/VERSION`. Each has its own release script and git tag prefix.
|
||||||
|
|
||||||
|
### Release client
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./release-client.sh --gitea # patch bump, release via Gitea
|
||||||
|
./release-client.sh --github --minor # minor bump, release via GitHub
|
||||||
|
./release-client.sh --gitea --no-increment # release current version as-is
|
||||||
|
./release-client.sh --gitea --dry-run # preview without doing anything
|
||||||
|
```
|
||||||
|
|
||||||
|
Creates tag `client-vX.Y.Z`, builds Go binaries for all platforms, and creates a Gitea/GitHub release with binaries attached.
|
||||||
|
|
||||||
|
The client embeds a `MinEngineVersion` (from `client/MIN_ENGINE_VERSION`) and will hard-fail if the connected engine is too old.
|
||||||
|
|
||||||
|
### Release engine
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./release-engine.sh --gitea # patch bump, release via Gitea
|
||||||
|
./release-engine.sh --github --minor # minor bump, release via GitHub
|
||||||
|
./release-engine.sh --gitea --no-increment # release current version as-is
|
||||||
|
./release-engine.sh --gitea --dry-run # preview without doing anything
|
||||||
|
```
|
||||||
|
|
||||||
|
Creates tag `engine-vX.Y.Z`, builds NVIDIA and ROCm Docker images, creates a Gitea/GitHub release, and pushes images to the registry.
|
||||||
|
|
||||||
|
### Checking versions
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Client
|
||||||
|
kb --version
|
||||||
|
|
||||||
|
# Engine
|
||||||
|
curl http://localhost:8000/api/v1/status | jq .version
|
||||||
|
```
|
||||||
|
|
||||||
|
### Docker images
|
||||||
|
|
||||||
|
Images are pushed to `docker.dcglab.co.uk/dcg/kb/engine` with tags:
|
||||||
|
|
||||||
|
- `engine-v2.0.6-nvidia` / `engine-v2.0.6-rocm` — versioned
|
||||||
|
- `latest-nvidia` / `latest-rocm` — latest release
|
||||||
|
|
||||||
|
Override the registry and org via environment variables:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
REGISTRY=ghcr.io IMAGE_ORG=myorg ./release-engine.sh --github
|
||||||
|
```
|
||||||
|
|
||||||
|
## API reference
|
||||||
|
|
||||||
|
All endpoints are under `/api/v1/`. Requires `Authorization: Bearer <key>` header when `KB_API_KEY` is set.
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `GET` | `/health` | Health check (bypasses auth) |
|
||||||
|
| `POST` | `/search` | Hybrid search (JSON body) |
|
||||||
|
| `POST` | `/jobs` | Upload file/note for ingestion (multipart, returns 202 or 409 if duplicate) |
|
||||||
|
| `GET` | `/jobs` | List ingestion jobs |
|
||||||
|
| `GET` | `/jobs/{id}` | Job details |
|
||||||
|
| `GET` | `/documents` | List documents |
|
||||||
|
| `GET` | `/documents/{id}` | Document details with chunks |
|
||||||
|
| `GET` | `/documents/{id}/file` | Download original file |
|
||||||
|
| `DELETE` | `/documents/{id}` | Remove a document (and stored file) |
|
||||||
|
| `PUT` | `/documents/{id}/tags` | Add/remove tags |
|
||||||
|
| `GET` | `/tags` | List all tags |
|
||||||
|
| `GET` | `/status` | Engine status, GPU info, DB stats |
|
||||||
|
| `POST` | `/reindex` | Re-embed all chunks |
|
||||||
|
|
||||||
|
## Future: ROCm runtime migration
|
||||||
|
|
||||||
|
The `onnxruntime-rocm` execution provider was removed from onnxruntime as of v1.23. AMD is pushing toward the **MIGraphX execution provider** as the replacement for ROCm GPU inference. When upgrading onnxruntime beyond v1.22, the ROCm Dockerfile will need to switch from `onnxruntime-rocm` to `onnxruntime` with the MIGraphX EP and install the `migraphx` runtime libraries instead.
|
||||||
@@ -83,17 +83,7 @@ services:
|
|||||||
KB_DATA_PATH=~/kb-data docker compose up -d
|
KB_DATA_PATH=~/kb-data docker compose up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
**From source** (for development):
|
See [DEVELOPER.md](DEVELOPER.md) to run the engine from source.
|
||||||
|
|
||||||
```bash
|
|
||||||
cd engine
|
|
||||||
|
|
||||||
# NVIDIA GPU
|
|
||||||
KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d
|
|
||||||
|
|
||||||
# AMD GPU (ROCm)
|
|
||||||
KB_DATA_PATH=~/kb-data docker compose -f compose.rocm.yaml up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
The engine will download the embedding model on first start (~90MB) and load it onto the GPU. Check readiness:
|
The engine will download the embedding model on first start (~90MB) and load it onto the GPU. Check readiness:
|
||||||
|
|
||||||
@@ -104,18 +94,32 @@ curl http://localhost:8000/api/v1/health
|
|||||||
|
|
||||||
### 2. Install the client
|
### 2. Install the client
|
||||||
|
|
||||||
Build from source:
|
**From a release** (recommended):
|
||||||
|
|
||||||
|
Check [releases](https://gitea.dcglab.co.uk/steve/kb/releases) for the latest client tag, then:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd client
|
# Set the version tag
|
||||||
make build # produces ./kb binary
|
TAG=client-v2.1.0
|
||||||
|
|
||||||
|
# Linux (amd64)
|
||||||
|
curl -L -o kb https://gitea.dcglab.co.uk/steve/kb/releases/download/${TAG}/kb-linux-amd64
|
||||||
|
|
||||||
|
# Linux (arm64)
|
||||||
|
curl -L -o kb https://gitea.dcglab.co.uk/steve/kb/releases/download/${TAG}/kb-linux-arm64
|
||||||
|
|
||||||
|
# macOS (Apple Silicon)
|
||||||
|
curl -L -o kb https://gitea.dcglab.co.uk/steve/kb/releases/download/${TAG}/kb-darwin-arm64
|
||||||
|
|
||||||
|
# macOS (Intel)
|
||||||
|
curl -L -o kb https://gitea.dcglab.co.uk/steve/kb/releases/download/${TAG}/kb-darwin-amd64
|
||||||
|
|
||||||
|
# Then install
|
||||||
|
chmod +x kb
|
||||||
|
sudo mv kb /usr/local/bin/
|
||||||
```
|
```
|
||||||
|
|
||||||
Or cross-compile for all platforms:
|
See [DEVELOPER.md](DEVELOPER.md) to build the client from source.
|
||||||
|
|
||||||
```bash
|
|
||||||
make all # produces dist/kb-{os}-{arch} binaries
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Configure the client
|
### 3. Configure the client
|
||||||
|
|
||||||
@@ -132,9 +136,9 @@ Override via environment variables (`KB_ENGINE_URL`, `KB_API_KEY`) or CLI flags
|
|||||||
### 4. Use it
|
### 4. Use it
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Quick notes (shorthand — no subcommand needed)
|
# Add notes
|
||||||
kb "Always restart nginx after config changes"
|
kb addnote "Always restart nginx after config changes"
|
||||||
kb "Server room is building 3, floor 2" --tags ops
|
kb addnote "Server room is building 3, floor 2" --tags ops
|
||||||
|
|
||||||
# Add files (async — uploads and exits immediately)
|
# Add files (async — uploads and exits immediately)
|
||||||
kb addfile ~/docs/manual.pdf --tags admin
|
kb addfile ~/docs/manual.pdf --tags admin
|
||||||
@@ -194,81 +198,6 @@ KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d
|
|||||||
|
|
||||||
Data is GPU-vendor-agnostic — you can ingest on NVIDIA and serve from AMD (or vice versa) with the same data directory.
|
Data is GPU-vendor-agnostic — you can ingest on NVIDIA and serve from AMD (or vice versa) with the same data directory.
|
||||||
|
|
||||||
## API reference
|
|
||||||
|
|
||||||
All endpoints are under `/api/v1/`. Requires `Authorization: Bearer <key>` header when `KB_API_KEY` is set.
|
|
||||||
|
|
||||||
| Method | Endpoint | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `GET` | `/health` | Health check (bypasses auth) |
|
|
||||||
| `POST` | `/search` | Hybrid search (JSON body) |
|
|
||||||
| `POST` | `/jobs` | Upload file/note for ingestion (multipart, returns 202 or 409 if duplicate) |
|
|
||||||
| `GET` | `/jobs` | List ingestion jobs |
|
|
||||||
| `GET` | `/jobs/{id}` | Job details |
|
|
||||||
| `GET` | `/documents` | List documents |
|
|
||||||
| `GET` | `/documents/{id}` | Document details with chunks |
|
|
||||||
| `GET` | `/documents/{id}/file` | Download original file |
|
|
||||||
| `DELETE` | `/documents/{id}` | Remove a document (and stored file) |
|
|
||||||
| `PUT` | `/documents/{id}/tags` | Add/remove tags |
|
|
||||||
| `GET` | `/tags` | List all tags |
|
|
||||||
| `GET` | `/status` | Engine status, GPU info, DB stats |
|
|
||||||
| `POST` | `/reindex` | Re-embed all chunks |
|
|
||||||
|
|
||||||
## Building and releasing
|
|
||||||
|
|
||||||
Client and engine are versioned independently via `client/VERSION` and `engine/VERSION`. Each has its own release script and git tag prefix.
|
|
||||||
|
|
||||||
### Release client
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./release-client.sh --gitea # patch bump, release via Gitea
|
|
||||||
./release-client.sh --github --minor # minor bump, release via GitHub
|
|
||||||
./release-client.sh --gitea --no-increment # release current version as-is
|
|
||||||
./release-client.sh --gitea --dry-run # preview without doing anything
|
|
||||||
```
|
|
||||||
|
|
||||||
Creates tag `client-vX.Y.Z`, builds Go binaries for all platforms, and creates a Gitea/GitHub release with binaries attached.
|
|
||||||
|
|
||||||
The client embeds a `MinEngineVersion` (from `client/MIN_ENGINE_VERSION`) and will hard-fail if the connected engine is too old.
|
|
||||||
|
|
||||||
### Release engine
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./release-engine.sh --gitea # patch bump, release via Gitea
|
|
||||||
./release-engine.sh --github --minor # minor bump, release via GitHub
|
|
||||||
./release-engine.sh --gitea --no-increment # release current version as-is
|
|
||||||
./release-engine.sh --gitea --dry-run # preview without doing anything
|
|
||||||
```
|
|
||||||
|
|
||||||
Creates tag `engine-vX.Y.Z`, builds NVIDIA and ROCm Docker images, creates a Gitea/GitHub release, and pushes images to the registry.
|
|
||||||
|
|
||||||
### Checking versions
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Client
|
|
||||||
kb --version
|
|
||||||
|
|
||||||
# Engine
|
|
||||||
curl http://localhost:8000/api/v1/status | jq .version
|
|
||||||
```
|
|
||||||
|
|
||||||
### Docker images
|
|
||||||
|
|
||||||
Images are pushed to `docker.dcglab.co.uk/dcg/kb/engine` with tags:
|
|
||||||
|
|
||||||
- `engine-v2.0.6-nvidia` / `engine-v2.0.6-rocm` — versioned
|
|
||||||
- `latest-nvidia` / `latest-rocm` — latest release
|
|
||||||
|
|
||||||
Override the registry and org via environment variables:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
REGISTRY=ghcr.io IMAGE_ORG=myorg ./release-engine.sh --github
|
|
||||||
```
|
|
||||||
|
|
||||||
## Future: ROCm runtime migration
|
|
||||||
|
|
||||||
The `onnxruntime-rocm` execution provider was removed from onnxruntime as of v1.23. AMD is pushing toward the **MIGraphX execution provider** as the replacement for ROCm GPU inference. When upgrading onnxruntime beyond v1.22, the ROCm Dockerfile will need to switch from `onnxruntime-rocm` to `onnxruntime` with the MIGraphX EP and install the `migraphx` runtime libraries instead.
|
|
||||||
|
|
||||||
## Claude Code skill
|
## Claude Code skill
|
||||||
|
|
||||||
This tool is designed to be wrapped as a Claude Code skill. See `SKILL.md` for the skill definition.
|
This tool is designed to be wrapped as a Claude Code skill. See `SKILL.md` for the skill definition.
|
||||||
|
|||||||
@@ -10,14 +10,14 @@ Search, manage, and add to the user's personal knowledge base containing PDFs, W
|
|||||||
- User asks "how do I..." style questions that their knowledge base likely covers
|
- User asks "how do I..." style questions that their knowledge base likely covers
|
||||||
- User wants to save a note, add a file, or manage their knowledge base
|
- User wants to save a note, add a file, or manage their knowledge base
|
||||||
|
|
||||||
## Quick notes
|
## Adding notes
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kb "remember to update DNS records" # add a note
|
kb addnote "remember to update DNS records" # add a note
|
||||||
kb "server room is building 3, floor 2" --tags ops # add a tagged note
|
kb addnote "server room is building 3, floor 2" --tags ops # add a tagged note
|
||||||
```
|
```
|
||||||
|
|
||||||
Bare text without a subcommand is treated as a note and submitted for ingestion.
|
The note text must be a single quoted argument.
|
||||||
|
|
||||||
## Search (primary use case)
|
## Search (primary use case)
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -1 +1 @@
|
|||||||
2.1.0
|
2.2.0
|
||||||
|
|||||||
@@ -206,53 +206,3 @@ func uploadFile(client *api.Client, path, tags string) (*uploadResult, error) {
|
|||||||
return &uploadResult{Raw: result}, nil
|
return &uploadResult{Raw: result}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func submitNote(client *api.Client, note, tags string) error {
|
|
||||||
fields := map[string]string{
|
|
||||||
"note": note,
|
|
||||||
}
|
|
||||||
if tags != "" {
|
|
||||||
fields["tags"] = tags
|
|
||||||
}
|
|
||||||
|
|
||||||
resp, err := client.PostMultipart("/api/v1/jobs", fields, nil)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintln(os.Stderr, err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
if resp.StatusCode == http.StatusConflict {
|
|
||||||
var result interface{}
|
|
||||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
|
||||||
return fmt.Errorf("failed to decode response: %w", err)
|
|
||||||
}
|
|
||||||
if output.IsJSON() {
|
|
||||||
output.PrintJSON(result)
|
|
||||||
} else {
|
|
||||||
if m, ok := result.(map[string]interface{}); ok {
|
|
||||||
if docID, ok := m["document_id"].(float64); ok {
|
|
||||||
fmt.Printf("Already imported: %s (doc ID: %.0f)\n", m["title"], docID)
|
|
||||||
} else if jobID, ok := m["job_id"].(float64); ok {
|
|
||||||
fmt.Printf("Already queued: %s (job ID: %.0f)\n", m["title"], jobID)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := api.CheckError(resp); err != nil {
|
|
||||||
fmt.Fprintln(os.Stderr, err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
var result interface{}
|
|
||||||
if err := api.DecodeJSON(resp, &result); err != nil {
|
|
||||||
return fmt.Errorf("failed to decode response: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if output.IsJSON() {
|
|
||||||
output.PrintJSON(result)
|
|
||||||
} else {
|
|
||||||
fmt.Println("Queued: note")
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"github.com/kb-search/kb/internal/api"
|
||||||
|
"github.com/kb-search/kb/internal/output"
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
)
|
||||||
|
|
||||||
|
var addnoteCmd = &cobra.Command{
|
||||||
|
Use: "addnote <text>",
|
||||||
|
Short: "Add a text note to the knowledge base",
|
||||||
|
Args: func(cmd *cobra.Command, args []string) error {
|
||||||
|
if len(args) == 0 {
|
||||||
|
return fmt.Errorf("requires a note text argument\n\n Usage: kb addnote \"your note text here\"")
|
||||||
|
}
|
||||||
|
if len(args) > 1 {
|
||||||
|
return fmt.Errorf("accepts 1 arg but received %d — quote your note text, e.g. kb addnote \"your note text here\"", len(args))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
RunE: runAddnote,
|
||||||
|
}
|
||||||
|
|
||||||
|
func init() {
|
||||||
|
addnoteCmd.Flags().String("tags", "", "tags (comma-separated)")
|
||||||
|
rootCmd.AddCommand(addnoteCmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
func runAddnote(cmd *cobra.Command, args []string) error {
|
||||||
|
tags, _ := cmd.Flags().GetString("tags")
|
||||||
|
client := api.NewClient()
|
||||||
|
return submitNote(client, args[0], tags)
|
||||||
|
}
|
||||||
|
|
||||||
|
func submitNote(client *api.Client, note, tags string) error {
|
||||||
|
fields := map[string]string{
|
||||||
|
"note": note,
|
||||||
|
}
|
||||||
|
if tags != "" {
|
||||||
|
fields["tags"] = tags
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := client.PostMultipart("/api/v1/jobs", fields, nil)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
if resp.StatusCode == http.StatusConflict {
|
||||||
|
var result interface{}
|
||||||
|
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||||
|
return fmt.Errorf("failed to decode response: %w", err)
|
||||||
|
}
|
||||||
|
if output.IsJSON() {
|
||||||
|
output.PrintJSON(result)
|
||||||
|
} else {
|
||||||
|
if m, ok := result.(map[string]interface{}); ok {
|
||||||
|
if docID, ok := m["document_id"].(float64); ok {
|
||||||
|
fmt.Printf("Already imported: %s (doc ID: %.0f)\n", m["title"], docID)
|
||||||
|
} else if jobID, ok := m["job_id"].(float64); ok {
|
||||||
|
fmt.Printf("Already queued: %s (job ID: %.0f)\n", m["title"], jobID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := api.CheckError(resp); err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
var result interface{}
|
||||||
|
if err := api.DecodeJSON(resp, &result); err != nil {
|
||||||
|
return fmt.Errorf("failed to decode response: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if output.IsJSON() {
|
||||||
|
output.PrintJSON(result)
|
||||||
|
} else {
|
||||||
|
fmt.Println("Queued: note")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -11,9 +11,9 @@ var examplesCmd = &cobra.Command{
|
|||||||
Short: "Show common usage examples",
|
Short: "Show common usage examples",
|
||||||
Args: cobra.NoArgs,
|
Args: cobra.NoArgs,
|
||||||
Run: func(cmd *cobra.Command, args []string) {
|
Run: func(cmd *cobra.Command, args []string) {
|
||||||
fmt.Print(`Quick notes:
|
fmt.Print(`Add notes:
|
||||||
kb "Remember to update DNS records"
|
kb addnote "Remember to update DNS records"
|
||||||
kb "Server room is building 3" --tags ops
|
kb addnote "Server room is building 3" --tags ops
|
||||||
|
|
||||||
Add files:
|
Add files:
|
||||||
kb addfile report.pdf
|
kb addfile report.pdf
|
||||||
|
|||||||
+1
-30
@@ -3,7 +3,6 @@ package cmd
|
|||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"strings"
|
|
||||||
|
|
||||||
"github.com/kb-search/kb/internal/api"
|
"github.com/kb-search/kb/internal/api"
|
||||||
"github.com/kb-search/kb/internal/config"
|
"github.com/kb-search/kb/internal/config"
|
||||||
@@ -23,10 +22,9 @@ var (
|
|||||||
)
|
)
|
||||||
|
|
||||||
var rootCmd = &cobra.Command{
|
var rootCmd = &cobra.Command{
|
||||||
Use: "kb [\"note text\" | command]",
|
Use: "kb [command]",
|
||||||
Short: "kb-search CLI client",
|
Short: "kb-search CLI client",
|
||||||
Long: "A CLI client for the kb-search v2 engine API.\nRun 'kb examples' for common usage patterns.",
|
Long: "A CLI client for the kb-search v2 engine API.\nRun 'kb examples' for common usage patterns.",
|
||||||
Args: cobra.ArbitraryArgs,
|
|
||||||
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
|
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
|
||||||
if err := config.Load(); err != nil {
|
if err := config.Load(); err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -34,41 +32,14 @@ var rootCmd = &cobra.Command{
|
|||||||
config.ApplyFlags(flagEngine, flagFormat, flagAPIKey)
|
config.ApplyFlags(flagEngine, flagFormat, flagAPIKey)
|
||||||
return nil
|
return nil
|
||||||
},
|
},
|
||||||
RunE: func(cmd *cobra.Command, args []string) error {
|
|
||||||
if len(args) == 0 {
|
|
||||||
return cmd.Help()
|
|
||||||
}
|
|
||||||
note := strings.Join(args, " ")
|
|
||||||
tags, _ := cmd.Flags().GetString("tags")
|
|
||||||
client := api.NewClient()
|
|
||||||
return submitNote(client, note, tags)
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func init() {
|
func init() {
|
||||||
api.SetVersionInfo(Version, MinEngineVersion)
|
api.SetVersionInfo(Version, MinEngineVersion)
|
||||||
rootCmd.Version = Version
|
rootCmd.Version = Version
|
||||||
rootCmd.SetUsageTemplate(`Quick note taking:
|
|
||||||
kb "note text" [flags]
|
|
||||||
|
|
||||||
Normal usage:
|
|
||||||
kb [command] [flags]{{if .HasAvailableSubCommands}}
|
|
||||||
|
|
||||||
Available Commands:{{range .Commands}}{{if (or .IsAvailableCommand (eq .Name "help"))}}
|
|
||||||
{{rpad .Name .NamePadding }} {{.Short}}{{end}}{{end}}{{end}}{{if .HasAvailableLocalFlags}}
|
|
||||||
|
|
||||||
Flags:
|
|
||||||
{{.LocalFlags.FlagUsages | trimTrailingWhitespaces}}{{end}}{{if .HasAvailableInheritedFlags}}
|
|
||||||
|
|
||||||
Global Flags:
|
|
||||||
{{.InheritedFlags.FlagUsages | trimTrailingWhitespaces}}{{end}}
|
|
||||||
|
|
||||||
Use "{{.CommandPath}} [command] --help" for more information about a command.
|
|
||||||
`)
|
|
||||||
rootCmd.PersistentFlags().StringVar(&flagEngine, "engine", "", "engine API URL")
|
rootCmd.PersistentFlags().StringVar(&flagEngine, "engine", "", "engine API URL")
|
||||||
rootCmd.PersistentFlags().StringVar(&flagFormat, "format", "", "output format (human|json)")
|
rootCmd.PersistentFlags().StringVar(&flagFormat, "format", "", "output format (human|json)")
|
||||||
rootCmd.PersistentFlags().StringVar(&flagAPIKey, "api-key", "", "API key for authentication")
|
rootCmd.PersistentFlags().StringVar(&flagAPIKey, "api-key", "", "API key for authentication")
|
||||||
rootCmd.Flags().String("tags", "", "tags for note shorthand (comma-separated)")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Execute runs the root command.
|
// Execute runs the root command.
|
||||||
|
|||||||
@@ -0,0 +1,69 @@
|
|||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRootCmd_NoArgs_ShowsHelp(t *testing.T) {
|
||||||
|
rootCmd.SetArgs([]string{})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
rootCmd.SetOut(&stdout)
|
||||||
|
|
||||||
|
err := rootCmd.Execute()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error for zero args, got: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
output := stdout.String()
|
||||||
|
if !strings.Contains(output, "Available Commands") {
|
||||||
|
t.Errorf("expected help output, got: %s", output)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRootCmd_UnknownCommand_ReturnsError(t *testing.T) {
|
||||||
|
rootCmd.SetArgs([]string{"notacommand"})
|
||||||
|
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
rootCmd.SetErr(&stderr)
|
||||||
|
|
||||||
|
err := rootCmd.Execute()
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for unknown command, got nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
errMsg := err.Error()
|
||||||
|
if !strings.Contains(errMsg, "unknown command") {
|
||||||
|
t.Errorf("expected 'unknown command' error, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAddnoteCmd_NoArgs_ReturnsError(t *testing.T) {
|
||||||
|
rootCmd.SetArgs([]string{"addnote"})
|
||||||
|
|
||||||
|
err := rootCmd.Execute()
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for addnote with no args, got nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
errMsg := err.Error()
|
||||||
|
if !strings.Contains(errMsg, "requires a note text argument") {
|
||||||
|
t.Errorf("expected 'requires a note text argument' error, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAddnoteCmd_TooManyArgs_ReturnsError(t *testing.T) {
|
||||||
|
rootCmd.SetArgs([]string{"addnote", "hello", "world"})
|
||||||
|
|
||||||
|
err := rootCmd.Execute()
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for addnote with too many args, got nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
errMsg := err.Error()
|
||||||
|
if !strings.Contains(errMsg, "quote your note text") {
|
||||||
|
t.Errorf("expected 'accepts 1 arg' error, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
+1
-1
@@ -1 +1 @@
|
|||||||
2.0.6
|
2.1.0
|
||||||
|
|||||||
Executable
+4
@@ -0,0 +1,4 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
docker stop engine-kb-engine-1
|
||||||
|
KB_MODEL=BAAI/bge-base-en-v1.5 KB_DATA_PATH=~/kb-data docker compose -f compose.nvidia.yaml up -d --build
|
||||||
+70
-7
@@ -10,6 +10,60 @@ import struct
|
|||||||
from typing import Any, Optional
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
|
||||||
|
def build_enriched_text(title: str, chunk_text: str, metadata: dict | None = None) -> str:
|
||||||
|
"""Build enriched text by prepending document title and optional section header.
|
||||||
|
|
||||||
|
Format: "{title} > {section_header}\\n\\n{chunk_text}" or "{title}\\n\\n{chunk_text}".
|
||||||
|
"""
|
||||||
|
section_header = (metadata or {}).get("section_header")
|
||||||
|
if section_header:
|
||||||
|
return f"{title} > {section_header}\n\n{chunk_text}"
|
||||||
|
return f"{title}\n\n{chunk_text}"
|
||||||
|
|
||||||
|
|
||||||
|
def _backfill_enriched_text(conn: sqlite3.Connection) -> None:
|
||||||
|
"""Backfill enriched_text for all existing chunks."""
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT c.id, c.text, c.metadata, d.title "
|
||||||
|
"FROM chunks c JOIN documents d ON c.document_id = d.id"
|
||||||
|
).fetchall()
|
||||||
|
for row in rows:
|
||||||
|
metadata = json.loads(row["metadata"]) if row["metadata"] else None
|
||||||
|
enriched = build_enriched_text(row["title"], row["text"], metadata)
|
||||||
|
conn.execute("UPDATE chunks SET enriched_text = ? WHERE id = ?", (enriched, row["id"]))
|
||||||
|
|
||||||
|
|
||||||
|
def _rebuild_fts(conn: sqlite3.Connection) -> None:
|
||||||
|
"""Drop and recreate chunks_fts to index enriched_text, with updated triggers."""
|
||||||
|
conn.executescript("""
|
||||||
|
DROP TRIGGER IF EXISTS chunks_ai;
|
||||||
|
DROP TRIGGER IF EXISTS chunks_ad;
|
||||||
|
DROP TRIGGER IF EXISTS chunks_au;
|
||||||
|
DROP TABLE IF EXISTS chunks_fts;
|
||||||
|
|
||||||
|
CREATE VIRTUAL TABLE chunks_fts USING fts5(
|
||||||
|
text,
|
||||||
|
content=chunks,
|
||||||
|
content_rowid=id
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TRIGGER chunks_ai AFTER INSERT ON chunks BEGIN
|
||||||
|
INSERT INTO chunks_fts(rowid, text) VALUES (new.id, new.enriched_text);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER chunks_ad AFTER DELETE ON chunks BEGIN
|
||||||
|
INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES ('delete', old.id, old.enriched_text);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER chunks_au AFTER UPDATE ON chunks BEGIN
|
||||||
|
INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES ('delete', old.id, old.enriched_text);
|
||||||
|
INSERT INTO chunks_fts(rowid, text) VALUES (new.id, new.enriched_text);
|
||||||
|
END;
|
||||||
|
""")
|
||||||
|
# Repopulate FTS from existing enriched_text
|
||||||
|
conn.execute("INSERT INTO chunks_fts(rowid, text) SELECT id, enriched_text FROM chunks")
|
||||||
|
|
||||||
|
|
||||||
def get_connection(db_path: str) -> sqlite3.Connection:
|
def get_connection(db_path: str) -> sqlite3.Connection:
|
||||||
"""Return a sqlite3 connection with WAL mode, Row factory, and foreign keys enabled."""
|
"""Return a sqlite3 connection with WAL mode, Row factory, and foreign keys enabled."""
|
||||||
import sqlite_vec
|
import sqlite_vec
|
||||||
@@ -44,6 +98,7 @@ def init_schema(conn: sqlite3.Connection, embedding_dim: int) -> None:
|
|||||||
document_id INTEGER REFERENCES documents(id) ON DELETE CASCADE,
|
document_id INTEGER REFERENCES documents(id) ON DELETE CASCADE,
|
||||||
chunk_index INTEGER,
|
chunk_index INTEGER,
|
||||||
text TEXT,
|
text TEXT,
|
||||||
|
enriched_text TEXT,
|
||||||
token_count INTEGER,
|
token_count INTEGER,
|
||||||
metadata TEXT DEFAULT '{{}}',
|
metadata TEXT DEFAULT '{{}}',
|
||||||
UNIQUE(document_id, chunk_index)
|
UNIQUE(document_id, chunk_index)
|
||||||
@@ -55,18 +110,18 @@ def init_schema(conn: sqlite3.Connection, embedding_dim: int) -> None:
|
|||||||
content_rowid=id
|
content_rowid=id
|
||||||
);
|
);
|
||||||
|
|
||||||
-- Triggers to keep FTS index in sync with chunks table
|
-- Triggers to keep FTS index in sync with chunks table (using enriched_text)
|
||||||
CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN
|
CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN
|
||||||
INSERT INTO chunks_fts(rowid, text) VALUES (new.id, new.text);
|
INSERT INTO chunks_fts(rowid, text) VALUES (new.id, new.enriched_text);
|
||||||
END;
|
END;
|
||||||
|
|
||||||
CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN
|
CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN
|
||||||
INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES ('delete', old.id, old.text);
|
INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES ('delete', old.id, old.enriched_text);
|
||||||
END;
|
END;
|
||||||
|
|
||||||
CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks BEGIN
|
CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks BEGIN
|
||||||
INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES ('delete', old.id, old.text);
|
INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES ('delete', old.id, old.enriched_text);
|
||||||
INSERT INTO chunks_fts(rowid, text) VALUES (new.id, new.text);
|
INSERT INTO chunks_fts(rowid, text) VALUES (new.id, new.enriched_text);
|
||||||
END;
|
END;
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS tags (
|
CREATE TABLE IF NOT EXISTS tags (
|
||||||
@@ -123,6 +178,13 @@ def init_schema(conn: sqlite3.Connection, embedding_dim: int) -> None:
|
|||||||
if "original_filename" not in doc_cols:
|
if "original_filename" not in doc_cols:
|
||||||
conn.execute("ALTER TABLE documents ADD COLUMN original_filename TEXT")
|
conn.execute("ALTER TABLE documents ADD COLUMN original_filename TEXT")
|
||||||
|
|
||||||
|
# Migrate: add enriched_text to chunks and rebuild FTS to index it
|
||||||
|
chunk_cols = {row[1] for row in conn.execute("PRAGMA table_info(chunks)").fetchall()}
|
||||||
|
if "enriched_text" not in chunk_cols:
|
||||||
|
conn.execute("ALTER TABLE chunks ADD COLUMN enriched_text TEXT")
|
||||||
|
_backfill_enriched_text(conn)
|
||||||
|
_rebuild_fts(conn)
|
||||||
|
|
||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
@@ -205,6 +267,7 @@ def insert_chunk(
|
|||||||
document_id: int,
|
document_id: int,
|
||||||
chunk_index: int,
|
chunk_index: int,
|
||||||
text: str,
|
text: str,
|
||||||
|
enriched_text: str | None = None,
|
||||||
token_count: Optional[int] = None,
|
token_count: Optional[int] = None,
|
||||||
metadata: Any = None,
|
metadata: Any = None,
|
||||||
) -> int:
|
) -> int:
|
||||||
@@ -217,8 +280,8 @@ def insert_chunk(
|
|||||||
metadata_str = str(metadata)
|
metadata_str = str(metadata)
|
||||||
|
|
||||||
cur = conn.execute(
|
cur = conn.execute(
|
||||||
"INSERT INTO chunks(document_id, chunk_index, text, token_count, metadata) VALUES (?, ?, ?, ?, ?)",
|
"INSERT INTO chunks(document_id, chunk_index, text, enriched_text, token_count, metadata) VALUES (?, ?, ?, ?, ?, ?)",
|
||||||
(document_id, chunk_index, text, token_count, metadata_str),
|
(document_id, chunk_index, text, enriched_text or text, token_count, metadata_str),
|
||||||
)
|
)
|
||||||
conn.commit()
|
conn.commit()
|
||||||
return cur.lastrowid
|
return cur.lastrowid
|
||||||
|
|||||||
@@ -19,10 +19,10 @@ async def reindex():
|
|||||||
|
|
||||||
conn = get_connection(cfg.db_path)
|
conn = get_connection(cfg.db_path)
|
||||||
try:
|
try:
|
||||||
# Fetch all chunks
|
# Fetch all chunks — use enriched_text for embedding (includes title context)
|
||||||
rows = conn.execute("SELECT id, text FROM chunks ORDER BY id").fetchall()
|
rows = conn.execute("SELECT id, enriched_text FROM chunks ORDER BY id").fetchall()
|
||||||
chunk_ids = [row["id"] for row in rows]
|
chunk_ids = [row["id"] for row in rows]
|
||||||
chunk_texts = [row["text"] for row in rows]
|
chunk_texts = [row["enriched_text"] or "" for row in rows]
|
||||||
|
|
||||||
logger.info("Reindexing %d chunks with model '%s'", len(chunk_ids), cfg.model)
|
logger.info("Reindexing %d chunks with model '%s'", len(chunk_ids), cfg.model)
|
||||||
|
|
||||||
|
|||||||
+19
-8
@@ -8,6 +8,7 @@ import shutil
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from kb import config, database, embeddings, staging
|
from kb import config, database, embeddings, staging
|
||||||
|
from kb.database import build_enriched_text
|
||||||
from kb.ingest import detector
|
from kb.ingest import detector
|
||||||
|
|
||||||
logger = logging.getLogger("kb.worker")
|
logger = logging.getLogger("kb.worker")
|
||||||
@@ -146,20 +147,30 @@ def _process_job(job_row) -> tuple[str, int | None, int]:
|
|||||||
)
|
)
|
||||||
|
|
||||||
chunk_texts = [c if isinstance(c, str) else c["text"] for c in chunks]
|
chunk_texts = [c if isinstance(c, str) else c["text"] for c in chunks]
|
||||||
vectors = embeddings.embed_texts(chunk_texts)
|
chunk_metas = []
|
||||||
|
for idx, c in enumerate(chunks):
|
||||||
|
if isinstance(c, str):
|
||||||
|
chunk_metas.append(None)
|
||||||
|
else:
|
||||||
|
meta = {k: v for k, v in c.items() if k != "text"} or None
|
||||||
|
chunk_metas.append(meta)
|
||||||
|
|
||||||
for idx, (chunk_text, vector) in enumerate(zip(chunk_texts, vectors)):
|
enriched_texts = [
|
||||||
metadata = None
|
build_enriched_text(title, ct, cm)
|
||||||
if not isinstance(chunks[idx], str):
|
for ct, cm in zip(chunk_texts, chunk_metas)
|
||||||
metadata = {
|
]
|
||||||
k: v for k, v in chunks[idx].items() if k != "text"
|
vectors = embeddings.embed_texts(enriched_texts)
|
||||||
} or None
|
|
||||||
|
for idx, (chunk_text, enriched, vector) in enumerate(
|
||||||
|
zip(chunk_texts, enriched_texts, vectors)
|
||||||
|
):
|
||||||
chunk_id = database.insert_chunk(
|
chunk_id = database.insert_chunk(
|
||||||
conn,
|
conn,
|
||||||
document_id=doc_id,
|
document_id=doc_id,
|
||||||
chunk_index=idx,
|
chunk_index=idx,
|
||||||
text=chunk_text,
|
text=chunk_text,
|
||||||
metadata=metadata,
|
enriched_text=enriched,
|
||||||
|
metadata=chunk_metas[idx],
|
||||||
)
|
)
|
||||||
database.insert_embedding(conn, chunk_id, vector)
|
database.insert_embedding(conn, chunk_id, vector)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
schema: spec-driven
|
||||||
|
created: 2026-03-29
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
## Context
|
||||||
|
|
||||||
|
When a document is ingested, the worker chunks its content and stores each chunk's text in the `chunks` table. FTS5 triggers index that text, and the embedding model embeds it. The document title is stored only in `documents.title` — it never participates in search. This means short documents (or documents whose content lacks the title keywords) are invisible to queries that match the title.
|
||||||
|
|
||||||
|
The reindex endpoint (`POST /api/v1/reindex`) currently reads `chunks.text` and re-embeds it. Any fix must apply consistently at both ingestion and reindex time.
|
||||||
|
|
||||||
|
## Goals / Non-Goals
|
||||||
|
|
||||||
|
**Goals:**
|
||||||
|
- Document titles are searchable via both FTS5 and vector search
|
||||||
|
- Section header breadcrumbs (when present in chunk metadata) are also searchable
|
||||||
|
- Search results continue to return the original chunk text (no title prefix in the `text` field returned to clients)
|
||||||
|
- Existing documents become searchable by title after a `kb reindex`
|
||||||
|
- No schema-breaking migration — additive column only
|
||||||
|
|
||||||
|
**Non-Goals:**
|
||||||
|
- Changing the chunking strategies themselves (note, markdown, code, docling)
|
||||||
|
- Adding a separate title-search endpoint or client-side title filtering
|
||||||
|
- Changing the search result JSON structure
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
### 1. Add an `enriched_text` column to the `chunks` table
|
||||||
|
|
||||||
|
Store the title-prefixed text in a new `chunks.enriched_text` column alongside the existing `chunks.text`. The `text` column remains the raw chunk content (used for display in search results). The `enriched_text` column holds `"{title}\n\n{section_header}\n\n{text}"` (with section_header omitted when absent).
|
||||||
|
|
||||||
|
**Why not just modify `chunks.text`?** The title would then appear in every search result's text field, which is redundant (title is already a separate field) and would confuse consumers that display results.
|
||||||
|
|
||||||
|
**Why not reconstruct enriched text on-the-fly at search time?** FTS5 uses an external content table and triggers — it needs a real column to index. Reconstructing via JOIN at FTS query time would defeat the purpose of the FTS index.
|
||||||
|
|
||||||
|
### 2. Point FTS5 at `enriched_text` instead of `text`
|
||||||
|
|
||||||
|
Update the FTS5 virtual table definition and its sync triggers to index `enriched_text` rather than `text`. This is the core change that makes titles searchable via keyword search.
|
||||||
|
|
||||||
|
Since FTS5 external content tables cannot be ALTERed, existing databases require a rebuild: drop and recreate `chunks_fts` and its triggers, then repopulate. This is handled as a schema migration in `init_schema`.
|
||||||
|
|
||||||
|
### 3. Embed `enriched_text` instead of `text`
|
||||||
|
|
||||||
|
At ingestion time, pass `enriched_text` values to `embed_texts()` instead of raw chunk text. At reindex time, read `enriched_text` from the database. This makes titles searchable via vector similarity too.
|
||||||
|
|
||||||
|
### 4. Build enriched text in the worker, not in the ingest modules
|
||||||
|
|
||||||
|
The enrichment format is: `"{title}\n\n{chunk_text}"` or `"{title} > {section_header}\n\n{chunk_text}"` when a section header exists in chunk metadata.
|
||||||
|
|
||||||
|
This happens in `worker._process_job()` after chunking and before embedding/insertion. The ingest modules remain unchanged — they continue to return raw chunk text and metadata.
|
||||||
|
|
||||||
|
### 5. Schema migration adds `enriched_text` and rebuilds FTS
|
||||||
|
|
||||||
|
The `init_schema` function will:
|
||||||
|
1. Add `enriched_text TEXT` column to `chunks` if missing
|
||||||
|
2. Backfill `enriched_text` from existing data (join with `documents.title` and chunk metadata)
|
||||||
|
3. Drop and recreate `chunks_fts` to index `enriched_text` instead of `text`
|
||||||
|
4. Recreate the FTS sync triggers
|
||||||
|
|
||||||
|
This is safe because the migration only runs when the column is missing (first startup after upgrade). The backfill uses a single UPDATE...FROM query.
|
||||||
|
|
||||||
|
## Risks / Trade-offs
|
||||||
|
|
||||||
|
**Slightly larger database** — Each chunk stores the title string twice (once in `enriched_text`, once via the document FK). For a typical KB with short titles this is negligible (< 1% size increase).
|
||||||
|
→ Acceptable for the search quality improvement.
|
||||||
|
|
||||||
|
**FTS rebuild on upgrade** — First startup after upgrade will rebuild the FTS index, which takes a few seconds for large KBs.
|
||||||
|
→ This is a one-time cost and happens automatically.
|
||||||
|
|
||||||
|
**Embedding drift** — Existing vector embeddings won't include title context until `kb reindex` is run. The FTS backfill happens automatically, but vectors require an explicit reindex.
|
||||||
|
→ Document this in release notes. The FTS improvement alone is a significant win even without reindexing vectors.
|
||||||
|
|
||||||
|
**Title changes not propagated** — If a document's title were ever updated, `enriched_text` would be stale. Currently the engine has no title-update endpoint, so this is not a concern.
|
||||||
|
→ No mitigation needed now. If title editing is added later, it should update enriched_text.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
## Why
|
||||||
|
|
||||||
|
Short documents and notes are unsearchable when the user's query matches the document title but not the chunk content. For example, a document titled "Suitcase Locks" containing only "Steve = 1234 / Theresa = 4567" is invisible to both FTS and vector search for the query "suitcase locks". This is because chunk text — the only thing indexed and embedded — does not include the document title. This is a standard RAG deficiency that most pipelines solve by prepending title context to each chunk.
|
||||||
|
|
||||||
|
## What Changes
|
||||||
|
|
||||||
|
- **Prepend document title to chunk text at ingestion time**: Before embedding and FTS indexing, each chunk's text will be prefixed with the document title (e.g., `"Suitcase Locks\n\n Steve = 363..."`). This ensures the title participates in both full-text and semantic search.
|
||||||
|
- **Include section header context in chunk text**: For chunks that have a `section_header` in their metadata, prepend the header breadcrumb too (e.g., `"DCG Lab Hardware > GRIMDAWN > motherboard\n\nMSI X870 Tomahawk..."`). This improves search for queries that reference section names.
|
||||||
|
- **Store the raw chunk text separately from the enriched text**: The original chunk text (without title prefix) must remain accessible so that search results don't display the prepended title redundantly — the title is already returned as a separate field.
|
||||||
|
- **Reindex command must apply the same enrichment**: When `kb reindex` re-embeds all chunks, it must reconstruct the enriched text (title + section header + chunk text) from stored metadata.
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
### New Capabilities
|
||||||
|
- `chunk-enrichment`: Prepending document title and section context to chunk text before indexing and embedding, while preserving the original text for display.
|
||||||
|
|
||||||
|
### Modified Capabilities
|
||||||
|
- `engine-api`: The search endpoint's returned `text` field must continue to show the original chunk text (without the prepended title), so no visible API change, but the internal indexing behaviour changes. The reindex endpoint must apply enrichment consistently.
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
|
||||||
|
- **Engine ingestion pipeline** (`worker.py`): The `_process_job` function must build enriched text from title + section headers + chunk text before passing to `embed_texts()` and `insert_chunk()`.
|
||||||
|
- **Database schema** (`database.py`): Need to store both raw `text` (for display) and enriched `text` (for FTS/embedding), or reconstruct enriched text at index time. Simplest approach: store raw text in `chunks.text`, use enriched text only for FTS content and embedding vectors.
|
||||||
|
- **FTS triggers** (`database.py`): The FTS5 external content table currently mirrors `chunks.text`. If we add an `enriched_text` column, the FTS index should be built from that instead.
|
||||||
|
- **Reindex flow** (`worker.py` / `database.py`): Must reconstruct enriched text by joining chunk metadata with document title.
|
||||||
|
- **Search result enrichment** (`routes/search.py`): No change needed — results already return `chunks.text` (raw) and `documents.title` separately.
|
||||||
|
- **All four ingest modules** (`note.py`, `markdown.py`, `code.py`, `docling_pipeline.py`): No changes needed — enrichment happens after chunking, in the worker.
|
||||||
|
- **Existing documents**: Require a `reindex` to benefit from the new enrichment. No data migration needed since the original text is preserved.
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
## ADDED Requirements
|
||||||
|
|
||||||
|
### Requirement: Chunk text enrichment with document title
|
||||||
|
|
||||||
|
The engine SHALL prepend the document title to each chunk's text before FTS indexing and vector embedding. The enriched text SHALL be stored in a dedicated `enriched_text` column on the `chunks` table. The original chunk text SHALL remain in the `text` column for display purposes.
|
||||||
|
|
||||||
|
The enrichment format SHALL be:
|
||||||
|
- Without section header: `"{title}\n\n{chunk_text}"`
|
||||||
|
- With section header: `"{title} > {section_header}\n\n{chunk_text}"`
|
||||||
|
|
||||||
|
Where `section_header` is the value from the chunk's metadata `section_header` field, when present.
|
||||||
|
|
||||||
|
#### Scenario: Note ingestion with title enrichment
|
||||||
|
- **WHEN** a note titled "Suitcase Locks" with content "Steve = 363" is ingested
|
||||||
|
- **THEN** the `chunks.text` column SHALL contain "Steve = 363" and the `chunks.enriched_text` column SHALL contain "Suitcase Locks\n\nSteve = 363"
|
||||||
|
|
||||||
|
#### Scenario: Markdown chunk with section header enrichment
|
||||||
|
- **WHEN** a markdown document titled "DCG Lab Hardware" produces a chunk with section_header "GRIMDAWN > motherboard" and text "MSI X870 Tomahawk"
|
||||||
|
- **THEN** the `chunks.enriched_text` SHALL contain "DCG Lab Hardware > GRIMDAWN > motherboard\n\nMSI X870 Tomahawk"
|
||||||
|
|
||||||
|
#### Scenario: Chunk without section header
|
||||||
|
- **WHEN** a document titled "Docker Tips" produces a chunk with no section_header in metadata and text "dbash() { docker exec -it $1 bash; }"
|
||||||
|
- **THEN** the `chunks.enriched_text` SHALL contain "Docker Tips\n\ndbash() { docker exec -it $1 bash; }"
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Requirement: FTS5 indexes enriched text
|
||||||
|
|
||||||
|
The FTS5 virtual table `chunks_fts` SHALL index the `enriched_text` column instead of the `text` column. All FTS sync triggers (insert, update, delete) SHALL operate on `enriched_text`.
|
||||||
|
|
||||||
|
#### Scenario: FTS search matches document title
|
||||||
|
- **WHEN** a user searches for "suitcase locks" and a document titled "Suitcase Locks" exists with chunk text "Steve = 363"
|
||||||
|
- **THEN** the FTS5 search SHALL return that chunk as a match
|
||||||
|
|
||||||
|
#### Scenario: FTS search still matches chunk content
|
||||||
|
- **WHEN** a user searches for "MSI X870" and a chunk contains that text in its body
|
||||||
|
- **THEN** the FTS5 search SHALL return that chunk as a match (enrichment does not break content matching)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Requirement: Vector embeddings use enriched text
|
||||||
|
|
||||||
|
The embedding model SHALL receive `enriched_text` (not raw `text`) when generating vectors during both initial ingestion and reindex operations.
|
||||||
|
|
||||||
|
#### Scenario: Vector search matches document title
|
||||||
|
- **WHEN** a user searches semantically for "luggage combination codes" and a document titled "Suitcase Locks" exists
|
||||||
|
- **THEN** the vector search SHALL return that chunk with higher similarity than it would without title enrichment
|
||||||
|
|
||||||
|
#### Scenario: Reindex uses enriched text
|
||||||
|
- **WHEN** `POST /api/v1/reindex` is called
|
||||||
|
- **THEN** the engine SHALL read `enriched_text` from the chunks table and embed that (not `text`)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Requirement: Schema migration adds enriched_text column
|
||||||
|
|
||||||
|
On startup, `init_schema` SHALL add the `enriched_text` column to the `chunks` table if it does not exist. It SHALL then backfill `enriched_text` for all existing chunks by joining with `documents.title` and parsing chunk metadata for section headers. It SHALL rebuild the FTS5 table and triggers to index `enriched_text`.
|
||||||
|
|
||||||
|
#### Scenario: First startup after upgrade
|
||||||
|
- **WHEN** the engine starts and `chunks.enriched_text` column does not exist
|
||||||
|
- **THEN** the engine SHALL add the column, backfill all rows, drop and recreate `chunks_fts` to index `enriched_text`, and recreate the FTS sync triggers
|
||||||
|
|
||||||
|
#### Scenario: Subsequent startup
|
||||||
|
- **WHEN** the engine starts and `chunks.enriched_text` column already exists
|
||||||
|
- **THEN** the engine SHALL not perform any migration and start normally
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Requirement: Search results return raw text
|
||||||
|
|
||||||
|
Search results SHALL continue to return the original chunk text (from `chunks.text`) in the `text` field, not the enriched text. The document title is already returned as a separate `title` field.
|
||||||
|
|
||||||
|
#### Scenario: Search result text field
|
||||||
|
- **WHEN** a search returns a chunk from document "Suitcase Locks" with raw text "Steve = 363"
|
||||||
|
- **THEN** the result `text` field SHALL be "Steve = 363" (not "Suitcase Locks\n\nSteve = 363")
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
## MODIFIED Requirements
|
||||||
|
|
||||||
|
### Requirement: Background ingestion worker
|
||||||
|
|
||||||
|
The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.
|
||||||
|
|
||||||
|
#### Scenario: Successful PDF ingestion
|
||||||
|
- **WHEN** the background worker picks up a queued PDF job
|
||||||
|
- **THEN** it SHALL update the job status to `processing`, run Docling conversion and chunking, build enriched text for each chunk by prepending the document title, embed all chunks using enriched text, insert document and chunks into the database, move the staged file to `{data_dir}/documents/{content_hash}.pdf`, update `documents.stored_path` with the permanent path, store the original filename in `documents.original_filename`, update the job status to `done` with the resulting document_id and chunk count, and clean up the staging entry
|
||||||
|
|
||||||
|
#### Scenario: Ingestion failure
|
||||||
|
- **WHEN** the background worker encounters an error during processing (e.g., corrupt PDF)
|
||||||
|
- **THEN** it SHALL update the job status to `failed` with the error message, delete the staged file, and continue processing the next queued job
|
||||||
|
|
||||||
|
#### Scenario: Search during active ingestion
|
||||||
|
- **WHEN** a search request arrives while the background worker is processing a job
|
||||||
|
- **THEN** the search SHALL execute without blocking (SQLite WAL mode) and return results from already-ingested documents
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Requirement: Engine status and reindex
|
||||||
|
|
||||||
|
The engine SHALL provide status information and support re-embedding all chunks. The `version` field in the status response SHALL always be present and SHALL reflect the engine's release version as read from the `VERSION` file. This field is the contract used by clients for compatibility checking.
|
||||||
|
|
||||||
|
#### Scenario: Get engine status
|
||||||
|
- **WHEN** a client sends `GET /api/v1/status`
|
||||||
|
- **THEN** the engine SHALL return JSON with `version` (string, from VERSION file), model_name, embedding_dim, GPU device info, database stats (document count by type, total chunks, DB size), and queue stats (queued/processing job count)
|
||||||
|
|
||||||
|
#### Scenario: Trigger reindex
|
||||||
|
- **WHEN** a client sends `POST /api/v1/reindex`
|
||||||
|
- **THEN** the engine SHALL re-embed all existing chunks using the `enriched_text` column and the currently loaded model, and return progress information. This operation SHALL NOT block search queries.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
## 1. Schema Migration
|
||||||
|
|
||||||
|
- [x] 1.1 Add `enriched_text TEXT` column to `chunks` table in `database.py:init_schema` (with migration check for existing DBs)
|
||||||
|
- [x] 1.2 Write backfill query: `UPDATE chunks SET enriched_text = ... FROM documents` joining title and parsing chunk metadata for section_header
|
||||||
|
- [x] 1.3 Drop and recreate `chunks_fts` virtual table to index `enriched_text` instead of `text`
|
||||||
|
- [x] 1.4 Update FTS sync triggers (`chunks_ai`, `chunks_ad`, `chunks_au`) to use `enriched_text`
|
||||||
|
|
||||||
|
## 2. Enrichment Helper
|
||||||
|
|
||||||
|
- [x] 2.1 Create `build_enriched_text(title: str, chunk_text: str, metadata: dict | None) -> str` helper function in `worker.py` (or a shared util) that formats `"{title} > {section_header}\n\n{chunk_text}"` or `"{title}\n\n{chunk_text}"`
|
||||||
|
|
||||||
|
## 3. Ingestion Pipeline
|
||||||
|
|
||||||
|
- [x] 3.1 Update `worker._process_job()` to build enriched text for each chunk after chunking
|
||||||
|
- [x] 3.2 Pass enriched text to `embed_texts()` instead of raw chunk text
|
||||||
|
- [x] 3.3 Pass enriched text to `database.insert_chunk()` as the new `enriched_text` parameter
|
||||||
|
- [x] 3.4 Update `database.insert_chunk()` to accept and store `enriched_text`
|
||||||
|
|
||||||
|
## 4. Reindex
|
||||||
|
|
||||||
|
- [x] 4.1 Update `routes/reindex.py` to read `enriched_text` from chunks table and embed that instead of `text`
|
||||||
|
|
||||||
|
## 5. Search Results
|
||||||
|
|
||||||
|
- [x] 5.1 Verify `search.py:_enrich()` returns `chunks.text` (raw) not `enriched_text` — no change expected, but confirm
|
||||||
|
|
||||||
|
## 6. Testing
|
||||||
|
|
||||||
|
- [x] 6.1 Test: ingest a short note with a descriptive title, search by title keywords, confirm it is found
|
||||||
|
- [x] 6.2 Test: ingest a markdown doc, search by section header, confirm chunks are found
|
||||||
|
- [x] 6.3 Test: verify search result `text` field does not contain the prepended title
|
||||||
|
- [x] 6.4 Test: run `reindex`, verify enriched text is used for new embeddings
|
||||||
|
- [x] 6.5 Test: verify schema migration backfills enriched_text for pre-existing chunks on startup
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
schema: spec-driven
|
||||||
|
created: 2026-03-29
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
## Context
|
||||||
|
|
||||||
|
The root cobra command in `client/cmd/root.go` uses `cobra.ArbitraryArgs` and its `RunE` handler to catch any arguments not matching a subcommand. Currently, any non-empty args are joined and submitted as a note. This means a single mistyped word (e.g., `kb infow` instead of `kb info`) silently creates a junk note in the knowledge base.
|
||||||
|
|
||||||
|
## Goals / Non-Goals
|
||||||
|
|
||||||
|
**Goals:**
|
||||||
|
- Prevent single bare words from being silently ingested as notes
|
||||||
|
- Provide a clear error message that helps the user correct their input
|
||||||
|
- Preserve the multi-word implicit note shorthand (`kb remember to update dns`)
|
||||||
|
|
||||||
|
**Non-Goals:**
|
||||||
|
- Detecting "close matches" to real commands (fuzzy matching / did-you-mean)
|
||||||
|
- Changing how quoted strings work at the shell level (we can't detect quotes after shell expansion)
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
### Guard on argument count in RunE
|
||||||
|
|
||||||
|
When `len(args) == 1`, reject with an error message instead of submitting as a note. When `len(args) > 1`, continue treating as implicit note shorthand.
|
||||||
|
|
||||||
|
**Rationale**: This is the simplest reliable heuristic. The shell strips quotes before cobra sees args, so we cannot distinguish `kb "singleword"` from `kb singleword`. However, single-word notes are rare in practice, and the error message tells the user how to work around it (use multiple words or the full note workflow). Multi-word input is almost certainly intentional note text, not a mistyped command.
|
||||||
|
|
||||||
|
**Alternative considered**: Checking against a list of known subcommand names — rejected because it wouldn't catch typos of commands we don't know about and adds maintenance burden.
|
||||||
|
|
||||||
|
## Risks / Trade-offs
|
||||||
|
|
||||||
|
- **Single-word notes no longer work via shorthand** → Users must use `kb add --note "singleword"` or include additional words. This is an acceptable trade-off since single-word notes are uncommon and the error message is clear.
|
||||||
|
- **Shell quote stripping means we can't be perfect** → `kb "my note"` with exactly one word after quote removal will be rejected. This is a known limitation but very rare in practice.
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
## Why
|
||||||
|
|
||||||
|
A single unquoted word passed to `kb` (e.g., `kb infow`) is silently treated as a note and ingested. This is almost always a mistyped command, not an intentional note. Users lose trust when typos pollute their knowledge base.
|
||||||
|
|
||||||
|
## What Changes
|
||||||
|
|
||||||
|
- The implicit note shorthand will require **more than one argument** to be treated as a note. A single bare word will be rejected with a helpful error suggesting the user check their command or quote a multi-word note.
|
||||||
|
- This is a **BREAKING** change to the implicit note shorthand: `kb singleword` no longer creates a note. Users must write `kb "singleword is important"` or use multiple words.
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
### New Capabilities
|
||||||
|
|
||||||
|
_(none)_
|
||||||
|
|
||||||
|
### Modified Capabilities
|
||||||
|
|
||||||
|
- `go-client`: The "Implicit note shorthand" requirement changes to reject single-word bare arguments and print an error instead of submitting them as notes.
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
|
||||||
|
- **Code**: `client/cmd/root.go` — `RunE` handler for the root command
|
||||||
|
- **Tests**: `client/cmd/root_test.go` or equivalent — add/update tests for single-word rejection
|
||||||
|
- **Users**: Anyone who intentionally used `kb singleword` as a note shorthand will need to use multiple words or quotes
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
## MODIFIED Requirements
|
||||||
|
|
||||||
|
### Requirement: Implicit note shorthand
|
||||||
|
|
||||||
|
The client SHALL treat bare string arguments (with no subcommand) as an implicit note only when **more than one argument** is provided. `kb "my note"` SHALL behave identically to submitting a note via `POST /api/v1/jobs`. All persistent flags (`--format`, `--engine`, `--api-key`) and the root `--tags` flag SHALL work with the shorthand form. A single bare word SHALL be rejected with an error message.
|
||||||
|
|
||||||
|
#### Scenario: Quick note via bare argument
|
||||||
|
- **WHEN** the user runs `kb "remember to update DNS"`
|
||||||
|
- **THEN** the client SHALL submit the text as a note via `POST /api/v1/jobs` and print `Queued: note`
|
||||||
|
|
||||||
|
#### Scenario: Bare argument with tags
|
||||||
|
- **WHEN** the user runs `kb "server room is building 3" --tags ops`
|
||||||
|
- **THEN** the client SHALL submit the note with the specified tags
|
||||||
|
|
||||||
|
#### Scenario: Bare argument with JSON output
|
||||||
|
- **WHEN** the user runs `kb "my note" --format json`
|
||||||
|
- **THEN** the client SHALL output the raw JSON response from the engine
|
||||||
|
|
||||||
|
#### Scenario: Bare argument duplicate detection
|
||||||
|
- **WHEN** the user runs `kb "my note"` and the engine returns HTTP 409
|
||||||
|
- **THEN** the client SHALL handle the duplicate response identically to the previous `kb add --note` behaviour
|
||||||
|
|
||||||
|
#### Scenario: Multiple unquoted words
|
||||||
|
- **WHEN** the user runs `kb remember to update dns` (without quotes)
|
||||||
|
- **THEN** the client SHALL join all arguments into a single note string and submit it
|
||||||
|
|
||||||
|
#### Scenario: Single bare word rejected
|
||||||
|
- **WHEN** the user runs `kb infow` (a single unrecognized word)
|
||||||
|
- **THEN** the client SHALL print to stderr: `Unknown command "infow". Run 'kb --help' for available commands.` followed by a hint about note usage, and exit with a non-zero code
|
||||||
|
|
||||||
|
#### Scenario: No interference with subcommands
|
||||||
|
- **WHEN** the user runs `kb search "query"` or any other existing subcommand
|
||||||
|
- **THEN** the client SHALL route to the subcommand as before — the implicit note shorthand SHALL NOT interfere
|
||||||
|
|
||||||
|
#### Scenario: No arguments
|
||||||
|
- **WHEN** the user runs `kb` with no arguments
|
||||||
|
- **THEN** the client SHALL display the help text
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
## 1. Core Implementation
|
||||||
|
|
||||||
|
- [x] 1.1 Update `RunE` in `client/cmd/root.go` to reject single-word bare arguments with an error message and non-zero exit
|
||||||
|
- [x] 1.2 Update usage template in `root.go` to reflect that note shorthand requires multiple words
|
||||||
|
|
||||||
|
## 2. Tests
|
||||||
|
|
||||||
|
- [x] 2.1 Add test: single bare word prints error to stderr and exits non-zero
|
||||||
|
- [x] 2.2 Add test: multiple bare words are submitted as a note (existing behavior preserved)
|
||||||
|
- [x] 2.3 Add test: zero arguments shows help (existing behavior preserved)
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
schema: spec-driven
|
||||||
|
created: 2026-03-31
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
## Context
|
||||||
|
|
||||||
|
README.md currently serves as a single documentation file for both users and developers. It contains ~290 lines mixing installation/usage instructions with build-from-source steps, release scripts, Docker image internals, and developer notes (e.g., ROCm migration plans). There is no DEVELOPER.md or CONTRIBUTING.md file.
|
||||||
|
|
||||||
|
## Goals / Non-Goals
|
||||||
|
|
||||||
|
**Goals:**
|
||||||
|
- Separate user-facing documentation (README.md) from developer-facing documentation (DEVELOPER.md)
|
||||||
|
- README.md should answer: "What is this? How do I install it? How do I use it?"
|
||||||
|
- DEVELOPER.md should answer: "How do I build from source? How do I release? How do I contribute?"
|
||||||
|
- Provide a clear cross-reference link between the two files
|
||||||
|
|
||||||
|
**Non-Goals:**
|
||||||
|
- Rewriting or improving documentation content itself (just moving it)
|
||||||
|
- Creating additional docs files (CONTRIBUTING.md, architecture docs, etc.)
|
||||||
|
- Changing any code, build scripts, or CI configuration
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
### 1. Single DEVELOPER.md file (not multiple docs files)
|
||||||
|
|
||||||
|
All developer content goes into one top-level DEVELOPER.md rather than a `docs/` directory or separate CONTRIBUTING.md / BUILDING.md files. The total developer content is small enough (~80 lines) that splitting further would be unnecessary overhead. A single file at the repo root is immediately discoverable.
|
||||||
|
|
||||||
|
**Alternative considered**: `docs/` directory with multiple files. Rejected because the content volume doesn't justify the structure, and root-level DEVELOPER.md is a well-known convention.
|
||||||
|
|
||||||
|
### 2. Content split boundary
|
||||||
|
|
||||||
|
Content stays in README.md if it's needed by someone who just wants to **run** kb. Content moves to DEVELOPER.md if it's only needed by someone who wants to **build, modify, or release** kb.
|
||||||
|
|
||||||
|
Specifically moving to DEVELOPER.md:
|
||||||
|
- "From source" subsections under both engine and client install
|
||||||
|
- Entire "Building and releasing" section (release scripts, version checking, Docker image tags, registry overrides)
|
||||||
|
- "Future: ROCm runtime migration" developer note
|
||||||
|
|
||||||
|
Staying in README.md:
|
||||||
|
- Architecture overview (helps users understand what they're running)
|
||||||
|
- Pre-built image / release install instructions
|
||||||
|
- Client configuration
|
||||||
|
- Usage examples
|
||||||
|
- Engine configuration table
|
||||||
|
- Data portability
|
||||||
|
- API reference
|
||||||
|
- Claude Code skill reference
|
||||||
|
|
||||||
|
### 3. Cross-reference approach
|
||||||
|
|
||||||
|
A short note in README.md's Quick Start section pointing to DEVELOPER.md for building from source. No back-link needed from DEVELOPER.md since developers will naturally find README.md first.
|
||||||
|
|
||||||
|
## Risks / Trade-offs
|
||||||
|
|
||||||
|
- **[Stale cross-references]** If DEVELOPER.md sections are renamed, the link from README.md could break. Mitigation: link to the file, not to a specific anchor.
|
||||||
|
- **[Discoverability]** Some users who want to build from source might miss DEVELOPER.md. Mitigation: explicit "See DEVELOPER.md" callout in the Quick Start section where "from source" instructions used to be.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
## Why
|
||||||
|
|
||||||
|
README.md currently mixes user-facing content (what kb does, how to install and use it) with developer-facing content (building from source, releasing, Docker image internals, architecture deep-dives). Users looking for quick-start instructions have to scroll past release scripts and build commands. Developers looking for contribution/build info have to hunt through user docs. Splitting these into README.md (users) and DEVELOPER.md (developers/contributors) follows standard open-source convention and makes both audiences' experience cleaner.
|
||||||
|
|
||||||
|
## What Changes
|
||||||
|
|
||||||
|
- **Trim README.md** to focus on user-facing content: what kb is, how to install (from pre-built images/releases), how to configure, how to use, engine configuration reference, data portability, and API reference.
|
||||||
|
- **Remove "from source" build instructions** from README.md (both engine and client sections).
|
||||||
|
- **Remove "Building and releasing" section** from README.md entirely.
|
||||||
|
- **Remove "Future: ROCm runtime migration"** developer note from README.md.
|
||||||
|
- **Create DEVELOPER.md** containing: building engine from source, building client from source, release process (client and engine), Docker image details, version checking, ROCm migration notes, and any other contributor-oriented content.
|
||||||
|
- **Add a link** from README.md to DEVELOPER.md for developers who want to build from source or contribute.
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
### New Capabilities
|
||||||
|
- `developer-docs`: Developer-facing documentation covering building from source, releasing, and contributing.
|
||||||
|
|
||||||
|
### Modified Capabilities
|
||||||
|
|
||||||
|
(none - no spec-level behavior changes, this is a documentation restructuring)
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
|
||||||
|
- **Files modified**: `README.md` (trimmed)
|
||||||
|
- **Files created**: `DEVELOPER.md` (new)
|
||||||
|
- **No code changes**: purely documentation restructuring
|
||||||
|
- **No API changes**: no functional impact
|
||||||
+63
@@ -0,0 +1,63 @@
|
|||||||
|
## ADDED Requirements
|
||||||
|
|
||||||
|
### Requirement: DEVELOPER.md exists at repo root
|
||||||
|
The repository SHALL have a `DEVELOPER.md` file at the project root containing all developer-facing documentation.
|
||||||
|
|
||||||
|
#### Scenario: File exists
|
||||||
|
- **WHEN** a developer navigates to the repository root
|
||||||
|
- **THEN** a `DEVELOPER.md` file SHALL be present
|
||||||
|
|
||||||
|
### Requirement: DEVELOPER.md contains build-from-source instructions
|
||||||
|
DEVELOPER.md SHALL contain instructions for building both the engine and client from source.
|
||||||
|
|
||||||
|
#### Scenario: Engine build from source
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include instructions for starting the engine from source using compose files (both NVIDIA and ROCm)
|
||||||
|
|
||||||
|
#### Scenario: Client build from source
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include instructions for building the client binary from source using `make build` and `make all`
|
||||||
|
|
||||||
|
### Requirement: DEVELOPER.md contains release process
|
||||||
|
DEVELOPER.md SHALL document the release process for both client and engine, including release scripts, version bumping, and Docker image tagging.
|
||||||
|
|
||||||
|
#### Scenario: Client release documentation
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include `release-client.sh` usage with flag options (--gitea, --github, --minor, --no-increment, --dry-run)
|
||||||
|
|
||||||
|
#### Scenario: Engine release documentation
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include `release-engine.sh` usage with flag options and Docker image tag conventions
|
||||||
|
|
||||||
|
#### Scenario: Version checking
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include how to check client and engine versions
|
||||||
|
|
||||||
|
### Requirement: DEVELOPER.md contains developer notes
|
||||||
|
DEVELOPER.md SHALL include any forward-looking developer notes such as migration plans or technical debt items.
|
||||||
|
|
||||||
|
#### Scenario: ROCm migration note
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include the ROCm runtime migration note about onnxruntime and MIGraphX
|
||||||
|
|
||||||
|
### Requirement: README.md excludes developer-only content
|
||||||
|
README.md SHALL NOT contain build-from-source instructions, release processes, or developer-only notes.
|
||||||
|
|
||||||
|
#### Scenario: No from-source build steps in README
|
||||||
|
- **WHEN** a user reads README.md
|
||||||
|
- **THEN** there SHALL be no "From source" subsections under engine or client installation
|
||||||
|
|
||||||
|
#### Scenario: No release section in README
|
||||||
|
- **WHEN** a user reads README.md
|
||||||
|
- **THEN** there SHALL be no "Building and releasing" section
|
||||||
|
|
||||||
|
#### Scenario: No developer notes in README
|
||||||
|
- **WHEN** a user reads README.md
|
||||||
|
- **THEN** there SHALL be no "Future: ROCm runtime migration" section
|
||||||
|
|
||||||
|
### Requirement: README.md cross-references DEVELOPER.md
|
||||||
|
README.md SHALL include a link to DEVELOPER.md for users who want to build from source or contribute.
|
||||||
|
|
||||||
|
#### Scenario: Developer link in quick start
|
||||||
|
- **WHEN** a user reads the Quick Start section of README.md
|
||||||
|
- **THEN** there SHALL be a note pointing to DEVELOPER.md for building from source
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
## 1. Create DEVELOPER.md
|
||||||
|
|
||||||
|
- [x] 1.1 Create DEVELOPER.md at repo root with engine build-from-source instructions (compose.nvidia.yaml and compose.rocm.yaml)
|
||||||
|
- [x] 1.2 Add client build-from-source instructions (make build, make all)
|
||||||
|
- [x] 1.3 Add "Building and releasing" section: release-client.sh and release-engine.sh usage with all flag options
|
||||||
|
- [x] 1.4 Add version checking instructions (kb --version, curl status endpoint)
|
||||||
|
- [x] 1.5 Add Docker image tag conventions and registry override documentation
|
||||||
|
- [x] 1.6 Add "Future: ROCm runtime migration" developer note
|
||||||
|
|
||||||
|
## 2. Trim README.md
|
||||||
|
|
||||||
|
- [x] 2.1 Remove "From source (for development)" subsection under engine quick start
|
||||||
|
- [x] 2.2 Remove "From source (for development)" subsection under client installation
|
||||||
|
- [x] 2.3 Remove entire "Building and releasing" section
|
||||||
|
- [x] 2.4 Remove "Future: ROCm runtime migration" section
|
||||||
|
- [x] 2.5 Add cross-reference note to DEVELOPER.md in the Quick Start section for building from source
|
||||||
|
- [x] 2.6 Move API reference section from README.md to DEVELOPER.md
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
schema: spec-driven
|
||||||
|
created: 2026-03-31
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
## Context
|
||||||
|
|
||||||
|
The kb client currently overloads the root Cobra command to handle both command dispatch and implicit note ingestion. Any unrecognized multi-word input is silently submitted as a note via `POST /api/v1/jobs`. This was introduced to reduce friction for note-taking but has proven error-prone — typos in commands create unwanted notes. A single-word guard was added but multi-word typos still slip through.
|
||||||
|
|
||||||
|
The root command has: custom `ArbitraryArgs` validation, a `RunE` with arg-count branching, a `--tags` flag for the note shorthand, a custom usage template with `isRootCmd` template function, and `submitNote()` living in `add.go`.
|
||||||
|
|
||||||
|
## Goals / Non-Goals
|
||||||
|
|
||||||
|
**Goals:**
|
||||||
|
- Eliminate accidental note creation from mistyped commands
|
||||||
|
- Provide a clean, explicit `addnote` command that pairs with existing `addfile`
|
||||||
|
- Revert root command to standard Cobra behaviour (no custom args, no custom template)
|
||||||
|
- Keep the same API contract — `POST /api/v1/jobs` with `note` field unchanged
|
||||||
|
|
||||||
|
**Non-Goals:**
|
||||||
|
- Changing the engine API
|
||||||
|
- Modifying `addfile` behaviour
|
||||||
|
- Adding new content types (url, bookmark, etc.)
|
||||||
|
- Backward compatibility shim for `kb "text"` syntax
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
### 1. New `addnote` command in its own file
|
||||||
|
|
||||||
|
Create `client/cmd/addnote.go` with a `cobra.Command` that takes `ExactArgs(1)` — a single quoted string. This mirrors `addfile` which also takes `ExactArgs(1)`.
|
||||||
|
|
||||||
|
**Rationale**: Keeps each command in its own file (consistent with the existing pattern). `ExactArgs(1)` means the user must quote multi-word notes, which is unambiguous and avoids the flag-parsing edge cases that plagued the implicit shorthand.
|
||||||
|
|
||||||
|
**Alternative considered**: Joining `ArbitraryArgs` like the old shorthand. Rejected — this is exactly the ambiguity we're removing.
|
||||||
|
|
||||||
|
### 2. Move `submitNote()` from `add.go` to `addnote.go`
|
||||||
|
|
||||||
|
The function is only used by the addnote command, so it belongs in the same file.
|
||||||
|
|
||||||
|
**Rationale**: `add.go` becomes purely about file operations (it already is, aside from hosting `submitNote()`). Clean separation.
|
||||||
|
|
||||||
|
### 3. Fully revert root command to Cobra defaults
|
||||||
|
|
||||||
|
Remove: `ArbitraryArgs`, custom `RunE` (replace with nil — Cobra shows help by default), `--tags` flag on root, custom usage template, `isRootCmd` template function.
|
||||||
|
|
||||||
|
**Rationale**: The root command should do one thing — dispatch to subcommands. All the custom logic was there to support the implicit shorthand which is being removed.
|
||||||
|
|
||||||
|
### 4. `addnote` gets its own `--tags` flag
|
||||||
|
|
||||||
|
The `--tags` flag moves from the root command to `addnote`, matching how `addfile` already has its own `--tags` flag.
|
||||||
|
|
||||||
|
## Risks / Trade-offs
|
||||||
|
|
||||||
|
- **Breaking change for existing users** → Mitigated by clear error messaging. If someone types `kb "some text"`, Cobra will say "unknown command". The `examples` command will show the new syntax.
|
||||||
|
- **Slightly more typing for notes** (`kb addnote "text"` vs `kb "text"`) → Acceptable trade-off for eliminating accidental ingestion. Tab-completion helps.
|
||||||
|
- **Scripts using old syntax will break** → This is intentional. The old syntax was a foot-gun.
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
## Why
|
||||||
|
|
||||||
|
The implicit note shorthand (`kb "some text"`) makes it too easy to accidentally add notes when mistyping commands. Despite the single-word guard, any multi-word typo (e.g. `kb lisst --type pdf`) silently creates a note. The root command doing double-duty as both command dispatcher and note ingester undermines user trust. Reverting to explicit, structured add commands eliminates accidental ingestion and gives every content type a clear, discoverable verb.
|
||||||
|
|
||||||
|
## What Changes
|
||||||
|
|
||||||
|
- **New `addnote` command**: `kb addnote <text>` takes a single quoted positional argument and submits it as a note. Supports `--tags`. The `submitNote()` logic moves from `root.go` to a new `addnote.go` command file.
|
||||||
|
- **Remove implicit note shorthand**: The root command reverts to standard Cobra behaviour — no `ArbitraryArgs`, no special arg-count logic, no `--tags` flag on root. Unknown input gets Cobra's default "unknown command" error.
|
||||||
|
- **Remove custom usage template**: The root command no longer needs the `isRootCmd` template logic. Standard Cobra usage template for all commands.
|
||||||
|
- **Update examples**: `examples.go` updated to show `kb addnote` instead of bare `kb "text"`.
|
||||||
|
- **Update tests**: Remove implicit note shorthand tests, add `addnote` command tests.
|
||||||
|
- **`addfile` unchanged**: Stays exactly as-is.
|
||||||
|
- **BREAKING**: `kb "note text"` no longer works. Users must use `kb addnote "note text"`.
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
### New Capabilities
|
||||||
|
|
||||||
|
_(none)_
|
||||||
|
|
||||||
|
### Modified Capabilities
|
||||||
|
|
||||||
|
- `go-client`: The "Implicit note shorthand" requirement is removed entirely and replaced by a new "Add note command" requirement. The "Add command (file and note ingestion)" requirement description is updated to reflect `addnote` / `addfile` as the two ingestion commands. The root command reverts to standard Cobra behaviour with no custom arg handling or usage template.
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
|
||||||
|
- `client/cmd/root.go` — remove `ArbitraryArgs`, `RunE` note logic, `--tags` flag, custom usage template, `isRootCmd` template func
|
||||||
|
- `client/cmd/add.go` — `submitNote()` function moves to new `addnote.go` (or stays in `add.go` alongside `addfile` — design decision)
|
||||||
|
- `client/cmd/addnote.go` — new file defining the `addnote` command
|
||||||
|
- `client/cmd/examples.go` — update example text
|
||||||
|
- `client/cmd/root_test.go` — remove implicit note shorthand tests, add standard Cobra behaviour tests
|
||||||
|
- No engine changes — the API contract (`POST /api/v1/jobs` with `note` field) is unchanged
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
## ADDED Requirements
|
||||||
|
|
||||||
|
### Requirement: Add note command
|
||||||
|
|
||||||
|
The client SHALL provide a `kb addnote <text>` command that submits a text note to the engine for ingestion. The command SHALL take exactly one positional argument (the note text) and support a `--tags` flag for comma-separated tags. The note SHALL be submitted via `POST /api/v1/jobs` with the `note` field in a multipart request.
|
||||||
|
|
||||||
|
#### Scenario: Add a note
|
||||||
|
- **WHEN** the user runs `kb addnote "remember to update DNS records"`
|
||||||
|
- **THEN** the client SHALL submit the text as a note via `POST /api/v1/jobs` and print `Queued: note`
|
||||||
|
|
||||||
|
#### Scenario: Add a note with tags
|
||||||
|
- **WHEN** the user runs `kb addnote "server room is building 3" --tags ops`
|
||||||
|
- **THEN** the client SHALL submit the note with the specified tags
|
||||||
|
|
||||||
|
#### Scenario: Add a note with JSON output
|
||||||
|
- **WHEN** the user runs `kb addnote "my note" --format json`
|
||||||
|
- **THEN** the client SHALL output the raw JSON response from the engine
|
||||||
|
|
||||||
|
#### Scenario: Duplicate note detection
|
||||||
|
- **WHEN** the user runs `kb addnote "my note"` and the engine returns HTTP 409
|
||||||
|
- **THEN** the client SHALL display the duplicate information (document ID or job ID) and exit with code 0
|
||||||
|
|
||||||
|
#### Scenario: Missing argument
|
||||||
|
- **WHEN** the user runs `kb addnote` with no arguments
|
||||||
|
- **THEN** the client SHALL display an error indicating that the note text argument is required
|
||||||
|
|
||||||
|
#### Scenario: Too many arguments
|
||||||
|
- **WHEN** the user runs `kb addnote remember to update dns` (unquoted, multiple args)
|
||||||
|
- **THEN** the client SHALL display an error indicating that exactly one argument is required, with a hint to quote the text
|
||||||
|
|
||||||
|
## MODIFIED Requirements
|
||||||
|
|
||||||
|
### Requirement: Add command (file and note ingestion)
|
||||||
|
|
||||||
|
The client SHALL provide a `kb addfile` command that uploads files to the engine for async ingestion. The command SHALL validate file extensions before uploading and reject unsupported types. The client SHALL handle duplicate rejection (HTTP 409) and display the existing document information. Notes are handled by the separate `addnote` command — `addfile` is exclusively for file uploads.
|
||||||
|
|
||||||
|
#### Scenario: Add a single file
|
||||||
|
- **WHEN** the user runs `kb addfile report.pdf`
|
||||||
|
- **THEN** the client SHALL validate the file extension, upload the file via `POST /api/v1/jobs` (multipart), print "Queued: report.pdf", and exit
|
||||||
|
|
||||||
|
#### Scenario: Add a file with tags
|
||||||
|
- **WHEN** the user runs `kb addfile manual.pdf --tags car,maintenance`
|
||||||
|
- **THEN** the client SHALL include the tags in the multipart upload metadata
|
||||||
|
|
||||||
|
#### Scenario: Add a directory recursively
|
||||||
|
- **WHEN** the user runs `kb addfile ~/documents/ --recursive`
|
||||||
|
- **THEN** the client SHALL discover all supported files in the directory tree, upload each one sequentially, and print "Queued: N files"
|
||||||
|
|
||||||
|
#### Scenario: Unsupported file extension
|
||||||
|
- **WHEN** the user runs `kb addfile photo.jpg`
|
||||||
|
- **THEN** the client SHALL print an error listing supported extensions and exit with a non-zero code without making any API call
|
||||||
|
|
||||||
|
#### Scenario: Duplicate file rejected (already ingested)
|
||||||
|
- **WHEN** the user runs `kb addfile report.pdf` and the engine returns HTTP 409 with `{"error": "duplicate", "document_id": 42, "title": "report.pdf"}`
|
||||||
|
- **THEN** the client SHALL print "Already imported: report.pdf (doc ID: 42)" and exit with code 0
|
||||||
|
|
||||||
|
#### Scenario: Duplicate file rejected (in-flight job)
|
||||||
|
- **WHEN** the user runs `kb addfile report.pdf` and the engine returns HTTP 409 with `{"error": "duplicate", "job_id": 7, "title": "report.pdf"}`
|
||||||
|
- **THEN** the client SHALL print "Already queued: report.pdf (job ID: 7)" and exit with code 0
|
||||||
|
|
||||||
|
#### Scenario: Duplicate file in recursive add
|
||||||
|
- **WHEN** the user runs `kb addfile ~/documents/ --recursive` and some files are rejected as duplicates
|
||||||
|
- **THEN** the client SHALL print the duplicate message for each rejected file, continue uploading remaining files, and include a summary (e.g., "Queued: 5 files, 2 duplicates skipped")
|
||||||
|
|
||||||
|
#### Scenario: Duplicate with JSON output
|
||||||
|
- **WHEN** the user runs `kb addfile report.pdf --format json` and the engine returns HTTP 409
|
||||||
|
- **THEN** the client SHALL output the raw JSON response from the engine including the document_id and title
|
||||||
|
|
||||||
|
#### Scenario: Add with JSON output
|
||||||
|
- **WHEN** the user runs `kb addfile report.pdf --format json`
|
||||||
|
- **THEN** the client SHALL output the JSON response from the engine including the job_id
|
||||||
|
|
||||||
|
#### Scenario: File not found
|
||||||
|
- **WHEN** the user runs `kb addfile nonexistent.pdf`
|
||||||
|
- **THEN** the client SHALL print an error and exit with a non-zero code without making any API call
|
||||||
|
|
||||||
|
#### Scenario: Upload failure
|
||||||
|
- **WHEN** the upload fails (network error, engine returns 4xx/5xx other than 409)
|
||||||
|
- **THEN** the client SHALL print the error and exit with a non-zero code
|
||||||
|
|
||||||
|
## REMOVED Requirements
|
||||||
|
|
||||||
|
### Requirement: Implicit note shorthand
|
||||||
|
|
||||||
|
**Reason**: The implicit shorthand caused accidental note creation from mistyped commands. Any unrecognized multi-word input was silently ingested as a note. Replaced by the explicit `addnote` command.
|
||||||
|
|
||||||
|
**Migration**: Replace `kb "note text"` with `kb addnote "note text"`. Replace `kb "note text" --tags foo` with `kb addnote "note text" --tags foo`.
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
## 1. Create addnote command
|
||||||
|
|
||||||
|
- [x] 1.1 Create `client/cmd/addnote.go` with `addnoteCmd` using `ExactArgs(1)`, `--tags` flag, and `RunE` calling `submitNote()`
|
||||||
|
- [x] 1.2 Move `submitNote()` function from `client/cmd/add.go` to `client/cmd/addnote.go`
|
||||||
|
|
||||||
|
## 2. Revert root command to standard Cobra behaviour
|
||||||
|
|
||||||
|
- [x] 2.1 Remove `ArbitraryArgs`, custom `RunE` logic, and `--tags` flag from root command in `client/cmd/root.go`
|
||||||
|
- [x] 2.2 Remove custom usage template and `isRootCmd` template function — let Cobra use its default template
|
||||||
|
- [x] 2.3 Set root command to show help when called with no args (standard Cobra `RunE` returning `cmd.Help()` or nil)
|
||||||
|
|
||||||
|
## 3. Update examples and help text
|
||||||
|
|
||||||
|
- [x] 3.1 Update `client/cmd/examples.go` to show `kb addnote` syntax instead of `kb "text"` shorthand
|
||||||
|
- [x] 3.2 Update root command `Long` description to remove reference to note shorthand
|
||||||
|
|
||||||
|
## 4. Update tests
|
||||||
|
|
||||||
|
- [x] 4.1 Remove implicit note shorthand tests from `client/cmd/root_test.go` (`TestRootCmd_SingleWordRejected`, `TestRootCmd_MultipleWordsNotRejected`)
|
||||||
|
- [x] 4.2 Add test for `addnote` command (verify it wires up correctly, takes exactly one arg)
|
||||||
|
- [x] 4.3 Add test that root command with unknown args returns an error (standard Cobra behaviour)
|
||||||
|
- [x] 4.4 Verify `addfile` tests still pass (no changes expected)
|
||||||
|
|
||||||
|
## 5. Build and verify
|
||||||
|
|
||||||
|
- [x] 5.1 Run `go build` and verify all commands appear in `kb --help`
|
||||||
|
- [x] 5.2 Run `go test ./...` and verify all tests pass
|
||||||
|
- [x] 5.3 Verify `kb addnote --help` shows correct usage line and flags
|
||||||
|
- [x] 5.4 Verify `kb addfile --help` is unchanged
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# Chunk Enrichment
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Chunk enrichment prepends document titles and section headers to chunk text before indexing and embedding, ensuring that document-level context participates in both full-text and semantic search.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
### Requirement: Chunk text enrichment with document title
|
||||||
|
|
||||||
|
The engine SHALL prepend the document title to each chunk's text before FTS indexing and vector embedding. The enriched text SHALL be stored in a dedicated `enriched_text` column on the `chunks` table. The original chunk text SHALL remain in the `text` column for display purposes.
|
||||||
|
|
||||||
|
The enrichment format SHALL be:
|
||||||
|
- Without section header: `"{title}\n\n{chunk_text}"`
|
||||||
|
- With section header: `"{title} > {section_header}\n\n{chunk_text}"`
|
||||||
|
|
||||||
|
Where `section_header` is the value from the chunk's metadata `section_header` field, when present.
|
||||||
|
|
||||||
|
#### Scenario: Note ingestion with title enrichment
|
||||||
|
- **WHEN** a note titled "Suitcase Locks" with content "Steve = 363" is ingested
|
||||||
|
- **THEN** the `chunks.text` column SHALL contain "Steve = 363" and the `chunks.enriched_text` column SHALL contain "Suitcase Locks\n\nSteve = 363"
|
||||||
|
|
||||||
|
#### Scenario: Markdown chunk with section header enrichment
|
||||||
|
- **WHEN** a markdown document titled "DCG Lab Hardware" produces a chunk with section_header "GRIMDAWN > motherboard" and text "MSI X870 Tomahawk"
|
||||||
|
- **THEN** the `chunks.enriched_text` SHALL contain "DCG Lab Hardware > GRIMDAWN > motherboard\n\nMSI X870 Tomahawk"
|
||||||
|
|
||||||
|
#### Scenario: Chunk without section header
|
||||||
|
- **WHEN** a document titled "Docker Tips" produces a chunk with no section_header in metadata and text "dbash() { docker exec -it $1 bash; }"
|
||||||
|
- **THEN** the `chunks.enriched_text` SHALL contain "Docker Tips\n\ndbash() { docker exec -it $1 bash; }"
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Requirement: FTS5 indexes enriched text
|
||||||
|
|
||||||
|
The FTS5 virtual table `chunks_fts` SHALL index the `enriched_text` column instead of the `text` column. All FTS sync triggers (insert, update, delete) SHALL operate on `enriched_text`.
|
||||||
|
|
||||||
|
#### Scenario: FTS search matches document title
|
||||||
|
- **WHEN** a user searches for "suitcase locks" and a document titled "Suitcase Locks" exists with chunk text "Steve = 363"
|
||||||
|
- **THEN** the FTS5 search SHALL return that chunk as a match
|
||||||
|
|
||||||
|
#### Scenario: FTS search still matches chunk content
|
||||||
|
- **WHEN** a user searches for "MSI X870" and a chunk contains that text in its body
|
||||||
|
- **THEN** the FTS5 search SHALL return that chunk as a match (enrichment does not break content matching)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Requirement: Vector embeddings use enriched text
|
||||||
|
|
||||||
|
The embedding model SHALL receive `enriched_text` (not raw `text`) when generating vectors during both initial ingestion and reindex operations.
|
||||||
|
|
||||||
|
#### Scenario: Vector search matches document title
|
||||||
|
- **WHEN** a user searches semantically for "luggage combination codes" and a document titled "Suitcase Locks" exists
|
||||||
|
- **THEN** the vector search SHALL return that chunk with higher similarity than it would without title enrichment
|
||||||
|
|
||||||
|
#### Scenario: Reindex uses enriched text
|
||||||
|
- **WHEN** `POST /api/v1/reindex` is called
|
||||||
|
- **THEN** the engine SHALL read `enriched_text` from the chunks table and embed that (not `text`)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Requirement: Schema migration adds enriched_text column
|
||||||
|
|
||||||
|
On startup, `init_schema` SHALL add the `enriched_text` column to the `chunks` table if it does not exist. It SHALL then backfill `enriched_text` for all existing chunks by joining with `documents.title` and parsing chunk metadata for section headers. It SHALL rebuild the FTS5 table and triggers to index `enriched_text`.
|
||||||
|
|
||||||
|
#### Scenario: First startup after upgrade
|
||||||
|
- **WHEN** the engine starts and `chunks.enriched_text` column does not exist
|
||||||
|
- **THEN** the engine SHALL add the column, backfill all rows, drop and recreate `chunks_fts` to index `enriched_text`, and recreate the FTS sync triggers
|
||||||
|
|
||||||
|
#### Scenario: Subsequent startup
|
||||||
|
- **WHEN** the engine starts and `chunks.enriched_text` column already exists
|
||||||
|
- **THEN** the engine SHALL not perform any migration and start normally
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Requirement: Search results return raw text
|
||||||
|
|
||||||
|
Search results SHALL continue to return the original chunk text (from `chunks.text`) in the `text` field, not the enriched text. The document title is already returned as a separate `title` field.
|
||||||
|
|
||||||
|
#### Scenario: Search result text field
|
||||||
|
- **WHEN** a search returns a chunk from document "Suitcase Locks" with raw text "Steve = 363"
|
||||||
|
- **THEN** the result `text` field SHALL be "Steve = 363" (not "Suitcase Locks\n\nSteve = 363")
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
### Requirement: DEVELOPER.md exists at repo root
|
||||||
|
The repository SHALL have a `DEVELOPER.md` file at the project root containing all developer-facing documentation.
|
||||||
|
|
||||||
|
#### Scenario: File exists
|
||||||
|
- **WHEN** a developer navigates to the repository root
|
||||||
|
- **THEN** a `DEVELOPER.md` file SHALL be present
|
||||||
|
|
||||||
|
### Requirement: DEVELOPER.md contains build-from-source instructions
|
||||||
|
DEVELOPER.md SHALL contain instructions for building both the engine and client from source.
|
||||||
|
|
||||||
|
#### Scenario: Engine build from source
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include instructions for starting the engine from source using compose files (both NVIDIA and ROCm)
|
||||||
|
|
||||||
|
#### Scenario: Client build from source
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include instructions for building the client binary from source using `make build` and `make all`
|
||||||
|
|
||||||
|
### Requirement: DEVELOPER.md contains release process
|
||||||
|
DEVELOPER.md SHALL document the release process for both client and engine, including release scripts, version bumping, and Docker image tagging.
|
||||||
|
|
||||||
|
#### Scenario: Client release documentation
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include `release-client.sh` usage with flag options (--gitea, --github, --minor, --no-increment, --dry-run)
|
||||||
|
|
||||||
|
#### Scenario: Engine release documentation
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include `release-engine.sh` usage with flag options and Docker image tag conventions
|
||||||
|
|
||||||
|
#### Scenario: Version checking
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include how to check client and engine versions
|
||||||
|
|
||||||
|
### Requirement: DEVELOPER.md contains developer notes
|
||||||
|
DEVELOPER.md SHALL include any forward-looking developer notes such as migration plans or technical debt items.
|
||||||
|
|
||||||
|
#### Scenario: ROCm migration note
|
||||||
|
- **WHEN** a developer reads DEVELOPER.md
|
||||||
|
- **THEN** it SHALL include the ROCm runtime migration note about onnxruntime and MIGraphX
|
||||||
|
|
||||||
|
### Requirement: README.md excludes developer-only content
|
||||||
|
README.md SHALL NOT contain build-from-source instructions, release processes, or developer-only notes.
|
||||||
|
|
||||||
|
#### Scenario: No from-source build steps in README
|
||||||
|
- **WHEN** a user reads README.md
|
||||||
|
- **THEN** there SHALL be no "From source" subsections under engine or client installation
|
||||||
|
|
||||||
|
#### Scenario: No release section in README
|
||||||
|
- **WHEN** a user reads README.md
|
||||||
|
- **THEN** there SHALL be no "Building and releasing" section
|
||||||
|
|
||||||
|
#### Scenario: No developer notes in README
|
||||||
|
- **WHEN** a user reads README.md
|
||||||
|
- **THEN** there SHALL be no "Future: ROCm runtime migration" section
|
||||||
|
|
||||||
|
### Requirement: README.md cross-references DEVELOPER.md
|
||||||
|
README.md SHALL include a link to DEVELOPER.md for users who want to build from source or contribute.
|
||||||
|
|
||||||
|
#### Scenario: Developer link in quick start
|
||||||
|
- **WHEN** a user reads the Quick Start section of README.md
|
||||||
|
- **THEN** there SHALL be a note pointing to DEVELOPER.md for building from source
|
||||||
@@ -128,11 +128,11 @@ The engine SHALL maintain job records in SQLite with status tracking. Jobs SHALL
|
|||||||
|
|
||||||
### Requirement: Background ingestion worker
|
### Requirement: Background ingestion worker
|
||||||
|
|
||||||
The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes), generate embeddings using the resident model, insert chunks and vectors into the database, and move the original file to persistent storage.
|
The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.
|
||||||
|
|
||||||
#### Scenario: Successful PDF ingestion
|
#### Scenario: Successful PDF ingestion
|
||||||
- **WHEN** the background worker picks up a queued PDF job
|
- **WHEN** the background worker picks up a queued PDF job
|
||||||
- **THEN** it SHALL update the job status to `processing`, run Docling conversion and chunking, embed all chunks, insert document and chunks into the database, move the staged file to `{data_dir}/documents/{content_hash}.pdf`, update `documents.stored_path` with the permanent path, store the original filename in `documents.original_filename`, update the job status to `done` with the resulting document_id and chunk count, and clean up the staging entry
|
- **THEN** it SHALL update the job status to `processing`, run Docling conversion and chunking, build enriched text for each chunk by prepending the document title, embed all chunks using enriched text, insert document and chunks into the database, move the staged file to `{data_dir}/documents/{content_hash}.pdf`, update `documents.stored_path` with the permanent path, store the original filename in `documents.original_filename`, update the job status to `done` with the resulting document_id and chunk count, and clean up the staging entry
|
||||||
|
|
||||||
#### Scenario: Ingestion failure
|
#### Scenario: Ingestion failure
|
||||||
- **WHEN** the background worker encounters an error during processing (e.g., corrupt PDF)
|
- **WHEN** the background worker encounters an error during processing (e.g., corrupt PDF)
|
||||||
@@ -202,7 +202,7 @@ The engine SHALL provide status information and support re-embedding all chunks.
|
|||||||
|
|
||||||
#### Scenario: Trigger reindex
|
#### Scenario: Trigger reindex
|
||||||
- **WHEN** a client sends `POST /api/v1/reindex`
|
- **WHEN** a client sends `POST /api/v1/reindex`
|
||||||
- **THEN** the engine SHALL re-embed all existing chunks using the currently loaded model and return progress information. This operation SHALL NOT block search queries.
|
- **THEN** the engine SHALL re-embed all existing chunks using the `enriched_text` column and the currently loaded model, and return progress information. This operation SHALL NOT block search queries.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -68,43 +68,39 @@ The client SHALL provide a `kb search <query>` command that sends the query to t
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Requirement: Implicit note shorthand
|
### Requirement: Add note command
|
||||||
|
|
||||||
The client SHALL treat bare string arguments (with no subcommand) as an implicit note. `kb "my note"` SHALL behave identically to submitting a note via `POST /api/v1/jobs`. All persistent flags (`--format`, `--engine`, `--api-key`) and the root `--tags` flag SHALL work with the shorthand form.
|
The client SHALL provide a `kb addnote <text>` command that submits a text note to the engine for ingestion. The command SHALL take exactly one positional argument (the note text) and support a `--tags` flag for comma-separated tags. The note SHALL be submitted via `POST /api/v1/jobs` with the `note` field in a multipart request.
|
||||||
|
|
||||||
#### Scenario: Quick note via bare argument
|
#### Scenario: Add a note
|
||||||
- **WHEN** the user runs `kb "remember to update DNS"`
|
- **WHEN** the user runs `kb addnote "remember to update DNS records"`
|
||||||
- **THEN** the client SHALL submit the text as a note via `POST /api/v1/jobs` and print `Queued: note`
|
- **THEN** the client SHALL submit the text as a note via `POST /api/v1/jobs` and print `Queued: note`
|
||||||
|
|
||||||
#### Scenario: Bare argument with tags
|
#### Scenario: Add a note with tags
|
||||||
- **WHEN** the user runs `kb "server room is building 3" --tags ops`
|
- **WHEN** the user runs `kb addnote "server room is building 3" --tags ops`
|
||||||
- **THEN** the client SHALL submit the note with the specified tags
|
- **THEN** the client SHALL submit the note with the specified tags
|
||||||
|
|
||||||
#### Scenario: Bare argument with JSON output
|
#### Scenario: Add a note with JSON output
|
||||||
- **WHEN** the user runs `kb "my note" --format json`
|
- **WHEN** the user runs `kb addnote "my note" --format json`
|
||||||
- **THEN** the client SHALL output the raw JSON response from the engine
|
- **THEN** the client SHALL output the raw JSON response from the engine
|
||||||
|
|
||||||
#### Scenario: Bare argument duplicate detection
|
#### Scenario: Duplicate note detection
|
||||||
- **WHEN** the user runs `kb "my note"` and the engine returns HTTP 409
|
- **WHEN** the user runs `kb addnote "my note"` and the engine returns HTTP 409
|
||||||
- **THEN** the client SHALL handle the duplicate response identically to the previous `kb add --note` behaviour
|
- **THEN** the client SHALL display the duplicate information (document ID or job ID) and exit with code 0
|
||||||
|
|
||||||
#### Scenario: Multiple unquoted words
|
#### Scenario: Missing argument
|
||||||
- **WHEN** the user runs `kb remember to update dns` (without quotes)
|
- **WHEN** the user runs `kb addnote` with no arguments
|
||||||
- **THEN** the client SHALL join all arguments into a single note string and submit it
|
- **THEN** the client SHALL display an error indicating that the note text argument is required
|
||||||
|
|
||||||
#### Scenario: No interference with subcommands
|
#### Scenario: Too many arguments
|
||||||
- **WHEN** the user runs `kb search "query"` or any other existing subcommand
|
- **WHEN** the user runs `kb addnote remember to update dns` (unquoted, multiple args)
|
||||||
- **THEN** the client SHALL route to the subcommand as before — the implicit note shorthand SHALL NOT interfere
|
- **THEN** the client SHALL display an error indicating that exactly one argument is required, with a hint to quote the text
|
||||||
|
|
||||||
#### Scenario: No arguments
|
|
||||||
- **WHEN** the user runs `kb` with no arguments
|
|
||||||
- **THEN** the client SHALL display the help text
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Requirement: Add command (file and note ingestion)
|
### Requirement: Add command (file and note ingestion)
|
||||||
|
|
||||||
The client SHALL provide a `kb addfile` command that uploads files to the engine for async ingestion. The command SHALL validate file extensions before uploading and reject unsupported types. The client SHALL handle duplicate rejection (HTTP 409) and display the existing document information. The command SHALL NOT handle notes — notes are submitted via the implicit note shorthand (`kb "text"`).
|
The client SHALL provide a `kb addfile` command that uploads files to the engine for async ingestion. The command SHALL validate file extensions before uploading and reject unsupported types. The client SHALL handle duplicate rejection (HTTP 409) and display the existing document information. Notes are handled by the separate `addnote` command — `addfile` is exclusively for file uploads.
|
||||||
|
|
||||||
#### Scenario: Add a single file
|
#### Scenario: Add a single file
|
||||||
- **WHEN** the user runs `kb addfile report.pdf`
|
- **WHEN** the user runs `kb addfile report.pdf`
|
||||||
|
|||||||
Reference in New Issue
Block a user