kb/openspec/specs/engine-api/spec.md

# Engine API

## Purpose

The engine API provides an HTTP interface for knowledge base operations including search, document ingestion, document management, tag management, and system status.

## Requirements

### Requirement: Engine startup and model loading

The engine SHALL load the embedding model eagerly at startup before accepting HTTP requests. The engine SHALL expose a health endpoint that returns unhealthy until the model is fully loaded and the database is initialised.

#### Scenario: Cold start with model download
- **WHEN** the engine starts for the first time with no cached model
- **THEN** it SHALL download the configured embedding model, load it into memory (GPU if available, CPU otherwise), enable WAL mode on the SQLite database, and begin accepting requests only after all initialisation completes

#### Scenario: Health check during startup
- **WHEN** a client sends `GET /api/v1/health` before the model is loaded
- **THEN** the engine SHALL respond with HTTP 503 and `{"status": "starting"}`

#### Scenario: Health check after startup
- **WHEN** a client sends `GET /api/v1/health` after initialisation completes
- **THEN** the engine SHALL respond with HTTP 200 and `{"status": "healthy"}`

---

### Requirement: Hybrid search

The engine SHALL provide hybrid search combining BM25 full-text search (via FTS5) and vector similarity search (via sqlite-vec), merged using Reciprocal Rank Fusion with a top-rank bonus (+0.05 for rank 1, +0.02 for ranks 2-3 in either arm) that preserves exact matches. Search SHALL complete in under 100ms when the model is warm and reranking is disabled; reranked searches SHALL complete in under 500ms on GPU. The engine SHALL sanitize user query strings to prevent FTS5 syntax errors for any input.

#### Scenario: Hybrid search with results
- **WHEN** a client sends `POST /api/v1/search` with body `{"query": "how to change oil", "top": 5}`
- **THEN** the engine SHALL embed the query using the resident model, run both FTS5 and vector searches, merge results via RRF with top-rank bonus, and return a JSON response with matched chunks including scores, `document_id`, document metadata, tags, and `tag_contexts`

#### Scenario: Explain traces
- **WHEN** a client sends `POST /api/v1/search` with `"explain": true`
- **THEN** each result SHALL include an `explain` object with per-arm raw scores and ranks (`fts_score`, `fts_rank`, `vec_score`, `vec_rank`), RRF contributions (`rrf_fts`, `rrf_vec`), the top-rank `bonus`, rerank blend fields when reranking ran (`pre_rerank_rank`, `retrieval_norm`, `rerank_score`, `blend_weight`), and the `final_score`; fields for an arm that did not match SHALL be null

#### Scenario: Search with filters
- **WHEN** a client sends `POST /api/v1/search` with body `{"query": "brakes", "tags": ["maintenance"], "doc_type": "pdf", "top": 3}`
- **THEN** the engine SHALL apply tag and document type filters to both FTS5 and vector results before merging

#### Scenario: Search with mode override
- **WHEN** a client sends `POST /api/v1/search` with body `{"query": "error log", "fts_only": true}`
- **THEN** the engine SHALL return only FTS5 results without running vector search

#### Scenario: Empty knowledge base
- **WHEN** a client searches against an empty database
- **THEN** the engine SHALL return HTTP 200 with `{"query": "...", "results": [], "total_matches": 0}`

#### Scenario: Search with special characters
- **WHEN** a client sends `POST /api/v1/search` with body `{"query": "what color is grass?"}`
- **THEN** the engine SHALL sanitize the query for FTS5, execute the search successfully, and return results (not a 500 error)

#### Scenario: Search with FTS5 operators in query
- **WHEN** a client sends `POST /api/v1/search` with body `{"query": "NOT something OR (other)"}`
- **THEN** the engine SHALL treat the input as literal search terms, not FTS5 operators, and return matching results

#### Scenario: Search with only special characters
- **WHEN** a client sends `POST /api/v1/search` with body `{"query": "??!@#"}`
- **THEN** the engine SHALL return HTTP 200 with an empty result set (not a 500 error)

#### Scenario: Search with quotes in query
- **WHEN** a client sends `POST /api/v1/search` with body `{"query": "the \"quick\" fox"}`
- **THEN** the engine SHALL sanitize embedded quotes and return results normally

---

### Requirement: Cross-encoder reranking

The engine SHALL support optional server-side reranking of hybrid search results using a local cross-encoder model, enabled via `KB_RERANK_ENABLED` (default false) with the model set by `KB_RERANKER_MODEL` (default `BAAI/bge-reranker-v2-m3`). When active, the engine SHALL over-fetch candidates (`KB_RERANK_CANDIDATES`, default 40), score each (query, chunk) pair, and blend retrieval and rerank scores position-aware: 75% retrieval weight for pre-rerank ranks 1-3, 60% for 4-10, 40% for 11+, with retrieval scores min-max normalised over the candidate set. Blended scores are on a 0-1 scale distinct from RRF scores; the score threshold SHALL be applied before reranking only. Every search response SHALL include a top-level `"reranked"` boolean.

#### Scenario: Reranked search
- **WHEN** reranking is enabled with a loaded model and a client sends a hybrid search
- **THEN** the engine SHALL rerank the top candidates and return results ordered by blended score with `"reranked": true`

#### Scenario: Per-request opt-out
- **WHEN** a client sends `POST /api/v1/search` with `"rerank": false`
- **THEN** the engine SHALL skip reranking and return plain hybrid results with `"reranked": false`

#### Scenario: Graceful degradation
- **WHEN** reranking is requested but the model is disabled or failed to load
- **THEN** the engine SHALL return plain hybrid results with `"reranked": false` and no error

#### Scenario: Single-arm searches never rerank
- **WHEN** a client sends a search with `fts_only` or `vec_only` set
- **THEN** the engine SHALL NOT rerank, keeping single-arm results pure for benchmarking

---

### Requirement: Async ingestion via job queue

The engine SHALL accept file uploads and text notes for ingestion asynchronously. Uploaded content SHALL be written to a staging area and a job record created in the database. The engine SHALL return HTTP 202 immediately. A background worker SHALL process queued jobs sequentially. Before staging, the engine SHALL compute a SHA256 hash of the uploaded content and reject duplicates immediately.

#### Scenario: Upload a PDF file
- **WHEN** a client sends `POST /api/v1/jobs` with a multipart form containing a PDF file and optional fields (tags, doc_type)
- **THEN** the engine SHALL compute the SHA256 hash of the file bytes, verify no existing document has the same hash, write the file to the staging directory, create a job record with status `queued`, and return HTTP 202 with `{"job_id": "<id>", "status": "queued", "filename": "report.pdf"}`

#### Scenario: Upload a text note
- **WHEN** a client sends `POST /api/v1/jobs` with a multipart form containing a `note` text field and optional `title` field
- **THEN** the engine SHALL compute the SHA256 hash of the note text (UTF-8 encoded), verify no existing document has the same hash, write the note content to a staging file, create a job record with status `queued`, and return HTTP 202 with the job ID

#### Scenario: Upload multiple files in sequence
- **WHEN** a client sends multiple `POST /api/v1/jobs` requests in quick succession
- **THEN** the engine SHALL queue each job independently and the background worker SHALL process them in FIFO order

#### Scenario: Duplicate file detected at upload time (already ingested)
- **WHEN** a client uploads a file whose SHA256 content hash matches an already-ingested document
- **THEN** the engine SHALL NOT stage the file or create a job record, and SHALL return HTTP 409 with `{"error": "duplicate", "document_id": <id>, "title": "<title>"}`

#### Scenario: Duplicate file detected at upload time (in-flight job)
- **WHEN** a client uploads a file whose SHA256 content hash matches a queued or processing job
- **THEN** the engine SHALL NOT stage the file or create a job record, and SHALL return HTTP 409 with `{"error": "duplicate", "job_id": <id>, "title": "<filename>"}`

#### Scenario: Duplicate note detected at upload time (already ingested)
- **WHEN** a client submits a note whose SHA256 content hash matches an already-ingested document
- **THEN** the engine SHALL NOT stage the note or create a job record, and SHALL return HTTP 409 with `{"error": "duplicate", "document_id": <id>, "title": "<title>"}`

#### Scenario: Duplicate note detected at upload time (in-flight job)
- **WHEN** a client submits a note whose SHA256 content hash matches a queued or processing job
- **THEN** the engine SHALL NOT stage the note or create a job record, and SHALL return HTTP 409 with `{"error": "duplicate", "job_id": <id>, "title": "<filename>"}`

#### Scenario: Duplicate uploaded during concurrent request handling
- **WHEN** two identical files are uploaded in the same instant, both passing the API hash check before either job is committed
- **THEN** both jobs SHALL be queued, and the background worker SHALL process the first normally and mark the second as `skipped` (worker-side safety net via `hash_exists()` and UNIQUE constraint)

#### Scenario: Upload failure due to unsupported file type
- **WHEN** a client uploads a file with an unsupported extension
- **THEN** the engine SHALL return HTTP 422 with an error message listing supported types

---

### Requirement: Job status tracking

The engine SHALL maintain job records in SQLite with status tracking. Jobs SHALL transition through states: `queued` → `processing` → `done` | `failed` | `skipped`.

#### Scenario: List all jobs
- **WHEN** a client sends `GET /api/v1/jobs`
- **THEN** the engine SHALL return a JSON array of job records ordered by creation time (newest first), each including job_id, filename, status, created_at, and completed_at

#### Scenario: Filter jobs by status
- **WHEN** a client sends `GET /api/v1/jobs?status=failed`
- **THEN** the engine SHALL return only jobs with the specified status

#### Scenario: Get job details
- **WHEN** a client sends `GET /api/v1/jobs/{id}`
- **THEN** the engine SHALL return the full job record including status, filename, error message (if failed), document_id (if done), chunk count, and timing information

#### Scenario: Job not found
- **WHEN** a client sends `GET /api/v1/jobs/{id}` with a non-existent ID
- **THEN** the engine SHALL return HTTP 404

---

### Requirement: Background ingestion worker

The engine SHALL run a background worker that processes queued jobs. The worker SHALL process one job at a time. For each job, it SHALL: detect document type, run the appropriate chunking pipeline (Docling for PDFs, header-based for Markdown, AST-based for code, whole-text for notes, fixed-size text chunking for data files with minified JSON pretty-printed first), build enriched text by prepending the document title (and section header when present) to each chunk's text, generate embeddings using the enriched text and the resident model, insert chunks (with both raw text and enriched text) and vectors into the database, and move the original file to persistent storage.

#### Scenario: Successful PDF ingestion
- **WHEN** the background worker picks up a queued PDF job
- **THEN** it SHALL update the job status to `processing`, run Docling conversion and chunking, build enriched text for each chunk by prepending the document title, embed all chunks using enriched text, insert document and chunks into the database, move the staged file to `{data_dir}/documents/{content_hash}.pdf`, update `documents.stored_path` with the permanent path, store the original filename in `documents.original_filename`, update the job status to `done` with the resulting document_id and chunk count, and clean up the staging entry

#### Scenario: Ingestion failure
- **WHEN** the background worker encounters an error during processing (e.g., corrupt PDF)
- **THEN** it SHALL update the job status to `failed` with the error message, delete the staged file, and continue processing the next queued job

#### Scenario: Search during active ingestion
- **WHEN** a search request arrives while the background worker is processing a job
- **THEN** the search SHALL execute without blocking (SQLite WAL mode) and return results from already-ingested documents

---

### Requirement: Document management

The engine SHALL provide endpoints to list, inspect, remove, and download original files for ingested documents.

#### Scenario: List documents
- **WHEN** a client sends `GET /api/v1/documents`
- **THEN** the engine SHALL return a JSON array of documents with id, title, doc_type, tags, chunk_count, created_at, and updated_at

#### Scenario: List documents with filters
- **WHEN** a client sends `GET /api/v1/documents?type=pdf&tags=manual`
- **THEN** the engine SHALL return only documents matching all specified filters

#### Scenario: List documents sorted by most recent
- **WHEN** a client requests documents sorted by date
- **THEN** the engine SHALL use `COALESCE(updated_at, created_at)` for ordering, so un-mutated documents sort by creation time and mutated documents sort by their last update

#### Scenario: Get document details
- **WHEN** a client sends `GET /api/v1/documents/{id}`
- **THEN** the engine SHALL return the full document record including all chunks, their text content, `updated_at`, and whether the original file is available (`has_file: true/false`)

#### Scenario: Download original file
- **WHEN** a client sends `GET /api/v1/documents/{id}/file`
- **THEN** the engine SHALL return the original file with appropriate Content-Type and `Content-Disposition: attachment; filename="{original_filename}"` headers, or HTTP 404 if the file is not available

#### Scenario: Remove a document
- **WHEN** a client sends `DELETE /api/v1/documents/{id}`
- **THEN** the engine SHALL delete the document, all its chunks, associated embeddings, tag associations, and the stored original file from disk, and return HTTP 200 with a confirmation

#### Scenario: Remove non-existent document
- **WHEN** a client sends `DELETE /api/v1/documents/{id}` with a non-existent ID
- **THEN** the engine SHALL return HTTP 404

---

### Requirement: Note mutation endpoint

The engine SHALL provide a `PATCH /api/v1/notes/{id}` endpoint for updating existing notes in place. See the `note-mutation` spec for full details.

#### Scenario: Note update endpoint exists
- **WHEN** a client sends `PATCH /api/v1/notes/42` with body `{"text": "new content"}`
- **THEN** the engine SHALL process the update synchronously and return the updated document

---

### Requirement: Document updated_at tracking

The engine SHALL track when documents are modified via an `updated_at` column. This column SHALL be NULL for documents that have never been updated.

#### Scenario: New document has no updated_at
- **WHEN** a document is first ingested
- **THEN** `updated_at` SHALL be NULL and `created_at` SHALL be set to the ingestion timestamp

#### Scenario: Note update sets updated_at
- **WHEN** a note is updated via `PATCH /api/v1/notes/{id}`
- **THEN** `updated_at` SHALL be set to the current timestamp

#### Scenario: Tag change sets updated_at
- **WHEN** tags are modified via `PUT /api/v1/documents/{id}/tags`
- **THEN** `updated_at` SHALL be set to the current timestamp

#### Scenario: Schema migration for updated_at
- **WHEN** the engine starts against a v2 database without an `updated_at` column
- **THEN** the engine SHALL automatically add `ALTER TABLE documents ADD COLUMN updated_at TEXT` and all existing documents SHALL have `updated_at = NULL`

---

### Requirement: Tag management

The engine SHALL provide endpoints to list all tags and manage tags on documents.

#### Scenario: List all tags
- **WHEN** a client sends `GET /api/v1/tags`
- **THEN** the engine SHALL return a JSON array of tags with name, document count, and description (null when unset)

#### Scenario: Add tags to a document
- **WHEN** a client sends `PUT /api/v1/documents/{id}/tags` with body `{"add": ["manual", "v2"]}`
- **THEN** the engine SHALL add the specified tags to the document and return the updated tag list

#### Scenario: Remove tags from a document
- **WHEN** a client sends `PUT /api/v1/documents/{id}/tags` with body `{"remove": ["draft"]}`
- **THEN** the engine SHALL remove the specified tags from the document and return the updated tag list

---

### Requirement: Tag context descriptions

The engine SHALL support a one-line context description per tag, stored in a `description` column on the tags table (added via idempotent migration). Search results SHALL include a `tag_contexts` object mapping each of the document's described tags to its description, so consumers can judge which similar-scoring chunks answer the question.

#### Scenario: Set a tag description
- **WHEN** a client sends `PUT /api/v1/tags/{name}/description` with body `{"description": "Lab operations runbooks"}`
- **THEN** the engine SHALL store the description (matching the tag name case-insensitively) and return `{"name": "<name>", "description": "<description>"}`

#### Scenario: Clear a tag description
- **WHEN** a client sends `PUT /api/v1/tags/{name}/description` with a null or empty description
- **THEN** the engine SHALL clear the stored description

#### Scenario: Unknown tag
- **WHEN** a client sets a description for a tag that does not exist
- **THEN** the engine SHALL return HTTP 404

#### Scenario: Descriptions in search results
- **WHEN** a search result's document carries tags and at least one tag has a description
- **THEN** the result SHALL include `tag_contexts` with only the described tags; results with no described tags SHALL include an empty `tag_contexts` object

---

### Requirement: Engine status and reindex

The engine SHALL provide status information and support re-embedding all chunks. The `version` field in the status response SHALL always be present and SHALL reflect the engine's release version as read from the `VERSION` file. This field is the contract used by clients for compatibility checking.

#### Scenario: Get engine status
- **WHEN** a client sends `GET /api/v1/status`
- **THEN** the engine SHALL return JSON with `version` (string, from VERSION file), model_name, embedding_dim, GPU device info, database stats (document count by type, total chunks, DB size), queue stats (queued/processing job count), and a `rerank` object with `enabled`, `model`, `loaded`, and `candidates`

#### Scenario: Trigger reindex
- **WHEN** a client sends `POST /api/v1/reindex`
- **THEN** the engine SHALL re-embed all existing chunks using the `enriched_text` column and the currently loaded model, and return progress information. This operation SHALL NOT block search queries.

---

### Requirement: API authentication

The engine SHALL support optional API key authentication via Bearer token. When `KB_API_KEY` is set, all requests MUST include a matching `Authorization: Bearer <key>` header. When `KB_API_KEY` is not set, authentication SHALL be disabled.

#### Scenario: Valid API key
- **WHEN** `KB_API_KEY` is set and a request includes a matching Bearer token
- **THEN** the engine SHALL process the request normally

#### Scenario: Missing API key when required
- **WHEN** `KB_API_KEY` is set and a request has no Authorization header
- **THEN** the engine SHALL return HTTP 401 `{"error": "authentication required"}`

#### Scenario: Invalid API key
- **WHEN** `KB_API_KEY` is set and a request includes a non-matching Bearer token
- **THEN** the engine SHALL return HTTP 401 `{"error": "invalid api key"}`

#### Scenario: Auth disabled
- **WHEN** `KB_API_KEY` is not set
- **THEN** the engine SHALL process all requests without requiring authentication

---

### Requirement: Engine configuration via environment variables

The engine SHALL be configured via environment variables. No config file is read by the engine — all configuration comes from the environment (set via compose.yaml or Docker run).

#### Scenario: Default configuration
- **WHEN** the engine starts with no environment variables set
- **THEN** it SHALL use defaults: data directory `/data`, model `all-MiniLM-L6-v2`, device `auto`, no API key required. It SHALL create `staging/` and `documents/` subdirectories under the data directory.

#### Scenario: Custom model
- **WHEN** `KB_MODEL` is set to `BAAI/bge-small-en-v1.5`
- **THEN** the engine SHALL download and load that model instead of the default