Add manual Docker image rebuild workflow

The CPU image was 11.2GB — larger than the CUDA one — and its single 6.72GB
torch layer could not be pushed at all: the registry drops any blob upload
taking longer than 60s, and that layer needed ~61s.

Both problems came from install ordering. Dockerfile.cpu installed the project
and sentence-transformers first, which resolved the default CUDA torch and
pulled ~2.7GB of nvidia-* wheels, then reinstalled torch from the CPU index.
Reinstalling replaces torch but leaves its transitive CUDA dependencies behind,
orphaned and unused (torch reports 2.13.0+cpu, cuda.is_available() False).

Installing CPU torch first, from the CPU index, means nothing ever requests a
CUDA build. Everything else then resolves against the torch already present.

  CPU image     11.2GB -> 3.43GB
  largest layer  6.72GB -> 896MB
  push           failed 5/5 at 60s -> succeeds in 8.6s

Dockerfile.nvidia gets the same split. Its torch layer is unavoidably large
(~2.8GiB compressed, ~52s at current throughput) so it stays the tightest
thing in the stack, but splitting the application install off keeps that layer
cached and the app layer small.

Both now install torch before the source COPYs, so editing application code no
longer invalidates the multi-GB layer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.gitea/workflows/rebuild-docker.yml

@@ -0,0 +1,81 @@
+name: Rebuild Docker images
+
+on:
+  workflow_dispatch:
+
+concurrency:
+  group: rebuild-docker-images
+  cancel-in-progress: false
+
+jobs:
+  rebuild-and-push:
+    runs-on: ubuntu-latest
+    env:
+      REGISTRY: docker.dcglab.co.uk
+      IMAGE_BASE: docker.dcglab.co.uk/public/kb
+      REGISTRY_USERNAME: ${{ secrets.DOCKER_DCGLAB_CI_USERNAME }}
+      REGISTRY_PASSWORD: ${{ secrets.DOCKER_DCGLAB_CI_PASSWORD }}
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Log in to registry
+        shell: bash
+        run: |
+          set -euo pipefail
+          test -n "$REGISTRY_USERNAME" || { echo "DOCKER_DCGLAB_CI_USERNAME is not available" >&2; exit 1; }
+          test -n "$REGISTRY_PASSWORD" || { echo "DOCKER_DCGLAB_CI_PASSWORD is not available" >&2; exit 1; }
+          printf '%s' "$REGISTRY_PASSWORD" | docker login "$REGISTRY" --username "$REGISTRY_USERNAME" --password-stdin
+
+      - name: Build all images from scratch
+        shell: bash
+        run: |
+          set -euo pipefail
+          version="$(tr -d '[:space:]' < engine/VERSION)"
+
+          docker build --pull --no-cache --provenance=false --sbom=false \
+            -t "$IMAGE_BASE/engine:v${version}-nvidia" \
+            -t "$IMAGE_BASE/engine:latest-nvidia" \
+            -f engine/Dockerfile.nvidia engine
+
+          docker build --pull --no-cache --provenance=false --sbom=false \
+            -t "$IMAGE_BASE/engine:v${version}-cpu" \
+            -t "$IMAGE_BASE/engine:latest-cpu" \
+            -f engine/Dockerfile.cpu engine
+
+          docker build --pull --no-cache --provenance=false --sbom=false \
+            -t "$IMAGE_BASE/mcp:v${version}" \
+            -t "$IMAGE_BASE/mcp:latest" \
+            -f mcp/Dockerfile mcp
+
+      - name: Push and verify all tags
+        shell: bash
+        run: |
+          set -euo pipefail
+          version="$(tr -d '[:space:]' < engine/VERSION)"
+          images=(
+            "$IMAGE_BASE/engine:v${version}-nvidia"
+            "$IMAGE_BASE/engine:latest-nvidia"
+            "$IMAGE_BASE/engine:v${version}-cpu"
+            "$IMAGE_BASE/engine:latest-cpu"
+            "$IMAGE_BASE/mcp:v${version}"
+            "$IMAGE_BASE/mcp:latest"
+          )
+
+          push_image() {
+            local image="$1"
+            local attempt
+            for attempt in 1 2 3 4 5; do
+              docker push "$image" && return 0
+              if [[ "$attempt" -eq 5 ]]; then
+                echo "Failed to push $image after $attempt attempts" >&2
+                return 1
+              fi
+              sleep 10
+            done
+          }
+
+          for image in "${images[@]}"; do
+            push_image "$image"
+            docker manifest inspect "$image" >/dev/null
+          done

DEVELOPER.md

@@ -90,17 +90,30 @@ curl http://localhost:8000/api/v1/status | jq .version
 
 ### Docker images
 
-Images are pushed to `gitea.dcglab.co.uk/steve/kb/engine` with tags:
+Images are pushed to `docker.dcglab.co.uk/public/kb/engine` with tags:
 
 - `engine-v2.0.6-nvidia` / `engine-v2.0.6-cpu` — versioned
 - `latest-nvidia` / `latest-cpu` — latest release
 
+The release script authenticates to the registry using the
+`DOCKER_DCGLAB_CI_USERNAME` and `DOCKER_DCGLAB_CI_PASSWORD` environment
+variables.
+
 Override the registry and org via environment variables:
 
 ```bash
 REGISTRY=ghcr.io IMAGE_ORG=myorg ./release-engine.sh --github
 ```
 
+Pushes are retried on transient registry failures. The engine images carry a
+~5.6GB torch layer, and uploading it can fail with a 502 from the proxy in
+front of the registry (or a 500 on the manifest PUT that follows), which
+clears on a retry. Tune with:
+
+```bash
+PUSH_RETRIES=8 PUSH_RETRY_DELAY=20 ./release-engine.sh --gitea
+```
+
 ## API reference
 
 All endpoints are under `/api/v1/`. Requires `Authorization: Bearer <key>` header when `KB_API_KEY` is set.

MCP.md

@@ -20,7 +20,7 @@ docker run -d --name kb-mcp \
   -e KB_API_KEY=your-engine-key \
   -e KB_MCP_API_KEY=your-agent-key \
   --restart unless-stopped \
-  gitea.dcglab.co.uk/steve/kb/mcp:latest
+  docker.dcglab.co.uk/public/kb/mcp:latest
 ```
 
 ## MCP tools

README.md

@@ -33,7 +33,7 @@ docker run -d --name kb-engine \
   -e KB_DEVICE=auto \
   -e KB_API_KEY=your-secret-key \
   --restart unless-stopped \
-  gitea.dcglab.co.uk/steve/kb/engine:latest-nvidia
+  docker.dcglab.co.uk/public/kb/engine:latest-nvidia
 
 # CPU only (no GPU required — smaller image)
 docker run -d --name kb-engine \
@@ -42,7 +42,7 @@ docker run -d --name kb-engine \
   -e KB_MODEL=all-MiniLM-L6-v2 \
   -e KB_API_KEY=your-secret-key \
   --restart unless-stopped \
-  gitea.dcglab.co.uk/steve/kb/engine:latest-cpu
+  docker.dcglab.co.uk/public/kb/engine:latest-cpu
 ```
 
 Or use a compose file from the repo:

client/VERSION

@@ -1 +1 @@
-3.2.0
+3.3.0

engine/Dockerfile.cpu

@@ -13,16 +13,32 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
 
 WORKDIR /app
 
+# Install CPU torch first, on its own, from the CPU index.
+#
+# Order matters: anything that depends on torch (sentence-transformers) will
+# otherwise resolve the default CUDA build and pull ~2.7GB of nvidia-* wheels.
+# Reinstalling torch afterwards replaces torch but leaves those wheels behind,
+# orphaned and unused — which is how the CPU image ended up larger than the
+# CUDA one. Installing CPU torch up front means nothing ever requests CUDA.
+#
+# Keeping it in its own layer also bounds the blob size: the registry drops
+# uploads that take longer than 60s, so no single layer should approach ~3GB.
+# Placing it before the source COPYs keeps this expensive layer cached when
+# only application code changes.
+RUN uv venv .venv && \
+    . .venv/bin/activate && \
+    UV_HTTP_TIMEOUT=600 uv pip install torch torchvision \
+        --index-url https://download.pytorch.org/whl/cpu
+
 COPY pyproject.toml ./
 COPY kb/ kb/
 COPY main.py ./
 COPY VERSION ./
 
-RUN uv venv .venv && \
+# Remaining dependencies resolve against the CPU torch already present.
-    . .venv/bin/activate && \
+RUN . .venv/bin/activate && \
-    uv pip install -e . && \
     uv pip install "sentence-transformers[onnx]" && \
-    uv pip install --reinstall torch torchvision --index-url https://download.pytorch.org/whl/cpu
+    uv pip install -e .
 
 ENV PATH="/app/.venv/bin:$PATH"
 ENV VIRTUAL_ENV="/app/.venv"

engine/Dockerfile.nvidia

@@ -13,14 +13,23 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
 
 WORKDIR /app
 
+# Install CUDA torch on its own, before the source COPYs.
+#
+# This is the bulk of the image (~2.8GiB compressed). Splitting it from the
+# application install keeps it cached when only code changes, and keeps the
+# app layer small. The registry drops any blob upload that takes longer than
+# 60s, so this layer is deliberately the only large one.
+RUN uv venv .venv && \
+    . .venv/bin/activate && \
+    UV_HTTP_TIMEOUT=600 uv pip install torch torchvision \
+        --index-url https://download.pytorch.org/whl/cu130
+
 COPY pyproject.toml ./
 COPY kb/ kb/
 COPY main.py ./
 COPY VERSION ./
 
-RUN uv venv .venv && \
+RUN . .venv/bin/activate && \
-    . .venv/bin/activate && \
-    UV_HTTP_TIMEOUT=600 uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130 && \
     uv pip install -e .
 
 ENV PATH="/app/.venv/bin:$PATH"

openspec/specs/docker-deployment/spec.md

@@ -65,7 +65,7 @@ The project SHALL provide Docker Compose files for single-command deployment. Co
 
 #### Scenario: Pre-built image deployment
 - **WHEN** an admin wants to use a pre-built engine image without building from source
-- **THEN** the engine release notes SHALL include the exact `docker pull` command with the versioned tag (e.g. `gitea.dcglab.co.uk/steve/kb/engine:engine-v2.1.0-nvidia`)
+- **THEN** the engine release notes SHALL include the exact `docker pull` command with the versioned tag (e.g. `docker.dcglab.co.uk/public/kb/engine:engine-v2.1.0-nvidia`)
 
 #### Scenario: MCP allowed hosts in Compose
 - **WHEN** the kb-mcp service is defined in a Compose file

release-engine.sh

@@ -15,9 +15,17 @@ ENGINE_DIR="$SCRIPT_DIR/engine"
 VERSION_FILE="$ENGINE_DIR/VERSION"
 
 # Container registry
-REGISTRY="${REGISTRY:-gitea.dcglab.co.uk}"
+#
-IMAGE_ORG="${IMAGE_ORG:-steve}"
+# --provenance=false --sbom=false on every build: buildx would otherwise attach
-IMAGE_BASE="${REGISTRY}/${IMAGE_ORG}/kb"
+# attestation manifests, making the image an OCI image index. The Registry v2
+# host at docker.dcglab.co.uk rejects those with a 500 on manifest PUT.
+REGISTRY="${REGISTRY:-docker.dcglab.co.uk}"
+IMAGE_ORG="${IMAGE_ORG:-public}"
+IMAGE_BASE="${REGISTRY}${IMAGE_ORG:+/${IMAGE_ORG}}/kb"
+
+# Push retries — see push_image() below
+PUSH_RETRIES="${PUSH_RETRIES:-5}"
+PUSH_RETRY_DELAY="${PUSH_RETRY_DELAY:-10}"
 
 #──────────────────────────────────────────────────────────────────────
 # Parse args
@@ -98,6 +106,46 @@ run() {
     fi
 }
 
+registry_login() {
+    echo "  $ docker login $REGISTRY --username \$DOCKER_DCGLAB_CI_USERNAME --password-stdin"
+    [[ "$DRY_RUN" == true ]] && return 0
+
+    printf '%s' "$DOCKER_DCGLAB_CI_PASSWORD" |
+        docker login "$REGISTRY" \
+            --username "$DOCKER_DCGLAB_CI_USERNAME" \
+            --password-stdin
+}
+
+# Push one image tag, retrying on transient registry failures.
+#
+# The engine images carry a ~5.6GB torch layer. Uploading it intermittently
+# fails with a 502 from the reverse proxy in front of the registry, and a
+# manifest PUT can then fail with a 500 because the blob commit has not yet
+# registered. Both clear on a retry, so a whole release should not be lost to
+# one hiccup. Tune with PUSH_RETRIES / PUSH_RETRY_DELAY.
+push_image() {
+    local image="$1"
+    local attempt=1
+
+    echo "  $ docker push $image"
+    [[ "$DRY_RUN" == true ]] && return 0
+
+    while true; do
+        if docker push "$image"; then
+            return 0
+        fi
+
+        if (( attempt >= PUSH_RETRIES )); then
+            echo "Error: failed to push $image after $PUSH_RETRIES attempts" >&2
+            return 1
+        fi
+
+        echo "    push failed (attempt $attempt/$PUSH_RETRIES) — retrying in ${PUSH_RETRY_DELAY}s"
+        sleep "$PUSH_RETRY_DELAY"
+        attempt=$(( attempt + 1 ))
+    done
+}
+
 #──────────────────────────────────────────────────────────────────────
 # Determine release version
 #──────────────────────────────────────────────────────────────────────
@@ -127,6 +175,17 @@ echo ""
 echo "==> Pre-flight checks"
 
 if [[ "$DRY_RUN" == false ]]; then
+    if [[ -z "${DOCKER_DCGLAB_CI_USERNAME:-}" ]]; then
+        echo "Error: DOCKER_DCGLAB_CI_USERNAME is required" >&2
+        exit 1
+    fi
+    if [[ -z "${DOCKER_DCGLAB_CI_PASSWORD:-}" ]]; then
+        echo "Error: DOCKER_DCGLAB_CI_PASSWORD is required" >&2
+        exit 1
+    fi
+
+    registry_login
+
     if git -C "$SCRIPT_DIR" rev-parse "$GIT_TAG" &>/dev/null; then
         echo "Error: tag $GIT_TAG already exists"
         exit 1
@@ -155,8 +214,8 @@ CPU_IMAGE="${IMAGE_BASE}/engine:${DOCKER_TAG}-cpu"
 NVIDIA_LATEST="${IMAGE_BASE}/engine:latest-nvidia"
 CPU_LATEST="${IMAGE_BASE}/engine:latest-cpu"
 
-run docker build -t "$NVIDIA_IMAGE" -t "$NVIDIA_LATEST" -f "$ENGINE_DIR/Dockerfile.nvidia" "$ENGINE_DIR"
+run docker build --provenance=false --sbom=false -t "$NVIDIA_IMAGE" -t "$NVIDIA_LATEST" -f "$ENGINE_DIR/Dockerfile.nvidia" "$ENGINE_DIR"
-run docker build -t "$CPU_IMAGE" -t "$CPU_LATEST" -f "$ENGINE_DIR/Dockerfile.cpu" "$ENGINE_DIR"
+run docker build --provenance=false --sbom=false -t "$CPU_IMAGE" -t "$CPU_LATEST" -f "$ENGINE_DIR/Dockerfile.cpu" "$ENGINE_DIR"
 
 echo ""
 
@@ -171,7 +230,7 @@ if [[ -f "$MCP_DIR/Dockerfile" ]]; then
     MCP_IMAGE="${IMAGE_BASE}/mcp:${DOCKER_TAG}"
     MCP_LATEST="${IMAGE_BASE}/mcp:latest"
 
-    run docker build -t "$MCP_IMAGE" -t "$MCP_LATEST" -f "$MCP_DIR/Dockerfile" "$MCP_DIR"
+    run docker build --provenance=false --sbom=false -t "$MCP_IMAGE" -t "$MCP_LATEST" -f "$MCP_DIR/Dockerfile" "$MCP_DIR"
 
     echo ""
 fi
@@ -233,14 +292,14 @@ echo ""
 #──────────────────────────────────────────────────────────────────────
 echo "==> Pushing Docker images to $REGISTRY"
 
-run docker push "$NVIDIA_IMAGE"
+push_image "$NVIDIA_IMAGE"
-run docker push "$NVIDIA_LATEST"
+push_image "$NVIDIA_LATEST"
-run docker push "$CPU_IMAGE"
+push_image "$CPU_IMAGE"
-run docker push "$CPU_LATEST"
+push_image "$CPU_LATEST"
 
 if [[ -n "${MCP_IMAGE:-}" ]]; then
-    run docker push "$MCP_IMAGE"
+    push_image "$MCP_IMAGE"
-    run docker push "$MCP_LATEST"
+    push_image "$MCP_LATEST"
 fi
 
 echo ""