mouth/plan.md

# Mouth — Implementation Plan

## Overview

Mouth is a single-binary, offline speech-to-text tool for Windows (with Linux/macOS support where possible). Press a hotkey, speak, and transcribed text is pasted at your cursor. Configured entirely via YAML.

## Architecture

```
┌─────────────┐     ┌───────────┐     ┌─────────────┐     ┌────────────┐
│  Hotkey      │────▶│  Recorder │────▶│ Transcriber  │────▶│   Paste    │
│  Listener    │     │  (cpal)   │     │  (ort/ONNX)  │     │  (enigo)   │
│  (rdev)      │     │           │     │              │     │            │
└─────────────┘     └───────────┘     └─────────────┘     └────────────┘
       │                  │                  │                    │
       │                  ▼                  │                    │
       │            ┌───────────┐            │                    │
       │            │    VAD    │            │                    │
       │            │ (silero)  │            │                    │
       │            └───────────┘            │                    │
       │                                     │                    │
       ▼                                     ▼                    ▼
┌──────────────────────────────────────────────────────────────────────┐
│                        Overlay (winit)                               │
│                  State: idle → recording → transcribing → done       │
└──────────────────────────────────────────────────────────────────────┘
```

### Component Communication

All components communicate via channels (`std::sync::mpsc` or `tokio::sync`). The main thread owns the overlay window (required by most windowing systems). A coordinator task receives events from hotkey/recorder/transcriber and drives state transitions.

```
HotkeyEvent(Pressed/Released) ──┐
AudioReady(Vec<f32>) ───────────┼──▶ Coordinator ──▶ OverlayState
TranscriptionDone(String) ──────┘                 ──▶ PasteAction
CancelRequested ────────────────┘
```

## Crate Dependencies

| Crate | Purpose | Notes |
|-------|---------|-------|
| `rdev` | Global hotkey capture | Cross-platform key events, no focus required |
| `cpal` | Audio capture | Cross-platform mic input |
| `rubato` | Audio resampling | Resample to 16kHz for Parakeet |
| `ort` | ONNX Runtime | Run Parakeet v3 + Silero VAD |
| `hf-hub` | Model download | Download from HuggingFace, standard cache dir |
| `enigo` | Keyboard simulation | Simulate Ctrl+V, Shift+Insert, etc. |
| `arboard` | Clipboard access | Read/write clipboard, save/restore |
| `winit` | Windowing | Minimal overlay window |
| `softbuffer` | Pixel rendering | Draw coloured overlay (no GPU needed for overlay) |
| `serde` + `serde_yaml` | Config | Deserialize YAML config |
| `clap` | CLI | Subcommands: `run`, `config`, `models` |
| `dialoguer` | Interactive TUI | `mouth config` interactive setup |
| `rodio` | Audio playback | Blip up/down sounds |
| `indicatif` | Progress bars | Model download progress |
| `dirs` | Platform dirs | Config/cache paths |
| `tracing` | Logging | Structured logging |

## Config File

Location: `~/.config/mouth/config.yaml` (Linux/macOS), `%APPDATA%\mouth\config.yaml` (Windows)

```yaml
# Hotkey to activate recording
hotkey: "ctrl+space"

# Recording mode: push_to_talk or toggle
mode: push_to_talk

# Cancel hotkey (only active while recording)
cancel_key: "escape"

# Speech-to-text model
model: "parakeet-tdt-0.6b-v3"

# Inference accelerator: auto, cpu, cuda, directml
accelerator: auto

# GPU device index (only used when accelerator is cuda/directml)
gpu_device: 0

# How to paste text
paste_method: ctrl_v  # ctrl_v | shift_insert | ctrl_shift_v | clipboard_only

# Also keep transcribed text on clipboard after pasting
copy_to_clipboard: true

# Overlay position on screen
overlay_position: top  # top | bottom | none

# Audio feedback
audio_feedback: true

# Audio input device (null = system default)
input_device: null

# VAD: trim silence from audio before transcription
vad_enabled: true

# Language (for model hint, if supported)
language: en
```

## CLI Interface

```
mouth run              # Start the daemon (default if no subcommand)
mouth config           # Interactive TUI to edit config
mouth config --show    # Print current config to stdout
mouth config --reset   # Reset config to defaults
mouth models           # List available/downloaded models
mouth models download  # Download configured model (if not cached)
mouth status           # Show daemon status, loaded model, app version
```

## Implementation Phases

### Phase 1: Project Skeleton + Config

- Cargo.toml with all dependencies
- Config struct with serde, defaults, load/save
- CLI with clap (run, config, models subcommands)
- `mouth config` interactive TUI with dialoguer
- Platform-aware config/cache directory resolution

### Phase 2: Hotkey Listener

- Global hotkey capture using rdev
- Support configurable key combinations (parse from string like "ctrl+space")
- Push-to-talk mode: record on press, stop on release
- Toggle mode: start on first press, stop on second press
- Cancel on Escape while recording
- Debounce rapid key events (~30ms)

### Phase 3: Audio Capture + VAD

- Open mic input via cpal (default device or configured)
- Convert to f32 mono
- Resample to 16kHz via rubato
- Buffer audio chunks during recording
- Run Silero VAD to trim leading/trailing silence
- Produce final `Vec<f32>` of clean speech at 16kHz

### Phase 4: Model Management

- Use hf-hub to download Parakeet v3 ONNX model from HuggingFace
- Store in standard HF cache (`~/.cache/huggingface/hub/`)
- Show download progress with indicatif
- `mouth models` command to list/download models
- Auto-download on first run if model not cached

### Phase 5: Transcription

- Load Parakeet v3 ONNX model via ort
- Auto-detect GPU (DirectML on Windows, CUDA if available, CPU fallback)
- Respect accelerator override from config
- Run inference on captured audio
- Return transcribed text string

### Phase 6: Overlay

- Create a small always-on-top window using winit
- Render with softbuffer (simple coloured rectangle + text)
- States and colours:
  - Recording: red pulsing indicator
  - Transcribing: amber/yellow
  - Done: brief green flash, then hide
  - Error: brief red flash with error hint
- Window flags (Windows): `WS_EX_TOPMOST | WS_EX_TOOLWINDOW | WS_EX_NOACTIVATE`
- Position: centered horizontally at top or bottom of current monitor
- No focus steal, no taskbar entry

### Phase 7: Paste System

- Save current clipboard content (if preserving)
- Set transcribed text to clipboard via arboard
- Simulate keypress via enigo based on paste_method:
  - `ctrl_v`: Ctrl+V (Cmd+V on macOS)
  - `shift_insert`: Shift+Insert
  - `ctrl_shift_v`: Ctrl+Shift+V
  - `clipboard_only`: no keypress, just clipboard
- Restore previous clipboard content (unless copy_to_clipboard is true)
- Small delay between clipboard set and paste simulation (~50ms)

### Phase 8: Audio Feedback

- Bundle two short PCM blip sounds in the binary (via `include_bytes!`)
- "Blip up" on recording start
- "Blip down" on recording stop / transcription complete
- Play via rodio on a separate thread (non-blocking)
- Respect audio_feedback config flag

### Phase 9: Coordinator + Integration

- Wire all components together with channel-based message passing
- Main thread: overlay window event loop (winit requires this)
- Spawned threads/tasks: hotkey listener, audio recorder, transcriber
- Coordinator receives events, drives state machine:
  ```
  Idle ──[hotkey press]──▶ Recording
  Recording ──[hotkey release/press]──▶ Transcribing
  Recording ──[cancel]──▶ Idle
  Transcribing ──[result]──▶ Pasting ──▶ Idle
  Transcribing ──[error]──▶ Error ──▶ Idle
  ```
- Graceful shutdown on SIGINT / tray quit

### Phase 10: Daemon IPC + Status

- The running daemon listens on a local Unix domain socket (Linux/macOS) or named pipe (Windows) for status queries
- Socket/pipe path: `/tmp/mouth.sock` (Linux/macOS), `\\.\pipe\mouth` (Windows)
- `mouth status` connects and requests current state; daemon responds with JSON:
  ```json
  {
    "version": "0.1.0",
    "state": "idle",
    "model": "parakeet-tdt-0.6b-v3",
    "accelerator": "directml",
    "uptime_secs": 3420
  }
  ```
- If the daemon is not running, `mouth status` reports "Mouth is not running" and exits with code 1
- Also used internally to prevent launching a second daemon instance (lock check)

### Phase 11: Polish + Distribution

- Error handling: user-friendly messages for common failures (no mic, model not found, etc.)
- Windows installer via `cargo-wix` or distribute as standalone .exe
- Test on Windows 10/11 primarily
- Test on Linux (X11 + Wayland) and macOS as secondary
- Update CLAUDE.md with build/run/test instructions
- Write user-facing README with setup instructions

## Risks & Mitigations

| Risk | Impact | Mitigation |
|------|--------|------------|
| Parakeet v3 ONNX model compatibility with `ort` | Blocks core functionality | Test early in Phase 5; Parakeet v2 as fallback |
| `rdev` hotkey reliability on Windows | Broken UX | Test early in Phase 2; fallback to Win32 `RegisterHotKey` |
| Overlay focus stealing | Annoying | Use proper window flags; test with various foreground apps |
| Audio resampling quality | Poor transcription | Use rubato SincInterpolation (high quality) |
| Binary size with bundled ONNX Runtime | Large download | ONNX Runtime is ~20-40MB; acceptable for a single-binary tool |
| winit event loop blocking | Unresponsive | All heavy work on background threads; overlay is lightweight |

## File Structure

```
mouth/
├── Cargo.toml
├── CLAUDE.md
├── README.md
├── plan.md
├── config.yaml.example
├── resources/
│   ├── blip_up.pcm          # bundled audio feedback
│   └── blip_down.pcm
└── src/
    ├── main.rs               # CLI entry, clap setup
    ├── config.rs             # Config struct, YAML load/save, defaults
    ├── hotkey.rs             # Global hotkey listener (rdev)
    ├── recorder.rs           # Audio capture (cpal + rubato + VAD)
    ├── vad.rs                # Silero VAD wrapper
    ├── transcriber.rs        # ONNX inference, model loading, GPU detection
    ├── model_cache.rs        # HuggingFace download, cache management
    ├── overlay.rs            # Minimal overlay window (winit + softbuffer)
    ├── paste.rs              # Clipboard + paste simulation
    ├── audio_feedback.rs     # Blip sounds via rodio
    ├── coordinator.rs        # State machine, channel hub
    └── cli/
        ├── mod.rs
        ├── run.rs            # `mouth run` handler
        ├── config_cmd.rs     # `mouth config` TUI
        ├── models_cmd.rs     # `mouth models` handler
        └── status_cmd.rs     # `mouth status` handler
```

## Not In Scope (v1)

- LLM post-processing of transcriptions
- Transcription history / database
- Multiple model support (v1 is Parakeet v3 only, architecture supports adding more later)
- Auto-submit (Enter after paste)
- Multi-language UI
- Tray icon / system tray integration
- Translate-to-English mode