# clanker-saddle

ClankerSaddle is an LLM task orchestration platform: a Statum + Spry web
application that queues work for humans and dispatches it to
[`saddle`](../saddle) daemons — one-container agents that work on real git
checkouts through bash sessions with full auditability. You design tasks
during the day; the agent works while you sleep.

## Architecture

```
Browser ←→ clanker-saddle (Spry app: SQLite queue, Statum UI, dispatcher)
                          ↕ HTTP (daemons push; asks/wakes only go out)
               saddle daemons (Vala: bash session + LLM loop, port 7100)
                          ↕ curl
               any OpenAI-compatible chat-completions endpoint
```

The web app owns projects, goals, flat tasks and the saddle registry.
Work acquisition is ask-based (plan 10): daemons sleep when idle and ask
`POST /api/saddle/next-task` (per-saddle bearer token); the allocation is
one write transaction that pairs the asking saddle with the best
offerable task — workspace affinity first (a task parked on this saddle
is offered only to it), then capability-tag matching. The dispatcher
wakes sleeping daemons when work becomes ready (`POST /wake`, the affine
saddle first for parked tasks) and declares a saddle offline when no ask
arrives within the lost-wake guard window. Telemetry is push-based
(phase 2): daemons flush raw PTY batches to `POST /api/saddle/chunks`
and round results with state transitions to `POST /api/saddle/results` —
`blocked` raises the question as an answerable block and parks the
saddle, `suspended` parks for the subtask-event resume, `done`/`failed`
complete or fail the task. Nothing polls for output; `GET /status`
against a daemon is exceptional only — the staleness probe (a silent
runner owes one probe after `SADDLE_STALE_S`; a failed probe requeues
its task and marks the saddle offline) and the manual refresh button.
A recovered daemon's ask flips an offline row back to idle.
Capability awareness (phase 3): every assignment carries a
`capabilities` context — `saddle_tags` (the allocated saddle's own
tags) and `fleet_tags` (every tag name known across the fleet) — which
the daemon renders into its system prompt; `GET
/api/saddle/capabilities` (per-saddle token) feeds its `saddle
capabilities` CLI, and the subtask callback's `capabilities: [tags]`
attach explicit, validated requirements (unknown tags reject with the
valid list). The home dashboard pins OPEN blocks (questions, RFCs,
reviews) as newest-first link-through cards above a cross-task activity
timeline (last ~50 events, newest first); the task page leads with its
status block and orders blocking sections newest-first.

| Component | Lives | Role |
|---|---|---|
| Web app | `src/` | pages, actions, services, dispatcher, saddle API |
| Saddle daemon | `../saddle` | bash PTY-free session, LLM loop, control server |
| Mock LLM | `../saddle/test/mock_llm.py` | scripted endpoint for testing |

Orchestration behaviour to know:

| Setting | Meaning |
|---|---|
| `SADDLE_LLM_BASE_URL` | LLM endpoint sent with every assignment; unset pauses dispatch |
| `SADDLE_LLM_API_KEY` / `SADDLE_LLM_MODEL` | Optional credentials/model for the above |
| `SADDLE_LOST_WAKE_S` | Lost-wake guard window (default 15s): no ask after a wake → the saddle goes offline and the next capable one is woken |
| `SADDLE_STALE_S` | Running-liveness window (default 90s): no chunks or results for this long → one exceptional `GET /status` probe; failure requeues the task and marks the saddle offline |
| `SADDLE_PROJECTS_ROOT` | Checkouts are addressed as `root/<project-name>` inside the saddle container (default `/workspace`) |

## Quick start (local)

```bash
export WS_PREFIX="$HOME/.local"
export PKG_CONFIG_PATH="$WS_PREFIX/lib64/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
export XDG_DATA_DIRS="$WS_PREFIX/share${XDG_DATA_DIRS:+:$XDG_DATA_DIRS}"
export LD_LIBRARY_PATH="$WS_PREFIX/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

meson setup builddir && ninja -C builddir
```

Run the app, a saddle daemon and a mock LLM in three shells (the mock's
port must be reachable from the daemon; project checkouts live under
`SADDLE_PROJECTS_ROOT`):

```bash
export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=change-me
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
./builddir/clanker-saddle 8080

../saddle/builddir/src/saddle --port 7100

python3 ../saddle/test/mock_llm.py --port 9999
```

Browse to `http://localhost:8080`, log in with the seeded admin account,
register the saddle on `/saddles` with host `localhost:7100`, and create
your first project and task. `spry dev` wraps the first shell (rebuild +
restart on save); `web-config.json` (from `spry keys`) keeps sessions
alive across restarts and is never packaged or baked into images.

The database is `./clanker-saddle.sqlite`, or `SPRY_DB_PATH` when set.

## Docker setup

Two multi-stage Alpine images build everything from source against musl
(see `../Dockerfile.saddle` and `../Dockerfile.clanker-saddle`):

```bash
podman build -f Dockerfile.clanker-saddle -t localhost/clanker-saddle:0.1 ..
podman build -f Dockerfile.saddle -t localhost/saddle:0.1 ..

podman network create clanker-net
podman run -d --name clanker-web --userns=keep-id --network clanker-net \
    -p 8080:8080 -v "$PWD/data:/var/lib/clanker-saddle:Z" \
    -e SPRY_SEED_ADMIN_USERNAME=admin -e SPRY_SEED_ADMIN_PASSWORD=change-me \
    -e SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 \
    -e SADDLE_PROJECTS_ROOT=/workspace \
    -e SADDLE_CALLBACK_BASE_URL=http://clanker-web:8080 \
    localhost/clanker-saddle:0.1
podman run -d --name clanker-saddle --userns=keep-id --network clanker-net \
    -p 7100:7100 -v "$PWD/workspace:/workspace:Z" \
    localhost/saddle:0.1
```

Register the saddle with its **network** address (`clanker-saddle:7100`)
— the web app talks to the daemon container-to-container, while the LLM
base URL is dialled by the daemon itself (a mock can run inside the
saddle container via `podman exec -d clanker-saddle …`, reachable at
`http://127.0.0.1:9999`). `SADDLE_CALLBACK_BASE_URL` (or the
`web-config.json` `saddle_callback_base_url` key) must be the app's
address **as the daemon dials it** — every registered daemon pushes
telemetry, task callbacks and notes there. Put the SQLite store on a volume
(`SPRY_DB_PATH` defaults to `/var/lib/clanker-saddle/clanker-saddle.sqlite`)
and mount git checkouts under `/workspace`. On SELinux hosts the bind
mounts need `:Z`, and rootless podman needs `--userns=keep-id` for the
mounts to be writable. For durable sessions mount `web-config.json`
read-only and point `ASTRALIS_CONFIG_PATH` at it; otherwise the app runs
with ephemeral keys.

## Creating your first project and task

1. **Project** — `/projects` → name, git URL, default branch. The
   checkout must exist at `$SADDLE_PROJECTS_ROOT/<name>` on the saddle's
   filesystem (clone it into the workspace volume; the agent does not
   clone). Link related projects as *co-developed* (agent may modify) or
   *reference* (read-only) from the project page.
2. **Task** — `/tasks` → project, title, description, priority (1 is
   highest) and the plan-11 options ("auto-approve plan",
   "auto-enqueue agent-created subtasks", "auto-approve subtasks"). The
   task is created in **draft**: edit every field, attach rules and
   capabilities, then press **Enqueue** (dependencies still in draft
   block the enqueue with an error naming them; nothing depends on a
   draft before you may delete it). Chain work with dependencies: an
   enqueued task waits until everything it depends on is done.
3. **Saddle** — `/saddles` (admin) → name plus the daemon's host:port.
   Registration mints a per-saddle token and delivers it to the daemon's
   `POST /register` (with this app's base URL); a phase-1 daemon then
   asks for work whenever idle-awake and sleeps on a 204 answer, so a
   ready task reaches it within one dispatcher tick (wake → ask →
   allocate). Pre-flip daemons without `/register` still register — copy
   the token from the saddle card instead.

Tasks can also arrive from the agent itself: `saddle subtask` lines
create child tasks (through the synchronous `POST /api/saddle/subtask`
callback), and push-style daemons report through `POST
/api/saddle/results` (round entries plus state transitions) and `POST
/api/saddle/chunks` (raw PTY batches) — every `/api/saddle/*` call is
authenticated by the per-saddle bearer token (removing the saddle
revokes it).

## Monitoring agent progress

- **`/tasks`** — live queue: status chips (`draft → queued → assigned →
  working → blocked → done/failed`), mode badges, goal links and the
  standalone/by-goal filter.
- **`/tasks/{id}`** — the control room; it refreshes on every state
  pulse while a saddle works: comments thread (yours and the agent's),
  blocks, goal context, session history, and the agent's final summary.
- **`/tasks/{id}/terminal`** — the full bash session log: the raw PTY
  stream the daemon pushes (chunk batches land in `session_raw_log` and
  stream over SSE byte-exact), with the typed entry view as the fallback
  for legacy sessions. The session picker honours `?session_id=` across
  every run a task ever had.
- **`/goals/{id}`** — the goal control room: lifecycle head, locked
  brief, plan expander with revision history, task strip, the
  supervision narrative and the completion review.
- **`/`** — dashboard: the "needs your attention" pile (goal AND task
  blocks interleaved, newest-first) above the activity timeline, saddle
  cards and queue summary.

## Answering block questions

When an agent needs a human decision it runs `saddle block "<question>"`;
the daemon pushes the `blocked` state with the block payload, the task
flips to **blocked** (the pulse bump lands the question on live pages
without a reload) and the saddle parks. Type the answer, submit —
answering IS allocating: the answered task becomes resume-offerable,
the dispatcher wakes the affine saddle, and the daemon's ask returns
the assignment carrying the verdict as its `resume_notice` (the stable
wordings — "Human selected: …", "Plan approved. Proceed with
implementation.", "Review approved. Continue with the task." &c.) with
a **fresh session id**; the parked session closes as `resumed`. A goal
agent's blocks attach to the GOAL (dual-rowed: the goal block plus a
quiet mirror on the run task) — one answer resolves both and resumes
the run. `saddle plan` parks with the goal's plan-review block:
Approve answers "Plan approved. Proceed with implementation." and
applies the markdown as the goal's plan; the feedback textarea returns
the agent to planning to revise. The completion review answers "Review
approved. Continue with the task." to complete the goal.

Resumed assignments also carry an optional `context_update` (markdown
string): the delta of the instruction context that changed since the
run's last round — the goal's plan approved in the meantime (the
approved markdown rides along), rules created, edited or attached to
the task/its projects, or new project notes. The daemon appends it as
a TURN on resume (prompt-cache friendly) instead of refreshing the
system prompt; the member is absent when nothing changed.

## Browser notifications (Web Push)

The app speaks the real Web Push protocol (VAPID + RFC 8291 payload
encryption, no third-party service). The VAPID keypair is generated at
startup on first run and persisted in **`web-config.json`** under the
`vapid` section (`public_key` / `private_key`, base64url P-256) — that
file is the single home for the keys; delete the section to rotate
(regenerate = restart). `SADDLE_VAPID_SUBJECT` overrides the JWT's
`mailto:` contact (default `mailto:clanker-saddle@localhost`).

To enable notifications in a browser: open the dashboard and press
**Enable notifications** — the page requests permission, registers the
service worker (`GET /sw.js`, scope `/`) and subscribes with the app's
VAPID key. "Turn off notifications" reverses it. Per task, **Notify
me** / **Don't notify me** (on the task page's status card) opts you
into that task's first start, failure and completion; task questions,
RFCs and review requests notify every subscribed user, and goal blocks
plus goal success/failure ALWAYS notify every subscribed user. Clicking
a notification opens the task's or goal's page.
Delivery is best-effort: failed sends retry on the dispatcher tick and
endpoints answering 410/404 are pruned automatically.

## Rules and project notes

**Rules** (`/rules`) are reusable instructions dispatched with every task
that carries them: create them on the rules page, then attach/detach them
on the project page (they lead every task of the project) and the task
page (they lead the task's own set). A dispatch ships the effective set —
task rules, then the primary project's, then the co-developed projects'
in link order, deduplicated; reference projects contribute nothing.

**Project notes** are written by agents mid-task via `saddle note` and
curated by humans on the project page (delete only — there is no manual
create). Daemons POST notes to this app's
`POST /api/saddle/note` callback, authenticated by the per-saddle bearer
token the daemon persisted at registration (plan 10; the shared
`saddle-callback-…` token is gone). The callback base
URL daemons must dial is **this app's own externally reachable address**
(it differs from the browser's when the daemon runs in a container) and
is configured with the top-level `saddle_callback_base_url` key in
`web-config.json`, falling back to the `SADDLE_CALLBACK_BASE_URL`
environment variable and then `http://localhost:8080`:

```json
{
  "statum": { "…": "…" },
  "saddle_callback_base_url": "http://clanker-web:8080"
}
```

Dispatches also carry the primary and co-developed projects' notes (up to
30 entries / 20 000 characters, oldest first) so agents start with the
accumulated project knowledge.

## Integration testing

`test/integration_test.py` drives BOTH real binaries — this app and a
real saddle daemon over a real git checkout — through the full goal-era
arc (83 checks, Statum-style PASS/FAIL): goal create → start → pre-plan
question (one answer resolving the goal row AND the mirrored run-task
row) → plan reject-with-feedback → revision → approval on the agreed
verdict words → task fan-out through the real `/api/saddle/task/*`
callbacks → the worker's ephemeral worktree (auto-pruned, branch kept)
→ the coalesced terminal-event resume → merge + branch delete +
push-before-review → completion review accept → goal complete, plus the
`/status` `goal_branches` cache, SSE over the agent's raw terminal, the
`goal_events` narrative, the legacy ingest API surface and the
stm-markdown renderer. Host run (the harness prepares a `demo` repo
with a pushable bare origin when the workspace is empty; the database
must be FRESH — the harness refuses a used one):

```bash
export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=adminpass123
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
export SADDLE_CALLBACK_BASE_URL=http://localhost:8080
export SPRY_DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite
export SADDLE_GOAL_RESUME_S=2   # shrink the coalescing window for tests
./builddir/clanker-saddle 8080 &

SADDLE_WORKSPACE_ROOT=/tmp/clanker-e2e/workspace \
    ../saddle/builddir/src/saddle --port 7100 &

WEB=http://localhost:8080 SADDLE=http://localhost:7100 \
WORKSPACE=/tmp/clanker-e2e/workspace \
DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite \
    python3 test/integration_test.py
```

The daemon's `SADDLE_WORKSPACE_ROOT` and the app's
`SADDLE_PROJECTS_ROOT` point at the SAME directory (the goal clone the
daemon cuts must anchor the worker worktrees — the shared-volume
deployment assumption). For a container run point `WEB`/`SADDLE` at the
published ports, pass `SADDLE_HOST=clanker-saddle:7100`, the container's
SQLite file as `DB_PATH`, and start the in-container mock via
`MOCK_CMD='podman exec -d clanker-saddle python3
/usr/libexec/saddle/mock_llm.py --port 9999'` (the test appends the
same scripted responses to whatever command `MOCK_CMD` prefixes). The
suite releases a daemon left blocked by an earlier run via `/stop`
before it starts.

`meson test` additionally runs `test/web_stack_test.py` — a
self-contained suite (no daemon needed) that spawns the app in a scratch
directory plus an in-process fake saddle daemon speaking the wire
contract, and asserts the web-stack features: plan-review body hiding,
the effective `rules` array of assignments, project notes with the
`/api/saddle/note` callback and the payload's `project_notes`/`callback`
caps, the ask/allocation flip (concurrent asks race one winner, affinity
pinning, the lost-wake guard), the phase-2 telemetry flip (pushed
chunks idempotent and ordered into the terminal view, results pushes
with blocked/suspended/done/failed state transitions, entry-redelivery
idempotency, the one-probe staleness policy with its dead-daemon
requeue, and the retired `/output`//`/log`//`/start` daemon routes 404ing
while the web side never calls them), and the phase-3 capability
awareness + UI (the assignment `capabilities` context,
`GET /api/saddle/capabilities`, task-callback `--capability` validation
with the valid-tag list in the error, the task page's top status block
and newest-first blocking sections, the dashboard's pile and timeline,
and the slow-ask lost-wake regression), and the plan-12 goal era
(lifecycle, plan/completion reviews with dual-rowed goal blocks and the
agreed verdict wordings, task callbacks, coalesced terminal-event
resumes, pause queueing, cancel cascade, auto-enqueue-after, goal
notifications and the goal context of worker assignments).

## Realtime channels and the browser's SSE budget

Browsers cap ~6 concurrent connections per origin on HTTP/1.1, and SSE
streams hold theirs forever: Statum's realtime channel (`/_statum/channel`,
one per browser via its SharedWorker — one per tab where no SharedWorker
is available) plus one `/terminal-stream` per open terminal tab. Deep
tab-stacking on HTTP/1.1 can starve the pulse channel silently (task
pages then miss live block updates). Mitigations built in: hidden
terminal tabs close their stream until focused, and task pages refresh
their state on focus in addition to each pulse. Serving the app over
HTTP/2 removes the cap entirely.

## Deployment assumptions (goal era)

- **Shared workspace volume across saddles.** A goal's clone workspace
  (`<root>/goal-<id>`, one clone per involved repo on the goal branch)
  anchors every task worktree (`<root>/task-<task-id>`, ephemeral
  branch `task/<goal-id>-<task-id>`). Whichever saddle runs a goal task
  must see the same `SADDLE_WORKSPACE_ROOT` volume as the saddle that
  runs the goal agent — today the fleet is single-saddle per goal by
  construction (one goal agent, one worker at a time on the one
  registered saddle); multi-saddle fleets must mount the same volume
  everywhere before fanning goal work out.
- **Database tear-down-and-recreate.** The plan-12 migrations flatten
  the schema again (a fresh `M0001_Initial`: goals, goal links/comments/
  blocks/events, the flattened tasks). There is NO upgrade path from a
  plan-11 database — delete the SQLite file (or point `SPRY_DB_PATH` at
  a fresh path) and let the app recreate it.
- **Restart both sides.** The daemon (`saddle`) and this app must both
  restart onto the goal-era builds together — the wire contract
  (assignment `kind`/`mode`/`creator`/`goal`/`ephemeral_branch`, the
  task callbacks, goal-keyed blocks, `/status` `goal_branches`) is only
  honoured by matching pairs, and the daemon's registration/token
  persists across its restarts while the app re-seeds on a fresh DB.

## Deployment (USM)

`MANIFEST.usm` + `usm-scripts/` package the app as `bin:clanker-saddle`
(`usm manifest package`), published to the Web-Stack repository:

```bash
usm manifest package
cd ../web-stack-repo && usm repository add ../clanker-saddle-0.1.usmc
```

`web-config.json` is never packaged (`.usmignore`); keys are mounted at
runtime, never baked into images.