# clanker-saddle

ClankerSaddle is an LLM task orchestration platform: a Statum + Spry web
application that queues work for humans and dispatches it to
[`saddle`](../saddle) daemons — one-container agents that work on real git
checkouts through bash sessions with full auditability. You design tasks
during the day; the agent works while you sleep.

## Architecture

```
Browser ←→ clanker-saddle (Spry app: SQLite queue, Statum UI, dispatcher)
                          ↕ HTTP (daemons push; asks/wakes only go out)
               saddle daemons (Vala: bash session + LLM loop, port 7100)
                          ↕ curl
               any OpenAI-compatible chat-completions endpoint
```

The web app owns projects, tasks, dependencies and the saddle registry.
Work acquisition is ask-based (plan 10): daemons sleep when idle and ask
`POST /api/saddle/next-task` (per-saddle bearer token); the allocation is
one write transaction that pairs the asking saddle with the best
offerable task — workspace affinity first (a task parked on this saddle
is offered only to it), then capability-tag matching. The dispatcher
wakes sleeping daemons when work becomes ready (`POST /wake`, the affine
saddle first for parked tasks) and declares a saddle offline when no ask
arrives within the lost-wake guard window. Telemetry is push-based
(phase 2): daemons flush raw PTY batches to `POST /api/saddle/chunks`
and round results with state transitions to `POST /api/saddle/results` —
`blocked` raises the question as an answerable block and parks the
saddle, `suspended` parks for the subtask-event resume, `done`/`failed`
complete or fail the task. Nothing polls for output; `GET /status`
against a daemon is exceptional only — the staleness probe (a silent
runner owes one probe after `SADDLE_STALE_S`; a failed probe requeues
its task and marks the saddle offline) and the manual refresh button.
A recovered daemon's ask flips an offline row back to idle.

| Component | Lives | Role |
|---|---|---|
| Web app | `src/` | pages, actions, services, dispatcher, saddle API |
| Saddle daemon | `../saddle` | bash PTY-free session, LLM loop, control server |
| Mock LLM | `../saddle/test/mock_llm.py` | scripted endpoint for testing |

Orchestration behaviour to know:

| Setting | Meaning |
|---|---|
| `SADDLE_LLM_BASE_URL` | LLM endpoint sent with every assignment; unset pauses dispatch |
| `SADDLE_LLM_API_KEY` / `SADDLE_LLM_MODEL` | Optional credentials/model for the above |
| `SADDLE_LOST_WAKE_S` | Lost-wake guard window (default 15s): no ask after a wake → the saddle goes offline and the next capable one is woken |
| `SADDLE_STALE_S` | Running-liveness window (default 90s): no chunks or results for this long → one exceptional `GET /status` probe; failure requeues the task and marks the saddle offline |
| `SADDLE_PROJECTS_ROOT` | Checkouts are addressed as `root/<project-name>` inside the saddle container (default `/workspace`) |

## Quick start (local)

```bash
export WS_PREFIX="$HOME/.local"
export PKG_CONFIG_PATH="$WS_PREFIX/lib64/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
export XDG_DATA_DIRS="$WS_PREFIX/share${XDG_DATA_DIRS:+:$XDG_DATA_DIRS}"
export LD_LIBRARY_PATH="$WS_PREFIX/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

meson setup builddir && ninja -C builddir
```

Run the app, a saddle daemon and a mock LLM in three shells (the mock's
port must be reachable from the daemon; project checkouts live under
`SADDLE_PROJECTS_ROOT`):

```bash
export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=change-me
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
./builddir/clanker-saddle 8080

../saddle/builddir/src/saddle --port 7100

python3 ../saddle/test/mock_llm.py --port 9999
```

Browse to `http://localhost:8080`, log in with the seeded admin account,
register the saddle on `/saddles` with host `localhost:7100`, and create
your first project and task. `spry dev` wraps the first shell (rebuild +
restart on save); `web-config.json` (from `spry keys`) keeps sessions
alive across restarts and is never packaged or baked into images.

The database is `./clanker-saddle.sqlite`, or `SPRY_DB_PATH` when set.

## Docker setup

Two multi-stage Alpine images build everything from source against musl
(see `../Dockerfile.saddle` and `../Dockerfile.clanker-saddle`):

```bash
podman build -f Dockerfile.clanker-saddle -t localhost/clanker-saddle:0.1 ..
podman build -f Dockerfile.saddle -t localhost/saddle:0.1 ..

podman network create clanker-net
podman run -d --name clanker-web --userns=keep-id --network clanker-net \
    -p 8080:8080 -v "$PWD/data:/var/lib/clanker-saddle:Z" \
    -e SPRY_SEED_ADMIN_USERNAME=admin -e SPRY_SEED_ADMIN_PASSWORD=change-me \
    -e SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 \
    -e SADDLE_PROJECTS_ROOT=/workspace \
    localhost/clanker-saddle:0.1
podman run -d --name clanker-saddle --userns=keep-id --network clanker-net \
    -p 7100:7100 -v "$PWD/workspace:/workspace:Z" \
    localhost/saddle:0.1
```

Register the saddle with its **network** address (`clanker-saddle:7100`)
— the web app talks to the daemon container-to-container, while the LLM
base URL is dialled by the daemon itself (a mock can run inside the
saddle container via `podman exec -d clanker-saddle …`, reachable at
`http://127.0.0.1:9999`). Put the SQLite store on a volume
(`SPRY_DB_PATH` defaults to `/var/lib/clanker-saddle/clanker-saddle.sqlite`)
and mount git checkouts under `/workspace`. On SELinux hosts the bind
mounts need `:Z`, and rootless podman needs `--userns=keep-id` for the
mounts to be writable. For durable sessions mount `web-config.json`
read-only and point `ASTRALIS_CONFIG_PATH` at it; otherwise the app runs
with ephemeral keys.

## Creating your first project and task

1. **Project** — `/projects` → name, git URL, default branch. The
   checkout must exist at `$SADDLE_PROJECTS_ROOT/<name>` on the saddle's
   filesystem (clone it into the workspace volume; the agent does not
   clone). Link related projects as *co-developed* (agent may modify) or
   *reference* (read-only) from the project page.
2. **Task** — `/tasks` → project, title, description, priority (1 is
   highest), optionally a design task (explore/plan/question only). Chain
   work with dependencies: a task waits until everything it depends on is
   done.
3. **Saddle** — `/saddles` (admin) → name plus the daemon's host:port.
   Registration mints a per-saddle token and delivers it to the daemon's
   `POST /register` (with this app's base URL); a phase-1 daemon then
   asks for work whenever idle-awake and sleeps on a 204 answer, so a
   ready task reaches it within one dispatcher tick (wake → ask →
   allocate). Pre-flip daemons without `/register` still register — copy
   the token from the saddle card instead.

Tasks can also arrive from the agent itself: `saddle subtask` lines
create child tasks (through the synchronous `POST /api/saddle/subtask`
callback), and push-style daemons report through `POST
/api/saddle/results` (round entries plus state transitions) and `POST
/api/saddle/chunks` (raw PTY batches) — every `/api/saddle/*` call is
authenticated by the per-saddle bearer token (removing the saddle
revokes it).

## Monitoring agent progress

- **`/tasks`** — live queue: status chips (`queued → assigned → working →
  blocked → done/failed`), dependency states and ready flags.
- **`/tasks/{id}`** — the control room; it refreshes on every state
  pulse while a saddle works: comments thread (yours and the agent's),
  blocks, subtasks, session history, and the agent's final summary.
- **`/tasks/{id}/terminal`** — the full bash session log: the raw PTY
  stream the daemon pushes (chunk batches land in `session_raw_log` and
  stream over SSE byte-exact), with the typed entry view as the fallback
  for legacy sessions.
- **`/`** — dashboard: saddle cards, queue summary, recent activity.

## Answering block questions

When an agent needs a human decision it runs `saddle block "<question>"`;
the daemon pushes the `blocked` state with the block payload, the task
flips to **blocked** (the pulse bump lands the question on live pages
without a reload) and the saddle parks. Type the answer, submit — the
task becomes resume-offerable, the dispatcher wakes the affine saddle,
and the daemon's ask returns the assignment with the answer in its
conversation; the task returns to `working`. Answers ride the resume
assignment exactly once per question.

## Rules and project notes

**Rules** (`/rules`) are reusable instructions dispatched with every task
that carries them: create them on the rules page, then attach/detach them
on the project page (they lead every task of the project) and the task
page (they lead the task's own set). A dispatch ships the effective set —
task rules, then the primary project's, then the co-developed projects'
in link order, deduplicated; reference projects contribute nothing.

**Project notes** are written by agents mid-task via `saddle note` and
curated by humans on the project page (delete only — there is no manual
create). Daemons POST notes to this app's
`POST /api/saddle/note` callback, authenticated by the per-saddle bearer
token the daemon persisted at registration (plan 10; the shared
`saddle-callback-…` token is gone). The callback base
URL daemons must dial is **this app's own externally reachable address**
(it differs from the browser's when the daemon runs in a container) and
is configured with the top-level `saddle_callback_base_url` key in
`web-config.json`, falling back to the `SADDLE_CALLBACK_BASE_URL`
environment variable and then `http://localhost:8080`:

```json
{
  "statum": { "…": "…" },
  "saddle_callback_base_url": "http://clanker-web:8080"
}
```

Dispatches also carry the primary and co-developed projects' notes (up to
30 entries / 20 000 characters, oldest first) so agents start with the
accumulated project knowledge.

## Integration testing

`test/integration_test.py` drives the whole loop against a running
app + daemon pair (88 checks, Statum-style PASS/FAIL): explore task,
dependency gating, then the Part 2 feature tour — `saddle plan`,
`saddle query` with piped stdin, progress updates, a multiple-choice
block answered through the web UI, an RFC block with a markdown body,
the `saddle review` banner approved via the UI, a quoted user comment
delivered over /notify, terminal SSE (replay + live), the bearer-token
ingest API and the stm-markdown renderer. Host run:

```bash
mkdir -p /tmp/clanker-e2e/workspace && cd /tmp/clanker-e2e/workspace \
    && git init demo && cd demo && git commit --allow-empty -m one \
    && git commit --allow-empty -m two

WEB=http://localhost:8080 SADDLE=http://localhost:7100 \
WORKSPACE=/tmp/clanker-e2e/workspace \
DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite \
    python3 test/integration_test.py
```

For a container run point `WEB`/`SADDLE` at the published ports, pass
`SADDLE_HOST=clanker-saddle:7100`, the container's SQLite file as
`DB_PATH`, and start the in-container mock via `MOCK_CMD='podman exec -d
clanker-saddle python3 /usr/libexec/saddle/mock_llm.py --port 9999'`
(the test appends the same scripted responses to whatever command
`MOCK_CMD` prefixes). The suite releases a daemon left blocked by an
earlier run via `/stop` before it starts.

`meson test` additionally runs `test/web_stack_test.py` — a
self-contained suite (no daemon needed) that spawns the app in a scratch
directory plus an in-process fake saddle daemon speaking the wire
contract, and asserts the web-stack features: plan-review body hiding,
the effective `rules` array of assignments, project notes with the
`/api/saddle/note` callback and the payload's `project_notes`/`callback`
caps, the ask/allocation flip (concurrent asks race one winner, affinity
pinning, the lost-wake guard), and the phase-2 telemetry flip (pushed
chunks idempotent and ordered into the terminal view, results pushes
with blocked/suspended/done/failed state transitions, entry-redelivery
idempotency, the one-probe staleness policy with its dead-daemon
requeue, and the retired `/output`//`/log`//`/start` daemon routes 404ing
while the web side never calls them).

## Realtime channels and the browser's SSE budget

Browsers cap ~6 concurrent connections per origin on HTTP/1.1, and SSE
streams hold theirs forever: Statum's realtime channel (`/_statum/channel`,
one per browser via its SharedWorker — one per tab where no SharedWorker
is available) plus one `/terminal-stream` per open terminal tab. Deep
tab-stacking on HTTP/1.1 can starve the pulse channel silently (task
pages then miss live block updates). Mitigations built in: hidden
terminal tabs close their stream until focused, and task pages refresh
their state on focus in addition to each pulse. Serving the app over
HTTP/2 removes the cap entirely.

## Deployment (USM)

`MANIFEST.usm` + `usm-scripts/` package the app as `bin:clanker-saddle`
(`usm manifest package`), published to the Web-Stack repository:

```bash
usm manifest package
cd ../web-stack-repo && usm repository add ../clanker-saddle-0.1.usmc
```

`web-config.json` is never packaged (`.usmignore`); keys are mounted at
runtime, never baked into images.