clanker-saddle

clanker-saddle

ClankerSaddle is an LLM task orchestration platform: a Statum + Spry web application that queues work for humans and dispatches it to saddle daemons — one-container agents that work on real git checkouts through bash sessions with full auditability. You design tasks during the day; the agent works while you sleep.

Architecture

Browser ←→ clanker-saddle (Spry app: SQLite queue, Statum UI, dispatcher)
                          ↕ HTTP (daemons push; asks/wakes only go out)
               saddle daemons (Vala: bash session + LLM loop, port 7100)
                          ↕ curl
               any OpenAI-compatible chat-completions endpoint

The web app owns projects, goals, flat tasks and the saddle registry. Work acquisition is ask-based (plan 10): daemons sleep when idle and ask POST /api/saddle/next-task (per-saddle bearer token); the allocation is one write transaction that pairs the asking saddle with the best offerable task — workspace affinity first (a task parked on this saddle is offered only to it), then capability-tag matching. The dispatcher wakes sleeping daemons when work becomes ready (POST /wake, the affine saddle first for parked tasks) and declares a saddle offline when no ask arrives within the lost-wake guard window. Telemetry is push-based (phase 2): daemons flush raw PTY batches to POST /api/saddle/chunks and round results with state transitions to POST /api/saddle/results — blocked raises the question as an answerable block and parks the saddle, suspended parks until the goal agent's next terminal-event resume, done/failed complete or fail the task. Nothing polls for output; GET /status against a daemon is exceptional only — the staleness probe (a silent runner owes one probe after SADDLE_STALE_S; a failed probe requeues its task and marks the saddle offline) and the manual refresh button. A recovered daemon's ask flips an offline row back to idle. Which saddle runs what derives from open sessions — busy iff an open session exists, idle otherwise; only the offline probe state is stored. Capability awareness: every assignment carries a capabilities context — saddle_tags (the allocated saddle's own tags) and fleet_tags (every tag name known across the fleet) — which the daemon renders into its system prompt; GET /api/saddle/capabilities (per-saddle token) feeds its saddle capabilities CLI, and the goal agent's task-create callback attaches explicit, validated capability requirements (unknown tags reject with the valid list). The home dashboard pins OPEN blocks (questions, RFCs, reviews) as newest-first link-through cards above a cross-task activity timeline (last ~50 events, newest first); the task page leads with its status block and orders blocking sections newest-first.

Component	Lives	Role
Web app	src/	pages, actions, services, dispatcher, saddle API
Saddle daemon	../saddle	bash PTY session, LLM loop, control server
Mock LLM	../saddle/test/mock_llm.py	scripted endpoint for testing

Orchestration behaviour to know:

Setting	Meaning
SADDLE_LOST_WAKE_S	Lost-wake guard window (default 15s): no ask after a wake → the saddle goes offline and the next capable one is woken
SADDLE_STALE_S	Running-liveness window (default 90s): no chunks or results for this long → one exceptional GET /status probe; failure requeues the task and marks the saddle offline
SADDLE_PROJECTS_ROOT	Fallback for advisory checkout paths when a saddle's daemon did not report a workspace_root at registration (itself falling back to /workspace); see Deployment assumptions

The model configuration (the LLM endpoint every assignment is sent to, its API key and model, plus the advisor's optional overrides) is NOT environment-driven: it lives in the database and is edited on the Admin → Model page (/models, admin-only). A pre-page deployment exports SADDLE_LLM_BASE_URL / SADDLE_LLM_API_KEY / SADDLE_LLM_MODEL (and the ADVISOR_LLM_* equivalents) exactly once — the first configuration read seeds the model_config row from them and the environment is ignored from then on. An empty base URL pauses task assignment (the dispatcher logs it; the page shows a paused badge).

Quick start (local)

export WS_PREFIX="$HOME/.local"
export PKG_CONFIG_PATH="$WS_PREFIX/lib64/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
export XDG_DATA_DIRS="$WS_PREFIX/share${XDG_DATA_DIRS:+:$XDG_DATA_DIRS}"
export LD_LIBRARY_PATH="$WS_PREFIX/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

meson setup builddir && ninja -C builddir

Run the app, a saddle daemon and a mock LLM in three shells (the mock's port must be reachable from the daemon; project checkouts live under SADDLE_PROJECTS_ROOT):

export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
./builddir/clanker-saddle 8080

Then log in as the admin and set the model endpoint on the Admin → Model page to http://127.0.0.1:9999 (mock, model mock) — or export SADDLE_LLM_BASE_URL before the very first boot and let the one-time seed import it.

SPRY_SEED_ADMIN_USERNAME/SPRY_SEED_ADMIN_PASSWORD default to admin/change-me when unset; export them to seed different credentials.

../saddle/builddir/src/saddle --port 7100

python3 ../saddle/test/mock_llm.py --port 9999

Browse to `http://localhost:8080`, log in with the seeded admin account,
register the saddle on `/saddles` with host `localhost:7100`, and create
your first project and task. `spry dev` wraps the first shell (rebuild +
restart on save); `web-config.json` (from `spry keys`) keeps sessions
alive across restarts and is never packaged or baked into images.

The database is `./clanker-saddle.sqlite`, or `SPRY_DB_PATH` when set.
It is UPGRADEABLE: migrations are incremental from `M0002` on and apply
automatically at startup, so a deployed database keeps its data across
upgrades. An old single-migration database (serial 1 only) upgrades in
place on first boot of a newer build.

## Container deployment (spry deploy / usm)

Both halves package as USM packages and build container images from
their own checkouts — no Dockerfiles outside the repos:

bash

this app (a Spry application — spry deploy wraps usm manifest deploy):

spry deploy --verbose

--repository https://packages.astrologue.nz/repo/web-stack/repo.usmr

the daemon (from ../saddle — usm manifest deploy directly; --spm dnf

is required for the in-container manage dependencies):

cd ../saddle && usm manifest deploy --spm dnf

--repository https://packages.astrologue.nz/repo/web-stack/repo.usmr

Each writes `<name>-<version>.image.tar.xz` (single-stage, built in
container against the canonical Web-Stack repository — allow time for
the toolchain build on first run). Load and run:

bash podman load -i clanker-saddle-0.1.image.tar.xz podman load -i saddle-0.1.image.tar.xz

podman network create clanker-net podman run -d --name clanker-web --userns=keep-id --network clanker-net

-p 8080:8080 -v "$PWD/data:/var/lib/clanker-saddle:Z" \
-e SPRY_SEED_ADMIN_USERNAME=admin -e SPRY_SEED_ADMIN_PASSWORD=change-me \
-e SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 \
-e SADDLE_CALLBACK_BASE_URL=http://clanker-web:8080 \
localhost/clanker-saddle:0.1

podman run -d --name clanker-saddle --userns=keep-id --network clanker-net

-p 7100:7100 -v "$PWD/workspace:/workspace:Z" \
localhost/saddle:0.1

(The `SADDLE_LLM_BASE_URL` export above is the ONE-TIME seed: the first
boot imports it into the database's model configuration, which the
**Admin → Model** page owns from then on.)

Register the saddle with its **network** address (`clanker-saddle:7100`)
— the web app talks to the daemon container-to-container, while the LLM
base URL is dialled by the daemon itself (a mock can run inside the
saddle container via `podman exec -d clanker-saddle …`, reachable at
`http://127.0.0.1:9999`). `SADDLE_CALLBACK_BASE_URL` (or the
`web-config.json` `saddle_callback_base_url` key) must be the app's
address **as the daemon dials it** — every registered daemon pushes
telemetry, task callbacks and notes there. Put the SQLite store on a volume
(`SPRY_DB_PATH` defaults to `/var/lib/clanker-saddle/clanker-saddle.sqlite`).
The saddle daemon clones into its own `/workspace` volume and reports
that root at registration, so the web container needs NO workspace mount;
`SADDLE_PROJECTS_ROOT` on the web side is only the fallback for daemons
that do not report one. On SELinux hosts the bind
mounts need `:Z`, and rootless podman needs `--userns=keep-id` for the
mounts to be writable. For durable sessions mount `web-config.json`
read-only and point `ASTRALIS_CONFIG_PATH` at it; otherwise the app runs
with ephemeral keys.

## Saddles with dev tools (the extendable agent container)

The stock `saddle` image carries the daemon, its CLI and git — nothing
else. Build specialised agents by extending it with the toolchains of
your choosing (see `../saddle`'s README, *Extending the saddle image*,
for the full guide): a two-line Containerfile `FROM` the loaded base
plus `dnf install` of your tools (C, Vala, GTK, Node, .NET, Android,
QEMU — anything), then tag the capability names you gave those tools in
the web UI's saddle page so only tasks requiring them route there.
Each saddle mounts its OWN workspace volume: goal task worktrees are cut
from `origin/<goal branch>` and merges travel through origin, so no two
saddles ever need to share a filesystem (see *Deployment assumptions*).

## Creating your first project and task

1. **Project** — `/projects` → name, git URL, default branch. The
   saddle daemon clones the repo itself into its own workspace; the
   paths the web app puts in assignments are ADVISORY and are rewritten
   onto the asking saddle's filesystem (the daemon owns its volumes —
   the web needs no access to them). Link related projects as *co-developed* (agent may modify) or
   *reference* (read-only) from the project page.
2. **Task** — `/tasks` → project, title, **mandate** (the description),
   mode (implement / investigate / review / test) and priority (1 is
   the highest). The task is created in **draft**: edit every field,
   attach rules and capabilities, then press **Enqueue** (only the
   creator enqueues — humans their own, the goal agent its own).
   Long-form fields edit in place: an **Edit** button beside the task
   page's Mandate card (drafts only) swaps the rendered markdown for a
   prefilled editor; the same pattern on a draft goal's Initial Brief
   card edits title, brief and priority before you start it.
   Chain human work with **auto-enqueue after**: pick a predecessor
   from the global task list; the dependent enqueues itself when the
   predecessor completes (a failed or cancelled predecessor leaves it
   in draft).
3. **Saddle** — `/saddles` (admin) → name plus the daemon's host:port.
   Registration mints a per-saddle token and delivers it to the daemon's
   `POST /register` (with this app's base URL); the daemon then asks
   for work whenever idle-awake and sleeps on a 204 answer, so a ready
   task reaches it within one dispatcher tick (wake → ask → allocate). The saddles page
   renders the daemon's `ssh_public_key` in a dark panel with the text
   colour lifted for readable contrast.

**Goals** (`/goals`) are the orchestrated path: write the Initial Brief
and a branch name, start it, and a goal agent investigates (asking
pre-plan questions liberally), submits its plan for review, fans flat
tasks out through the `POST /api/saddle/task/{create,delete,enqueue}`
callbacks (a per-task worktree per involved repo on an ephemeral
branch), merges finished branches into the goal branch, pushes, and
submits a completion report for your review. Push-style daemons report
through `POST /api/saddle/results` (round entries plus state
transitions) and `POST /api/saddle/chunks` (raw PTY batches) — every
`/api/saddle/*` call is authenticated by the per-saddle bearer token
(removing the saddle revokes it).

A goal's **involved projects** start as a one-time snapshot of the
primary project's linked projects: creating a goal copies the project's
co-developed/reference links into the goal's own link set, where you
prune or extend them before starting. Links added to the project later
never propagate to existing goals — the goal's link set is deliberately
frozen at creation (the project page remains the long-term graph).

## Users, roles and your profile

Two roles decide what a logged-in user can do (the account model is
Spry's; this app surfaces it):

- **user** — manages projects, tasks, goals and rules.
- **admin** — additionally manages **users** (`/users`), **saddles**
  (register/remove, git identity, capability tags) and the
  **capability catalogue**. Viewing `/saddles` stays open to every
  logged-in user; the mutation controls are hidden and server-refused
  for non-admins.

`/users` (admin-only) lists accounts newest-first with a role badge
and enabled state: create users, edit profile fields, reset passwords,
enable/disable, grant/revoke admin, delete. Guards keep you from
locking yourself out: you cannot delete or disable your own account,
cannot revoke your own admin permission, and the last enabled admin
cannot be removed.

Every logged-in user has **`/profile`** (linked beside the logout
button in the nav corner): shows your account details and changes your
own password — current, new, confirm. Change the seeded admin's
`change-me` password there before exposing the app.

## Monitoring agent progress

- **`/tasks`** — live queue: status chips (`draft → queued → assigned →
  working → blocked → done/failed`), mode badges, goal links and the
  standalone/by-goal filter.
- **`/tasks/{id}`** — the control room; it refreshes on every state
  pulse while a saddle works: comments thread (yours and the agent's),
  blocks, goal context, session history, and the agent's final summary.
  Your mutations on this page (and the goal page's) answer with the
  refreshed state slots themselves rather than a navigate or reload —
  the page lands in its post-mutation shape in one round trip.
- **`/tasks/{id}/terminal`** — the full bash session log: the raw PTY
  stream the daemon pushes (chunk batches land in `session_raw_log` and
  stream over SSE byte-exact), with the typed entry view as the fallback
  for legacy sessions. The session picker honours `?session_id=` across
  every run a task ever had.
- **`/goals/{id}`** — the goal control room: lifecycle head, locked
  brief, plan expander with revision history, task strip, the
  supervision narrative and the completion review.
- **`/`** — dashboard: the "needs your attention" pile (goal AND task
  blocks interleaved, newest-first) above the activity timeline, saddle
  cards and queue summary. The Queue card's **queued** figure counts
  waiting-for-attention work: ready-to-dispatch queued tasks, parked
  (suspended) tasks — suspended goal-agent runs included — and tasks
  whose raised blocks are all answered and now wait for a saddle to
  resume them; queued rows gated on an unfinished auto-enqueue
  predecessor stay out (the predecessor itself is already counted).
  **blocked** stays human-answerable rows only.

## Answering block questions

When an agent needs a human decision it runs `saddle block "<question>"`;
the daemon pushes the `blocked` state with the block payload, the task
flips to **blocked** (the pulse bump lands the question on live pages
without a reload) and the saddle parks. Type the answer, submit —
answering IS allocating: the answered task becomes resume-offerable,
the dispatcher wakes the affine saddle, and the daemon's ask returns
the assignment carrying the verdict as its `resume_notice` (the stable
wordings — "Human selected: …", "Plan approved. Proceed with
implementation.", "Review approved. Continue with the task." &c.) with
a **fresh session id**; the parked session closes as `parked`. A goal
agent's blocks ride its RUN TASK's single block row — one answer
resolves it and resumes the run. `saddle plan` parks with the goal's
plan-review block: Approve answers "Plan approved. Proceed with
implementation." and applies the markdown as the goal's plan (documents
land on the owning entity only at acceptance — the pending report lives
in its block alone); the feedback textarea returns the agent to
planning to revise. The completion review renders as the rich attention
block (highlight-to-remark annotations queue onto the rejection);
answering "Review approved. Continue with the task." completes the goal
and stores the report.

Resumed assignments also carry an optional `context_update` (markdown
string): the delta of the instruction context that changed since the
run's last round — the goal's plan approved in the meantime (the
approved markdown rides along), rules created, edited or attached to
the task/its projects, or new project notes. The daemon appends it as
a TURN on resume (prompt-cache friendly) instead of refreshing the
system prompt; the member is absent when nothing changed.

## Force Wake (human-triggered resume)

A parked agent normally resumes only when the machinery owes it a wake
(a block answered, an owned task's terminal transition). When a run
parks with nothing pending — the goal agent suspends itself right
after a typo'd enqueue, say — no wake ever fires and it would sit
suspended forever. **Force wake** is the human lever. On an active
goal whose run task is suspended, the goal page's lifecycle head
offers a **Force wake** button (beside Pause/Resume); on a suspended
plain task the task page offers the same, and on a suspended goal-run
task row it routes to the same goal force-wake path — one lever, no
duplicates. One press sets the flag; the next dispatcher pass makes
the run offerable immediately (no coalesce wait), and the resumed
assignment's `resume_notice` leads with an explicit line — "The user
explicitly force-woke this goal's supervision — you were parked with
no pending wake." — followed by the standard batched report, which
names draft-status tasks as "draft — NOT enqueued" so a parked agent
that typo'd an enqueue sees it immediately. The flag clears on the
wake (one wake per press). Paused goals and non-suspended tasks hide
the button and the API refuses them (400).

## Parking guard (the refused suspend)

The web refuses a `saddle suspend` it knows would sleep forever: a
goal agent whose goal owns **no task that could ever fire a terminal
transition** (every owned task is done, failed or a draft nobody
enqueued) has no wake source, so the park is rejected at the results
push. The park mechanics still run, but the run immediately carries a
one-shot refusal instruction — its next assignment's `resume_notice`
reads "Parking refused: the goal owns no live tasks to wait on —
finish the run (submit your report) or cancel." — and the daemon is
reclaimable the moment its round ends. The instruction is consumed by
the resume that delivers it. A goal with any live worker (queued,
assigned, working, blocked, review-requested or suspended) parks
normally, and plain (non-goal) tasks park freely: their park is a
human-facing pause by design, and the task page's force wake is the
resume path.

## Browser notifications (Web Push)

The app speaks the real Web Push protocol (VAPID + RFC 8291 payload
encryption, no third-party service). The VAPID keypair is generated at
startup on first run and persisted in **`web-config.json`** under the
`vapid` section (`public_key` / `private_key`, base64url P-256) — that
file is the single home for the keys; delete the section to rotate
(regenerate = restart). `SADDLE_VAPID_SUBJECT` overrides the JWT's
`mailto:` contact (default `mailto:clanker-saddle@localhost`).

To enable notifications in a browser: open the dashboard and press
**Enable notifications** — the page requests permission, registers the
service worker (`GET /sw.js`, scope `/`) and subscribes with the app's
VAPID key. "Turn off notifications" reverses it. The bell is
**per-browser**: it compares THIS browser's subscription (by a
SHA-256 fingerprint of its endpoint — the endpoint URL itself never
leaves the server) against your subscribed browsers, so each browser
shows its own state; when you are subscribed elsewhere it says so and
still offers Enable here. **Send test notification** (visible when
this browser is subscribed) pushes a real notification through the
full protocol and reports the true per-endpoint result, so a broken
setup surfaces its actual reason instead of silence. Per task,
**Notify me** / **Don't notify me** (on the task page's status card)
opts you into that task's first start, failure and completion; task
questions, RFCs and review requests notify every subscribed user, and
goal blocks plus goal success/failure ALWAYS notify every subscribed
user. Clicking a notification opens the task's or goal's page.
Delivery is best-effort and truthful: failed sends retry on the
dispatcher tick (up to five attempts — after which the drop is logged
loudly, never silently claimed as delivered), endpoints answering
410/404 are pruned automatically, and other 4xx refusals (e.g. a VAPID
rejection) keep the subscription row and retry, since the endpoint
may recover.

## Deleting a project

**Admin-only**, from the project page's *Delete project* card (and the
`DELETE /api/projects/{id}` JSON route). Deletion cascades by
**ownership only**: the project row and the rows it owns — its direct
tasks (goal-run tasks included), its goals, and those rows' children
(comments, blocks, progress/log events, sessions, rule and capability
attachments, the project's notes, understanding revisions, link rows
and stack memberships) — all go in **one transaction**.

Work that merely *references* the deleted project is **never touched**:
another project's tasks, goals or rows that name it as a co-developed or
reference link survive untouched; only the now-dangling link *entry*
(project_links / goal_links) is tidied away. Standalone tasks
(`project_id` 0) are never deleted either, and neither are the
rule/capability/stack *definitions* — only attachment rows go.

Two guards keep it safe: the delete **refuses while the project still
has tasks queued, working or awaiting review** (HTTP 409 — cancel or let
them finish first), and the confirmation dialog requires typing the
project's **exact name** (the server re-checks it, so a stale tab cannot
delete a renamed project). The dialog shows what will be removed
("3 goals, 9 tasks") and says explicitly that linked projects' work is
left alone. Git branches on saddles are out of scope — deleting the
database rows never touches a clone.

## Stacks

**Stacks** (`/stacks`) are named collections of reference projects:
build the collection once (projects with their relationships —
co-developed or reference), then **Apply stack** on a project page or a
goal page attaches the whole collection to that project's or goal's
linked projects in one click. Applying skips exact duplicates (same
project, same relationship), so re-applying after the collection grows
is safe; a different relationship on the same project coexists.
Deleting a stack never unlinks what it already applied.

## Rules and project notes

**Rules** (`/rules`) are reusable instructions dispatched with every task
that carries them: create them on the rules page, then attach/detach them
on the project page (they lead every task of the project) and the task
page (they lead the task's own set). A dispatch ships the effective set —
task rules, then the primary project's, then the co-developed projects'
in link order, deduplicated; reference projects contribute nothing.

**Project notes** are written by agents mid-task via `saddle note` and
curated by humans on the project page (delete only — there is no manual
create). Daemons POST notes to this app's
`POST /api/saddle/note` callback, authenticated by the per-saddle bearer
token the daemon persisted at registration (plan 10; the shared
`saddle-callback-…` token is gone). The callback base
URL daemons must dial is **this app's own externally reachable address**
(it differs from the browser's when the daemon runs in a container) and
is configured with the top-level `saddle_callback_base_url` key in
`web-config.json`, falling back to the `SADDLE_CALLBACK_BASE_URL`
environment variable and then `http://localhost:8080`:

json { "statum": { "…": "…" }, "saddle_callback_base_url": "http://clanker-web:8080" }

Dispatches also carry the primary and co-developed projects' notes (up to
30 entries / 20 000 characters, oldest first) so agents start with the
accumulated project knowledge.

## AI Chat (advisor)

**AI Chat** is a conversational advisor inside the web app: an LLM loop
that can READ the fleet's state — projects, tasks, goals (plans and
completion reports), task logs (summaries, comments and progress),
rules, discoveries, capability tags, the canonical texts and the
ethos — through a read-only tool set, ask you questions mid-chat, and
SUGGEST tasks or goals for your approval. It can never run commands,
edit files or dispatch work itself; nothing is created until you
approve a suggestion in the chat UI. It is a conversation partner,
not a report generator: after every tool call it tells you what it
found before deciding next steps, and it prefers asking you over
guessing whenever a decision or preference is involved.

The dashboard's top carries the **Ask the advisor** composer: type your
question and you land on a new chat session (`/chats/{id}`). The
**/chats** page lists your conversations (with per-row delete); chats
are **private per user** — nobody else can see, continue or delete
yours. Note that what the advisor READS is the whole fleet's work state
(projects, tasks and goals are not per-user in this app), the same as
every other page.

**Questions.** When the advisor needs your decision it asks a
multiple-choice question (with an "Other" free-text box, like the
saddle's own question tool): the chat parks until you answer, and the
answer resumes the turn.

**Suggestions and approvals.** The advisor proposes work as a card in
the chat — a task (title, mandate, mode, project, priority) or a goal
(title, brief, project, branch). The card is inert until you decide:

- **Approve** (task) → a draft task is created, creator *human* — you
  enqueue it yourself whenever you like.
- **Approve & enqueue** (task) → the draft is created and immediately
  queued for a saddle.
- **Approve** (goal) → a draft goal is created, **unqueued** — the
  advisor never starts a goal; you start it from the goal page.
- **Decline** → the card is dismissed, nothing is created.

**Completion callback.** A periodic pass (every 15 s,
`ADVISOR_CALLBACK_TICK_S`) watches every approved task suggestion for
its task's terminal state: when the task completes or fails, a system
message lands in the originating chat (the task's summary included),
the page refreshes through the state pulse, and the advisor is woken
to react — report the result back, suggest follow-ups — so the
conversation continues from real outcomes.

**Configuration.** The advisor uses its own OpenAI-compatible
chat-completions endpoint — the advisor-specific override fields on the
**Admin → Model** page (`/models`), falling back to the main endpoint's
fields when the override base URL is empty:

text Admin → Model → Advisor endpoint (optional):

base URL   http://127.0.0.1:9999     (empty = use the main endpoint)
API key    …                         (empty = use the main key)
model      mock                      (empty = use the main model)

When no endpoint resolves at all (neither the override nor the main
base URL), the composer disables itself and the chat pages show a
"not configured" notice pointing at the Admin → Model page — nothing
else changes.

## The canon and the ethos (single source of truth)

The canonical texts — the curated library the ethos is derived from —
live in `canon/` at the repo root and are installed into
`share/clanker-saddle/canon`. The web is their **only** owner: saddle
daemons ship no local copy, and both the advisor and every daemon
agent read them from here. `CLANKER_CANON_DIR` overrides the directory
for tests and unusual layouts.

Two per-saddle-token endpoints serve the library (the same bearer
auth as every `/api/saddle/*` route):

- `GET /api/saddle/canon` — the listing: `{count, source_dir?, note?,
  texts: [{slug, title, author, source, licence, summary, words,
  bytes}]}` (headers only, no bodies).
- `GET /api/saddle/canon/{slug}` — one full document: `{slug, title,
  author, source, licence, summary, bytes, body}` (404 on an unknown
  slug).

The **ethos** — the orientation section every system prompt carries —
is one web-side constant (`Services/CanonService.vala`):

- `GET /api/saddle/ethos` serves `{ethos: "…"}` for inspection.
- Every assignment payload carries it inline as the `ethos` member, so
  daemons need no second fetch; they append it verbatim to their
  system prompt (falling back to their compiled-in copy only when an
  older web sends nothing).
- The advisor's system prompt appends the same constant, and its
  `list_canonical_texts` / `read_canonical_text` /
  `query_canonical_text` tools mirror the daemon's — the third runs a
  fresh single-turn side call to the same LLM endpoint, so a canon
  document never enters the chat conversation.

## Integration testing

`test/integration_test.py` drives BOTH real binaries — this app and a
real saddle daemon over a real git checkout — through the full goal-era
arc (Statum-style PASS/FAIL): goal create → start → pre-plan question
(one answer resolving the goal-run task's single task_blocks row) →
plan reject-with-feedback → revision → approval on the agreed
verdict words → task fan-out through the real `/api/saddle/task/*`
callbacks → the worker's ephemeral worktree (auto-pruned, branch kept)
→ the coalesced terminal-event resume → merge + branch delete +
push-before-review → completion review accept (gated on the live
per-repo push flags) → goal complete, plus the `/status`
`goal_branches` live probe flags, SSE over the agent's raw terminal,
the DERIVED goal narrative, the push wire surface (typed entries and
block raises riding POST /api/saddle/results, the deleted poll-era
endpoints 404ing) and the stm-markdown renderer. Host run (the harness
prepares a `demo` repo
with a pushable bare origin when the workspace is empty; the database
must be FRESH — the harness refuses a used one):

bash export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=adminpass123 export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace export SPRY_DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite export SADDLE_GOAL_RESUME_S=2 # shrink the coalescing window for tests ./builddir/clanker-saddle 8180 & # 8180 — 8080 is the production port

SADDLE_LLM_BASE_URL above seeds the model configuration once (the

database must be FRESH — the harness refuses a used one)

SADDLE_WORKSPACE_ROOT=/tmp/clanker-e2e/workspace

../saddle/builddir/src/saddle --port 7100 &

WEB=http://localhost:8180 SADDLE=http://localhost:7100 \ WORKSPACE=/tmp/clanker-e2e/workspace \ DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite

python3 test/integration_test.py

The e2e callback base URL comes from the mounted `web-config.json`'s
`saddle_callback_base_url` key (`http://localhost:8180` for a host run)
— the config beats the `SADDLE_CALLBACK_BASE_URL` environment
variable, so set it there rather than exporting a second source of
truth. The harness-prepared `demo` repo and its bare origin both live
on branch `main` (the project registration's default branch matches).

The daemon's `SADDLE_WORKSPACE_ROOT` and the app's
`SADDLE_PROJECTS_ROOT` may point at the same directory for a local run
(the daemon reports its root at registration, and the app's value is
only the fallback for daemons that do not). For a container run point `WEB`/`SADDLE` at the
published ports, pass `SADDLE_HOST=clanker-saddle:7100`, the container's
SQLite file as `DB_PATH`, and start the in-container mock via
`MOCK_CMD='podman exec -d clanker-saddle python3
/usr/libexec/saddle/mock_llm.py --port 9999'` (the test appends the
same scripted responses to whatever command `MOCK_CMD` prefixes). The
suite releases a daemon left blocked by an earlier run via `/stop`
before it starts.

`meson test` additionally runs `test/web_stack_test.py` — a
self-contained suite (no daemon needed) that spawns the app in a scratch
directory plus an in-process fake saddle daemon speaking the wire
contract (thirteen scenario bundles — pages, goalui, push, dispatch,
answers, replay, subtasks, lineage, users, forcewake, goals, lifecycle,
allocation), and asserts the web-stack features: plan-review body hiding,
the effective `rules` array of assignments, project notes with the
`/api/saddle/note` callback and the payload's `project_notes`/`callback`
caps, the ask/allocation flip (concurrent asks race one winner, affinity
pinning, the lost-wake guard), the phase-2 telemetry flip (pushed
chunks idempotent and ordered into the terminal view, results pushes
with blocked/suspended/done/failed state transitions, entry-redelivery
idempotency, the one-probe staleness policy with its dead-daemon
requeue, and the retired `/output`//`/log`//`/start` daemon routes 404ing
while the web side never calls them), and the phase-3 capability
awareness + UI (the assignment `capabilities` context,
`GET /api/saddle/capabilities`, task-callback `--capability` validation
with the valid-tag list in the error, the task page's top status block
and newest-first blocking sections, the dashboard's pile and timeline,
and the slow-ask lost-wake regression), and the plan-12 goal era
(lifecycle, plan/completion reviews with the run-task's single block row
and the
agreed verdict wordings, task callbacks, coalesced terminal-event
resumes, pause queueing, cancel cascade, auto-enqueue-after, goal
notifications and the goal context of worker assignments).

## Realtime channels and the browser's SSE budget

Browsers cap ~6 concurrent connections per origin on HTTP/1.1, and SSE
streams hold theirs forever: Statum's realtime channel (`/_statum/channel`,
one per browser via its SharedWorker — one per tab where no SharedWorker
is available) plus one `/terminal-stream` per open terminal tab. Deep
tab-stacking on HTTP/1.1 can starve the pulse channel silently (task
pages then miss live block updates). Mitigations built in: hidden
terminal tabs close their stream until focused, and task pages refresh
their state on focus in addition to each pulse. Serving the app over
HTTP/2 removes the cap entirely.

## Deployment assumptions (goal era)

- **Per-saddle workspace volumes; git is the coordination medium.** A
  goal's clone workspace (`<root>/goal-<id>`, one clone per involved
  repo on the goal branch) lives on whichever saddle runs the goal
  agent; goal task worktrees (`<root>/task-<task-id>`, ephemeral branch
  `task/<goal-id>-<task-id>`) are cut from `origin/<goal branch>` — not
  the local clone — so a worker on ANOTHER saddle (its own volume)
  starts from the fleet's pushed tip. At every workspace adoption the
  daemon fetches origin and surfaces a loud *unpulled changes* notice
  per repo left behind, instructing `git pull --rebase`. The goal agent
  merges finished task branches via origin (`git fetch origin
  task/<id>` + `git merge origin/task/<id>`), so a branch that only
  exists on another saddle merges exactly like a local one. Consequence:
  every project involved in a goal needs a pushable origin (deploy
  keys), and saddles mount their own `SADDLE_WORKSPACE_ROOT` — no
  shared `/workspace` anywhere.
- **The web app has NO workspace access.** Assignment project paths are
  advisory: they are built from the ASKING saddle's registered
  `workspace_root` (its daemon reports `SADDLE_WORKSPACE_ROOT` at
  `POST /register`; the saddles page shows it), falling back to
  `SADDLE_PROJECTS_ROOT`, then the historical `/workspace`. The daemon
  owns its filesystem and rewrites paths onto its own root.
- **Database upgrades IN PLACE.** `M0001_Initial` remains the full
  fresh-install schema (goals with links/comments, the flattened tasks,
  and the plan-16 deletions already applied: no `api_tokens`, no
  `working_branch`, no saddle-row work cache — blocks, comments and
  narrative ride the run task's own structures). From goal 8 on,
  schema changes ship as INCREMENTAL migrations (`M0002_BatchImprovements`
  added `notes`, `stacks`, `stack_projects` and the per-saddle
  `prompt_note`/`workspace_root` columns) and run automatically at
  startup: a deployed database is upgraded in place, no data loss, no
  deleting the SQLite file. **Never edit an applied migration again** —
  a database that already recorded it will not re-run it, and the pages
  touching the new tables/columns 500 (exactly the live-instance break
  this fixed). Add a new `M0003_...` with the next serial instead; the
  pages bundle pins the recorded serials so a flattened edit fails the
  suite.
- **Restart both sides.** The daemon (`saddle`) and this app must both
  restart onto the goal-era builds together — the wire contract
  (assignment `kind`/`mode`/`creator`/`goal`/`ephemeral_branch`, the
  task callbacks, goal-keyed blocks, `/status` `goal_branches`) is only
  honoured by matching pairs, and the daemon's registration/token
  persists across its restarts while the app re-seeds on a fresh DB.

## Packaging (USM)

`MANIFEST.usm` + `usm-scripts/` package the app as
`bin:clanker-saddle` for the Web-Stack repository:

bash usm manifest package # → clanker-saddle-.usmc ```

Publishing to a USM repository is a repository-side operation (see the USM tooling; usm repository add from the repository's checkout). web-config.json is never packaged (.usmignore); keys are mounted at runtime, never baked into images.