clanker-saddle

clanker-saddle

ClankerSaddle is an LLM task orchestration platform: a Statum + Spry web application that queues work for humans and dispatches it to saddle daemons — one-container agents that work on real git checkouts through bash sessions with full auditability. You design tasks during the day; the agent works while you sleep.

Architecture

Browser ←→ clanker-saddle (Spry app: SQLite queue, Statum UI, dispatcher)
                          ↕ HTTP (daemons push; asks/wakes only go out)
               saddle daemons (Vala: bash session + LLM loop, port 7100)
                          ↕ curl
               any OpenAI-compatible chat-completions endpoint

The web app owns projects, goals, flat tasks and the saddle registry. Work acquisition is ask-based (plan 10): daemons sleep when idle and ask POST /api/saddle/next-task (per-saddle bearer token); the allocation is one write transaction that pairs the asking saddle with the best offerable task — workspace affinity first (a task parked on this saddle is offered only to it), then capability-tag matching. The dispatcher wakes sleeping daemons when work becomes ready (POST /wake, the affine saddle first for parked tasks) and declares a saddle offline when no ask arrives within the lost-wake guard window. Telemetry is push-based (phase 2): daemons flush raw PTY batches to POST /api/saddle/chunks and round results with state transitions to POST /api/saddle/results — blocked raises the question as an answerable block and parks the saddle, suspended parks until the goal agent's next terminal-event resume, done/failed complete or fail the task. Nothing polls for output; GET /status against a daemon is exceptional only — the staleness probe (a silent runner owes one probe after SADDLE_STALE_S; a failed probe requeues its task and marks the saddle offline) and the manual refresh button. A recovered daemon's ask flips an offline row back to idle. Which saddle runs what derives from open sessions — busy iff an open session exists, idle otherwise; only the offline probe state is stored. Capability awareness: every assignment carries a capabilities context — saddle_tags (the allocated saddle's own tags) and fleet_tags (every tag name known across the fleet) — which the daemon renders into its system prompt; GET /api/saddle/capabilities (per-saddle token) feeds its saddle capabilities CLI, and the goal agent's task-create callback attaches explicit, validated capability requirements (unknown tags reject with the valid list). The home dashboard pins OPEN blocks (questions, RFCs, reviews) as newest-first link-through cards above a cross-task activity timeline (last ~50 events, newest first); the task page leads with its status block and orders blocking sections newest-first.

Component	Lives	Role
Web app	src/	pages, actions, services, dispatcher, saddle API
Saddle daemon	../saddle	bash PTY session, LLM loop, control server
Mock LLM	../saddle/test/mock_llm.py	scripted endpoint for testing

Orchestration behaviour to know:

Setting	Meaning
SADDLE_LLM_BASE_URL	LLM endpoint sent with every assignment; unset pauses dispatch
SADDLE_LLM_API_KEY / SADDLE_LLM_MODEL	Optional credentials/model for the above
SADDLE_LOST_WAKE_S	Lost-wake guard window (default 15s): no ask after a wake → the saddle goes offline and the next capable one is woken
SADDLE_STALE_S	Running-liveness window (default 90s): no chunks or results for this long → one exceptional GET /status probe; failure requeues the task and marks the saddle offline
SADDLE_PROJECTS_ROOT	Checkouts are addressed as root/<project-name> inside the saddle container (default /workspace)

Quick start (local)

export WS_PREFIX="$HOME/.local"
export PKG_CONFIG_PATH="$WS_PREFIX/lib64/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
export XDG_DATA_DIRS="$WS_PREFIX/share${XDG_DATA_DIRS:+:$XDG_DATA_DIRS}"
export LD_LIBRARY_PATH="$WS_PREFIX/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

meson setup builddir && ninja -C builddir

Run the app, a saddle daemon and a mock LLM in three shells (the mock's port must be reachable from the daemon; project checkouts live under SADDLE_PROJECTS_ROOT):

export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=change-me
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
./builddir/clanker-saddle 8080

../saddle/builddir/src/saddle --port 7100

python3 ../saddle/test/mock_llm.py --port 9999

Browse to http://localhost:8080, log in with the seeded admin account, register the saddle on /saddles with host localhost:7100, and create your first project and task. spry dev wraps the first shell (rebuild + restart on save); web-config.json (from spry keys) keeps sessions alive across restarts and is never packaged or baked into images.

The database is ./clanker-saddle.sqlite, or SPRY_DB_PATH when set.

Container deployment (spry deploy / usm)

Both halves package as USM packages and build container images from their own checkouts — no Dockerfiles outside the repos:

# this app (a Spry application — spry deploy wraps usm manifest deploy):
spry deploy --verbose \
    --repository https://packages.astrologue.nz/repo/web-stack/repo.usmr

# the daemon (from ../saddle — usm manifest deploy directly; --spm dnf
# is required for the in-container manage dependencies):
cd ../saddle && usm manifest deploy --spm dnf \
    --repository https://packages.astrologue.nz/repo/web-stack/repo.usmr

Each writes <name>-<version>.image.tar.xz (single-stage, built in container against the canonical Web-Stack repository — allow time for the toolchain build on first run). Load and run:

podman load -i clanker-saddle-0.1.image.tar.xz
podman load -i saddle-0.1.image.tar.xz

podman network create clanker-net
podman run -d --name clanker-web --userns=keep-id --network clanker-net \
    -p 8080:8080 -v "$PWD/data:/var/lib/clanker-saddle:Z" \
    -e SPRY_SEED_ADMIN_USERNAME=admin -e SPRY_SEED_ADMIN_PASSWORD=change-me \
    -e SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 \
    -e SADDLE_PROJECTS_ROOT=/workspace \
    -e SADDLE_CALLBACK_BASE_URL=http://clanker-web:8080 \
    localhost/clanker-saddle:0.1
podman run -d --name clanker-saddle --userns=keep-id --network clanker-net \
    -p 7100:7100 -v "$PWD/workspace:/workspace:Z" \
    localhost/saddle:0.1

Register the saddle with its network address (clanker-saddle:7100) — the web app talks to the daemon container-to-container, while the LLM base URL is dialled by the daemon itself (a mock can run inside the saddle container via podman exec -d clanker-saddle …, reachable at http://127.0.0.1:9999). SADDLE_CALLBACK_BASE_URL (or the web-config.json saddle_callback_base_url key) must be the app's address as the daemon dials it — every registered daemon pushes telemetry, task callbacks and notes there. Put the SQLite store on a volume (SPRY_DB_PATH defaults to /var/lib/clanker-saddle/clanker-saddle.sqlite) and mount git checkouts under /workspace. On SELinux hosts the bind mounts need :Z, and rootless podman needs --userns=keep-id for the mounts to be writable. For durable sessions mount web-config.json read-only and point ASTRALIS_CONFIG_PATH at it; otherwise the app runs with ephemeral keys.

Saddles with dev tools (the extendable agent container)

The stock saddle image carries the daemon, its CLI and git — nothing else. Build specialised agents by extending it with the toolchains of your choosing (see ../saddle's README, Extending the saddle image, for the full guide): a two-line Containerfile FROM the loaded base plus dnf install of your tools (C, Vala, GTK, Node, .NET, Android, QEMU — anything), then tag the capability names you gave those tools in the web UI's saddle page so only tasks requiring them route there. Goal task worktrees need every container that may run a goal's tasks to share the /workspace volume.

Creating your first project and task

1. Project — /projects → name, git URL, default branch. The checkout must exist at $SADDLE_PROJECTS_ROOT/<name> on the saddle's filesystem (clone it into the workspace volume; the agent does not clone). Link related projects as co-developed (agent may modify) or reference (read-only) from the project page.
2. Task — /tasks → project, title, mandate (the description), mode (implement / investigate / review / test) and priority (1 is the highest). The task is created in draft: edit every field, attach rules and capabilities, then press Enqueue (only the creator enqueues — humans their own, the goal agent its own). Long-form fields edit in place: an Edit button beside the task page's Mandate card (drafts only) swaps the rendered markdown for a prefilled editor; the same pattern on a draft goal's Initial Brief card edits title, brief and priority before you start it. Chain human work with auto-enqueue after: pick a predecessor from the global task list; the dependent enqueues itself when the predecessor completes (a failed or cancelled predecessor leaves it in draft).
3. Saddle — /saddles (admin) → name plus the daemon's host:port. Registration mints a per-saddle token and delivers it to the daemon's POST /register (with this app's base URL); the daemon then asks for work whenever idle-awake and sleeps on a 204 answer, so a ready task reaches it within one dispatcher tick (wake → ask → allocate).

Goals (/goals) are the orchestrated path: write the Initial Brief and a branch name, start it, and a goal agent investigates (asking pre-plan questions liberally), submits its plan for review, fans flat tasks out through the POST /api/saddle/task/{create,delete,enqueue} callbacks (a per-task worktree per involved repo on an ephemeral branch), merges finished branches into the goal branch, pushes, and submits a completion report for your review. Push-style daemons report through POST /api/saddle/results (round entries plus state transitions) and POST /api/saddle/chunks (raw PTY batches) — every /api/saddle/* call is authenticated by the per-saddle bearer token (removing the saddle revokes it).

Users, roles and your profile

Two roles decide what a logged-in user can do (the account model is Spry's; this app surfaces it):

• user — manages projects, tasks, goals and rules.
• admin — additionally manages users (/users), saddles (register/remove, git identity, capability tags) and the capability catalogue. Viewing /saddles stays open to every logged-in user; the mutation controls are hidden and server-refused for non-admins.

/users (admin-only) lists accounts newest-first with a role badge and enabled state: create users, edit profile fields, reset passwords, enable/disable, grant/revoke admin, delete. Guards keep you from locking yourself out: you cannot delete or disable your own account, cannot revoke your own admin permission, and the last enabled admin cannot be removed.

Every logged-in user has /profile (linked beside the logout button in the nav corner): shows your account details and changes your own password — current, new, confirm. Change the seeded admin's change-me password there before exposing the app.

Monitoring agent progress

• /tasks — live queue: status chips (draft → queued → assigned → working → blocked → done/failed), mode badges, goal links and the standalone/by-goal filter.
• /tasks/{id} — the control room; it refreshes on every state pulse while a saddle works: comments thread (yours and the agent's), blocks, goal context, session history, and the agent's final summary.
• /tasks/{id}/terminal — the full bash session log: the raw PTY stream the daemon pushes (chunk batches land in session_raw_log and stream over SSE byte-exact), with the typed entry view as the fallback for legacy sessions. The session picker honours ?session_id= across every run a task ever had.
• /goals/{id} — the goal control room: lifecycle head, locked brief, plan expander with revision history, task strip, the supervision narrative and the completion review.
• / — dashboard: the "needs your attention" pile (goal AND task blocks interleaved, newest-first) above the activity timeline, saddle cards and queue summary.

Answering block questions

When an agent needs a human decision it runs saddle block "<question>"; the daemon pushes the blocked state with the block payload, the task flips to blocked (the pulse bump lands the question on live pages without a reload) and the saddle parks. Type the answer, submit — answering IS allocating: the answered task becomes resume-offerable, the dispatcher wakes the affine saddle, and the daemon's ask returns the assignment carrying the verdict as its resume_notice (the stable wordings — "Human selected: …", "Plan approved. Proceed with implementation.", "Review approved. Continue with the task." &c.) with a fresh session id; the parked session closes as parked. A goal agent's blocks ride its RUN TASK's single block row — one answer resolves it and resumes the run. saddle plan parks with the goal's plan-review block: Approve answers "Plan approved. Proceed with implementation." and applies the markdown as the goal's plan (documents land on the owning entity only at acceptance — the pending report lives in its block alone); the feedback textarea returns the agent to planning to revise. The completion review renders as the rich attention block (highlight-to-remark annotations queue onto the rejection); answering "Review approved. Continue with the task." completes the goal and stores the report.

Resumed assignments also carry an optional context_update (markdown string): the delta of the instruction context that changed since the run's last round — the goal's plan approved in the meantime (the approved markdown rides along), rules created, edited or attached to the task/its projects, or new project notes. The daemon appends it as a TURN on resume (prompt-cache friendly) instead of refreshing the system prompt; the member is absent when nothing changed.

Browser notifications (Web Push)

The app speaks the real Web Push protocol (VAPID + RFC 8291 payload encryption, no third-party service). The VAPID keypair is generated at startup on first run and persisted in web-config.json under the vapid section (public_key / private_key, base64url P-256) — that file is the single home for the keys; delete the section to rotate (regenerate = restart). SADDLE_VAPID_SUBJECT overrides the JWT's mailto: contact (default mailto:clanker-saddle@localhost).

To enable notifications in a browser: open the dashboard and press Enable notifications — the page requests permission, registers the service worker (GET /sw.js, scope /) and subscribes with the app's VAPID key. "Turn off notifications" reverses it. The bell is per-browser: it compares THIS browser's subscription (by a SHA-256 fingerprint of its endpoint — the endpoint URL itself never leaves the server) against your subscribed browsers, so each browser shows its own state; when you are subscribed elsewhere it says so and still offers Enable here. Send test notification (visible when this browser is subscribed) pushes a real notification through the full protocol and reports the true per-endpoint result, so a broken setup surfaces its actual reason instead of silence. Per task, Notify me / Don't notify me (on the task page's status card) opts you into that task's first start, failure and completion; task questions, RFCs and review requests notify every subscribed user, and goal blocks plus goal success/failure ALWAYS notify every subscribed user. Clicking a notification opens the task's or goal's page. Delivery is best-effort and truthful: failed sends retry on the dispatcher tick (up to five attempts — after which the drop is logged loudly, never silently claimed as delivered), endpoints answering 410/404 are pruned automatically, and other 4xx refusals (e.g. a VAPID rejection) keep the subscription row and retry, since the endpoint may recover.

Rules and project notes

Rules (/rules) are reusable instructions dispatched with every task that carries them: create them on the rules page, then attach/detach them on the project page (they lead every task of the project) and the task page (they lead the task's own set). A dispatch ships the effective set — task rules, then the primary project's, then the co-developed projects' in link order, deduplicated; reference projects contribute nothing.

Project notes are written by agents mid-task via saddle note and curated by humans on the project page (delete only — there is no manual create). Daemons POST notes to this app's POST /api/saddle/note callback, authenticated by the per-saddle bearer token the daemon persisted at registration (plan 10; the shared saddle-callback-… token is gone). The callback base URL daemons must dial is this app's own externally reachable address (it differs from the browser's when the daemon runs in a container) and is configured with the top-level saddle_callback_base_url key in web-config.json, falling back to the SADDLE_CALLBACK_BASE_URL environment variable and then http://localhost:8080:

{
  "statum": { "…": "…" },
  "saddle_callback_base_url": "http://clanker-web:8080"
}

Dispatches also carry the primary and co-developed projects' notes (up to 30 entries / 20 000 characters, oldest first) so agents start with the accumulated project knowledge.

Integration testing

test/integration_test.py drives BOTH real binaries — this app and a real saddle daemon over a real git checkout — through the full goal-era arc (Statum-style PASS/FAIL): goal create → start → pre-plan question (one answer resolving the goal-run task's single task_blocks row) → plan reject-with-feedback → revision → approval on the agreed verdict words → task fan-out through the real /api/saddle/task/* callbacks → the worker's ephemeral worktree (auto-pruned, branch kept) → the coalesced terminal-event resume → merge + branch delete + push-before-review → completion review accept (gated on the live per-repo push flags) → goal complete, plus the /status goal_branches live probe flags, SSE over the agent's raw terminal, the DERIVED goal narrative, the push wire surface (typed entries and block raises riding POST /api/saddle/results, the deleted poll-era endpoints 404ing) and the stm-markdown renderer. Host run (the harness prepares a demo repo with a pushable bare origin when the workspace is empty; the database must be FRESH — the harness refuses a used one):

export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=adminpass123
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
export SADDLE_CALLBACK_BASE_URL=http://localhost:8080
export SPRY_DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite
export SADDLE_GOAL_RESUME_S=2   # shrink the coalescing window for tests
./builddir/clanker-saddle 8080 &

SADDLE_WORKSPACE_ROOT=/tmp/clanker-e2e/workspace \
    ../saddle/builddir/src/saddle --port 7100 &

WEB=http://localhost:8080 SADDLE=http://localhost:7100 \
WORKSPACE=/tmp/clanker-e2e/workspace \
DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite \
    python3 test/integration_test.py

The daemon's SADDLE_WORKSPACE_ROOT and the app's SADDLE_PROJECTS_ROOT point at the SAME directory (the goal clone the daemon cuts must anchor the worker worktrees — the shared-volume deployment assumption). For a container run point WEB/SADDLE at the published ports, pass SADDLE_HOST=clanker-saddle:7100, the container's SQLite file as DB_PATH, and start the in-container mock via MOCK_CMD='podman exec -d clanker-saddle python3 /usr/libexec/saddle/mock_llm.py --port 9999' (the test appends the same scripted responses to whatever command MOCK_CMD prefixes). The suite releases a daemon left blocked by an earlier run via /stop before it starts.

meson test additionally runs test/web_stack_test.py — a self-contained suite (no daemon needed) that spawns the app in a scratch directory plus an in-process fake saddle daemon speaking the wire contract, and asserts the web-stack features: plan-review body hiding, the effective rules array of assignments, project notes with the /api/saddle/note callback and the payload's project_notes/callback caps, the ask/allocation flip (concurrent asks race one winner, affinity pinning, the lost-wake guard), the phase-2 telemetry flip (pushed chunks idempotent and ordered into the terminal view, results pushes with blocked/suspended/done/failed state transitions, entry-redelivery idempotency, the one-probe staleness policy with its dead-daemon requeue, and the retired /output///log///start daemon routes 404ing while the web side never calls them), and the phase-3 capability awareness + UI (the assignment capabilities context, GET /api/saddle/capabilities, task-callback --capability validation with the valid-tag list in the error, the task page's top status block and newest-first blocking sections, the dashboard's pile and timeline, and the slow-ask lost-wake regression), and the plan-12 goal era (lifecycle, plan/completion reviews with the run-task's single block row and the agreed verdict wordings, task callbacks, coalesced terminal-event resumes, pause queueing, cancel cascade, auto-enqueue-after, goal notifications and the goal context of worker assignments).

Realtime channels and the browser's SSE budget

Browsers cap ~6 concurrent connections per origin on HTTP/1.1, and SSE streams hold theirs forever: Statum's realtime channel (/_statum/channel, one per browser via its SharedWorker — one per tab where no SharedWorker is available) plus one /terminal-stream per open terminal tab. Deep tab-stacking on HTTP/1.1 can starve the pulse channel silently (task pages then miss live block updates). Mitigations built in: hidden terminal tabs close their stream until focused, and task pages refresh their state on focus in addition to each pulse. Serving the app over HTTP/2 removes the cap entirely.

Deployment assumptions (goal era)

• Shared workspace volume across saddles. A goal's clone workspace (<root>/goal-<id>, one clone per involved repo on the goal branch) anchors every task worktree (<root>/task-<task-id>, ephemeral branch task/<goal-id>-<task-id>). Whichever saddle runs a goal task must see the same SADDLE_WORKSPACE_ROOT volume as the saddle that runs the goal agent — today the fleet is single-saddle per goal by construction (one goal agent, one worker at a time on the one registered saddle); multi-saddle fleets must mount the same volume everywhere before fanning goal work out.
• Database tear-down-and-recreate. The schema is a single flattened M0001_Initial (goals with links/comments, the flattened tasks, and the plan-16 deletions already applied: no api_tokens, no working_branch, no saddle-row work cache — blocks, comments and narrative ride the run task's own structures). There is NO upgrade path from an earlier era's database — delete the SQLite file (or point SPRY_DB_PATH at a fresh path) and let the app recreate it.
• Restart both sides. The daemon (saddle) and this app must both restart onto the goal-era builds together — the wire contract (assignment kind/mode/creator/goal/ephemeral_branch, the task callbacks, goal-keyed blocks, /status goal_branches) is only honoured by matching pairs, and the daemon's registration/token persists across its restarts while the app re-seeds on a fresh DB.

Packaging (USM)

MANIFEST.usm + usm-scripts/ package the app as bin:clanker-saddle for the Web-Stack repository:

usm manifest package      # → clanker-saddle-<version>.usmc

Publishing to a USM repository is a repository-side operation (see the USM tooling; usm repository add from the repository's checkout). web-config.json is never packaged (.usmignore); keys are mounted at runtime, never baked into images.