clanker-saddle

clanker-saddle

ClankerSaddle is an LLM task orchestration platform: a Statum + Spry web application that queues work for humans and dispatches it to saddle daemons — one-container agents that work on real git checkouts through bash sessions with full auditability. You design tasks during the day; the agent works while you sleep.

Architecture

Browser ←→ clanker-saddle (Spry app: SQLite queue, Statum UI, dispatcher)
                          ↕ HTTP (daemons push; asks/wakes only go out)
               saddle daemons (Vala: bash session + LLM loop, port 7100)
                          ↕ curl
               any OpenAI-compatible chat-completions endpoint

The web app owns projects, goals, flat tasks and the saddle registry. Work acquisition is ask-based (plan 10): daemons sleep when idle and ask POST /api/saddle/next-task (per-saddle bearer token); the allocation is one write transaction that pairs the asking saddle with the best offerable task — workspace affinity first (a task parked on this saddle is offered only to it), then capability-tag matching. The dispatcher wakes sleeping daemons when work becomes ready (POST /wake, the affine saddle first for parked tasks) and declares a saddle offline when no ask arrives within the lost-wake guard window. Telemetry is push-based (phase 2): daemons flush raw PTY batches to POST /api/saddle/chunks and round results with state transitions to POST /api/saddle/results — blocked raises the question as an answerable block and parks the saddle, suspended parks until the goal agent's next terminal-event resume, done/failed complete or fail the task. Nothing polls for output; GET /status against a daemon is exceptional only — the staleness probe (a silent runner owes one probe after SADDLE_STALE_S; a failed probe requeues its task and marks the saddle offline) and the manual refresh button. A recovered daemon's ask flips an offline row back to idle. Which saddle runs what derives from open sessions — busy iff an open session exists, idle otherwise; only the offline probe state is stored. Capability awareness: every assignment carries a capabilities context — saddle_tags (the allocated saddle's own tags) and fleet_tags (every tag name known across the fleet) — which the daemon renders into its system prompt; GET /api/saddle/capabilities (per-saddle token) feeds its saddle capabilities CLI, and the goal agent's task-create callback attaches explicit, validated capability requirements (unknown tags reject with the valid list). The home dashboard pins OPEN blocks (questions, RFCs, reviews) as newest-first link-through cards above a cross-task activity timeline (last ~50 events, newest first); the task page leads with its status block and orders blocking sections newest-first.

Component	Lives	Role
Web app	src/	pages, actions, services, dispatcher, saddle API
Saddle daemon	../saddle	bash PTY session, LLM loop, control server
Mock LLM	../saddle/test/mock_llm.py	scripted endpoint for testing

Orchestration behaviour to know:

Setting	Meaning
SADDLE_LLM_BASE_URL	LLM endpoint sent with every assignment; unset pauses dispatch
SADDLE_LLM_API_KEY / SADDLE_LLM_MODEL	Optional credentials/model for the above
SADDLE_LOST_WAKE_S	Lost-wake guard window (default 15s): no ask after a wake → the saddle goes offline and the next capable one is woken
SADDLE_STALE_S	Running-liveness window (default 90s): no chunks or results for this long → one exceptional GET /status probe; failure requeues the task and marks the saddle offline
SADDLE_PROJECTS_ROOT	Checkouts are addressed as root/<project-name> inside the saddle container (default /workspace)

Quick start (local)

export WS_PREFIX="$HOME/.local"
export PKG_CONFIG_PATH="$WS_PREFIX/lib64/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
export XDG_DATA_DIRS="$WS_PREFIX/share${XDG_DATA_DIRS:+:$XDG_DATA_DIRS}"
export LD_LIBRARY_PATH="$WS_PREFIX/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

meson setup builddir && ninja -C builddir

Run the app, a saddle daemon and a mock LLM in three shells (the mock's port must be reachable from the daemon; project checkouts live under SADDLE_PROJECTS_ROOT):

export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=change-me
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
./builddir/clanker-saddle 8080

../saddle/builddir/src/saddle --port 7100

python3 ../saddle/test/mock_llm.py --port 9999

Browse to http://localhost:8080, log in with the seeded admin account, register the saddle on /saddles with host localhost:7100, and create your first project and task. spry dev wraps the first shell (rebuild + restart on save); web-config.json (from spry keys) keeps sessions alive across restarts and is never packaged or baked into images.

The database is ./clanker-saddle.sqlite, or SPRY_DB_PATH when set.

Docker setup

Two multi-stage Alpine images build everything from source against musl (see ../Dockerfile.saddle and ../Dockerfile.clanker-saddle):

podman build -f Dockerfile.clanker-saddle -t localhost/clanker-saddle:0.1 ..
podman build -f Dockerfile.saddle -t localhost/saddle:0.1 ..

podman network create clanker-net
podman run -d --name clanker-web --userns=keep-id --network clanker-net \
    -p 8080:8080 -v "$PWD/data:/var/lib/clanker-saddle:Z" \
    -e SPRY_SEED_ADMIN_USERNAME=admin -e SPRY_SEED_ADMIN_PASSWORD=change-me \
    -e SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 \
    -e SADDLE_PROJECTS_ROOT=/workspace \
    -e SADDLE_CALLBACK_BASE_URL=http://clanker-web:8080 \
    localhost/clanker-saddle:0.1
podman run -d --name clanker-saddle --userns=keep-id --network clanker-net \
    -p 7100:7100 -v "$PWD/workspace:/workspace:Z" \
    localhost/saddle:0.1

Register the saddle with its network address (clanker-saddle:7100) — the web app talks to the daemon container-to-container, while the LLM base URL is dialled by the daemon itself (a mock can run inside the saddle container via podman exec -d clanker-saddle …, reachable at http://127.0.0.1:9999). SADDLE_CALLBACK_BASE_URL (or the web-config.json saddle_callback_base_url key) must be the app's address as the daemon dials it — every registered daemon pushes telemetry, task callbacks and notes there. Put the SQLite store on a volume (SPRY_DB_PATH defaults to /var/lib/clanker-saddle/clanker-saddle.sqlite) and mount git checkouts under /workspace. On SELinux hosts the bind mounts need :Z, and rootless podman needs --userns=keep-id for the mounts to be writable. For durable sessions mount web-config.json read-only and point ASTRALIS_CONFIG_PATH at it; otherwise the app runs with ephemeral keys.

Creating your first project and task

1. Project — /projects → name, git URL, default branch. The checkout must exist at $SADDLE_PROJECTS_ROOT/<name> on the saddle's filesystem (clone it into the workspace volume; the agent does not clone). Link related projects as co-developed (agent may modify) or reference (read-only) from the project page.
2. Task — /tasks → project, title, mandate (the description), mode (implement / investigate / review / test) and priority (1 is the highest). The task is created in draft: edit every field, attach rules and capabilities, then press Enqueue (only the creator enqueues — humans their own, the goal agent its own). Chain human work with auto-enqueue after: pick a predecessor from the global task list; the dependent enqueues itself when the predecessor completes (a failed or cancelled predecessor leaves it in draft).
3. Saddle — /saddles (admin) → name plus the daemon's host:port. Registration mints a per-saddle token and delivers it to the daemon's POST /register (with this app's base URL); the daemon then asks for work whenever idle-awake and sleeps on a 204 answer, so a ready task reaches it within one dispatcher tick (wake → ask → allocate).

Goals (/goals) are the orchestrated path: write the Initial Brief and a branch name, start it, and a goal agent investigates (asking pre-plan questions liberally), submits its plan for review, fans flat tasks out through the POST /api/saddle/task/{create,delete,enqueue} callbacks (a per-task worktree per involved repo on an ephemeral branch), merges finished branches into the goal branch, pushes, and submits a completion report for your review. Push-style daemons report through POST /api/saddle/results (round entries plus state transitions) and POST /api/saddle/chunks (raw PTY batches) — every /api/saddle/* call is authenticated by the per-saddle bearer token (removing the saddle revokes it).

Monitoring agent progress

• /tasks — live queue: status chips (draft → queued → assigned → working → blocked → done/failed), mode badges, goal links and the standalone/by-goal filter.
• /tasks/{id} — the control room; it refreshes on every state pulse while a saddle works: comments thread (yours and the agent's), blocks, goal context, session history, and the agent's final summary.
• /tasks/{id}/terminal — the full bash session log: the raw PTY stream the daemon pushes (chunk batches land in session_raw_log and stream over SSE byte-exact), with the typed entry view as the fallback for legacy sessions. The session picker honours ?session_id= across every run a task ever had.
• /goals/{id} — the goal control room: lifecycle head, locked brief, plan expander with revision history, task strip, the supervision narrative and the completion review.
• / — dashboard: the "needs your attention" pile (goal AND task blocks interleaved, newest-first) above the activity timeline, saddle cards and queue summary.

Answering block questions

When an agent needs a human decision it runs saddle block "<question>"; the daemon pushes the blocked state with the block payload, the task flips to blocked (the pulse bump lands the question on live pages without a reload) and the saddle parks. Type the answer, submit — answering IS allocating: the answered task becomes resume-offerable, the dispatcher wakes the affine saddle, and the daemon's ask returns the assignment carrying the verdict as its resume_notice (the stable wordings — "Human selected: …", "Plan approved. Proceed with implementation.", "Review approved. Continue with the task." &c.) with a fresh session id; the parked session closes as parked. A goal agent's blocks ride its RUN TASK's single block row — one answer resolves it and resumes the run. saddle plan parks with the goal's plan-review block: Approve answers "Plan approved. Proceed with implementation." and applies the markdown as the goal's plan (documents land on the owning entity only at acceptance — the pending report lives in its block alone); the feedback textarea returns the agent to planning to revise. The completion review renders as the rich attention block (highlight-to-remark annotations queue onto the rejection); answering "Review approved. Continue with the task." completes the goal and stores the report.

Resumed assignments also carry an optional context_update (markdown string): the delta of the instruction context that changed since the run's last round — the goal's plan approved in the meantime (the approved markdown rides along), rules created, edited or attached to the task/its projects, or new project notes. The daemon appends it as a TURN on resume (prompt-cache friendly) instead of refreshing the system prompt; the member is absent when nothing changed.

Browser notifications (Web Push)

The app speaks the real Web Push protocol (VAPID + RFC 8291 payload encryption, no third-party service). The VAPID keypair is generated at startup on first run and persisted in web-config.json under the vapid section (public_key / private_key, base64url P-256) — that file is the single home for the keys; delete the section to rotate (regenerate = restart). SADDLE_VAPID_SUBJECT overrides the JWT's mailto: contact (default mailto:clanker-saddle@localhost).

To enable notifications in a browser: open the dashboard and press Enable notifications — the page requests permission, registers the service worker (GET /sw.js, scope /) and subscribes with the app's VAPID key. "Turn off notifications" reverses it. Per task, Notify me / Don't notify me (on the task page's status card) opts you into that task's first start, failure and completion; task questions, RFCs and review requests notify every subscribed user, and goal blocks plus goal success/failure ALWAYS notify every subscribed user. Clicking a notification opens the task's or goal's page. Delivery is best-effort: failed sends retry on the dispatcher tick and endpoints answering 410/404 are pruned automatically.

Rules and project notes

Rules (/rules) are reusable instructions dispatched with every task that carries them: create them on the rules page, then attach/detach them on the project page (they lead every task of the project) and the task page (they lead the task's own set). A dispatch ships the effective set — task rules, then the primary project's, then the co-developed projects' in link order, deduplicated; reference projects contribute nothing.

Project notes are written by agents mid-task via saddle note and curated by humans on the project page (delete only — there is no manual create). Daemons POST notes to this app's POST /api/saddle/note callback, authenticated by the per-saddle bearer token the daemon persisted at registration (plan 10; the shared saddle-callback-… token is gone). The callback base URL daemons must dial is this app's own externally reachable address (it differs from the browser's when the daemon runs in a container) and is configured with the top-level saddle_callback_base_url key in web-config.json, falling back to the SADDLE_CALLBACK_BASE_URL environment variable and then http://localhost:8080:

{
  "statum": { "…": "…" },
  "saddle_callback_base_url": "http://clanker-web:8080"
}

Dispatches also carry the primary and co-developed projects' notes (up to 30 entries / 20 000 characters, oldest first) so agents start with the accumulated project knowledge.

Integration testing

test/integration_test.py drives BOTH real binaries — this app and a real saddle daemon over a real git checkout — through the full goal-era arc (Statum-style PASS/FAIL): goal create → start → pre-plan question (one answer resolving the goal-run task's single task_blocks row) → plan reject-with-feedback → revision → approval on the agreed verdict words → task fan-out through the real /api/saddle/task/* callbacks → the worker's ephemeral worktree (auto-pruned, branch kept) → the coalesced terminal-event resume → merge + branch delete + push-before-review → completion review accept (gated on the live per-repo push flags) → goal complete, plus the /status goal_branches live probe flags, SSE over the agent's raw terminal, the DERIVED goal narrative, the push wire surface (typed entries and block raises riding POST /api/saddle/results, the deleted poll-era endpoints 404ing) and the stm-markdown renderer. Host run (the harness prepares a demo repo with a pushable bare origin when the workspace is empty; the database must be FRESH — the harness refuses a used one):

export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=adminpass123
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
export SADDLE_CALLBACK_BASE_URL=http://localhost:8080
export SPRY_DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite
export SADDLE_GOAL_RESUME_S=2   # shrink the coalescing window for tests
./builddir/clanker-saddle 8080 &

SADDLE_WORKSPACE_ROOT=/tmp/clanker-e2e/workspace \
    ../saddle/builddir/src/saddle --port 7100 &

WEB=http://localhost:8080 SADDLE=http://localhost:7100 \
WORKSPACE=/tmp/clanker-e2e/workspace \
DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite \
    python3 test/integration_test.py

The daemon's SADDLE_WORKSPACE_ROOT and the app's SADDLE_PROJECTS_ROOT point at the SAME directory (the goal clone the daemon cuts must anchor the worker worktrees — the shared-volume deployment assumption). For a container run point WEB/SADDLE at the published ports, pass SADDLE_HOST=clanker-saddle:7100, the container's SQLite file as DB_PATH, and start the in-container mock via MOCK_CMD='podman exec -d clanker-saddle python3 /usr/libexec/saddle/mock_llm.py --port 9999' (the test appends the same scripted responses to whatever command MOCK_CMD prefixes). The suite releases a daemon left blocked by an earlier run via /stop before it starts.

meson test additionally runs test/web_stack_test.py — a self-contained suite (no daemon needed) that spawns the app in a scratch directory plus an in-process fake saddle daemon speaking the wire contract, and asserts the web-stack features: plan-review body hiding, the effective rules array of assignments, project notes with the /api/saddle/note callback and the payload's project_notes/callback caps, the ask/allocation flip (concurrent asks race one winner, affinity pinning, the lost-wake guard), the phase-2 telemetry flip (pushed chunks idempotent and ordered into the terminal view, results pushes with blocked/suspended/done/failed state transitions, entry-redelivery idempotency, the one-probe staleness policy with its dead-daemon requeue, and the retired /output///log///start daemon routes 404ing while the web side never calls them), and the phase-3 capability awareness + UI (the assignment capabilities context, GET /api/saddle/capabilities, task-callback --capability validation with the valid-tag list in the error, the task page's top status block and newest-first blocking sections, the dashboard's pile and timeline, and the slow-ask lost-wake regression), and the plan-12 goal era (lifecycle, plan/completion reviews with the run-task's single block row and the agreed verdict wordings, task callbacks, coalesced terminal-event resumes, pause queueing, cancel cascade, auto-enqueue-after, goal notifications and the goal context of worker assignments).

Realtime channels and the browser's SSE budget

Browsers cap ~6 concurrent connections per origin on HTTP/1.1, and SSE streams hold theirs forever: Statum's realtime channel (/_statum/channel, one per browser via its SharedWorker — one per tab where no SharedWorker is available) plus one /terminal-stream per open terminal tab. Deep tab-stacking on HTTP/1.1 can starve the pulse channel silently (task pages then miss live block updates). Mitigations built in: hidden terminal tabs close their stream until focused, and task pages refresh their state on focus in addition to each pulse. Serving the app over HTTP/2 removes the cap entirely.

Deployment assumptions (goal era)

• Shared workspace volume across saddles. A goal's clone workspace (<root>/goal-<id>, one clone per involved repo on the goal branch) anchors every task worktree (<root>/task-<task-id>, ephemeral branch task/<goal-id>-<task-id>). Whichever saddle runs a goal task must see the same SADDLE_WORKSPACE_ROOT volume as the saddle that runs the goal agent — today the fleet is single-saddle per goal by construction (one goal agent, one worker at a time on the one registered saddle); multi-saddle fleets must mount the same volume everywhere before fanning goal work out.
• Database tear-down-and-recreate. The schema is a single flattened M0001_Initial (goals with links/comments, the flattened tasks, and the plan-16 deletions already applied: no api_tokens, no working_branch, no saddle-row work cache — blocks, comments and narrative ride the run task's own structures). There is NO upgrade path from an earlier era's database — delete the SQLite file (or point SPRY_DB_PATH at a fresh path) and let the app recreate it.
• Restart both sides. The daemon (saddle) and this app must both restart onto the goal-era builds together — the wire contract (assignment kind/mode/creator/goal/ephemeral_branch, the task callbacks, goal-keyed blocks, /status goal_branches) is only honoured by matching pairs, and the daemon's registration/token persists across its restarts while the app re-seeds on a fresh DB.

Deployment (USM)

MANIFEST.usm + usm-scripts/ package the app as bin:clanker-saddle (usm manifest package), published to the Web-Stack repository:

usm manifest package
cd ../web-stack-repo && usm repository add ../clanker-saddle-0.1.usmc

web-config.json is never packaged (.usmignore); keys are mounted at runtime, never baked into images.