clanker-saddle

clanker-saddle

ClankerSaddle is an LLM task orchestration platform: a Statum + Spry web application that queues work for humans and dispatches it to saddle daemons — one-container agents that work on real git checkouts through bash sessions with full auditability. You design tasks during the day; the agent works while you sleep.

Architecture

Browser ←→ clanker-saddle (Spry app: SQLite queue, Statum UI, dispatcher)
                          ↕ HTTP (daemons push; asks/wakes only go out)
               saddle daemons (Vala: bash session + LLM loop, port 7100)
                          ↕ curl
               any OpenAI-compatible chat-completions endpoint

The web app owns projects, goals, flat tasks and the saddle registry. Work acquisition is ask-based (plan 10): daemons sleep when idle and ask POST /api/saddle/next-task (per-saddle bearer token); the allocation is one write transaction that pairs the asking saddle with the best offerable task — workspace affinity first (a task parked on this saddle is offered only to it), then capability-tag matching. The dispatcher wakes sleeping daemons when work becomes ready (POST /wake, the affine saddle first for parked tasks) and declares a saddle offline when no ask arrives within the lost-wake guard window. Telemetry is push-based (phase 2): daemons flush raw PTY batches to POST /api/saddle/chunks and round results with state transitions to POST /api/saddle/results — blocked raises the question as an answerable block and parks the saddle, suspended parks for the subtask-event resume, done/failed complete or fail the task. Nothing polls for output; GET /status against a daemon is exceptional only — the staleness probe (a silent runner owes one probe after SADDLE_STALE_S; a failed probe requeues its task and marks the saddle offline) and the manual refresh button. A recovered daemon's ask flips an offline row back to idle. Capability awareness (phase 3): every assignment carries a capabilities context — saddle_tags (the allocated saddle's own tags) and fleet_tags (every tag name known across the fleet) — which the daemon renders into its system prompt; GET /api/saddle/capabilities (per-saddle token) feeds its saddle capabilities CLI, and the subtask callback's capabilities: [tags] attach explicit, validated requirements (unknown tags reject with the valid list). The home dashboard pins OPEN blocks (questions, RFCs, reviews) as newest-first link-through cards above a cross-task activity timeline (last ~50 events, newest first); the task page leads with its status block and orders blocking sections newest-first.

Component	Lives	Role
Web app	src/	pages, actions, services, dispatcher, saddle API
Saddle daemon	../saddle	bash PTY-free session, LLM loop, control server
Mock LLM	../saddle/test/mock_llm.py	scripted endpoint for testing

Orchestration behaviour to know:

Setting	Meaning
SADDLE_LLM_BASE_URL	LLM endpoint sent with every assignment; unset pauses dispatch
SADDLE_LLM_API_KEY / SADDLE_LLM_MODEL	Optional credentials/model for the above
SADDLE_LOST_WAKE_S	Lost-wake guard window (default 15s): no ask after a wake → the saddle goes offline and the next capable one is woken
SADDLE_STALE_S	Running-liveness window (default 90s): no chunks or results for this long → one exceptional GET /status probe; failure requeues the task and marks the saddle offline
SADDLE_PROJECTS_ROOT	Checkouts are addressed as root/<project-name> inside the saddle container (default /workspace)

Quick start (local)

export WS_PREFIX="$HOME/.local"
export PKG_CONFIG_PATH="$WS_PREFIX/lib64/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
export XDG_DATA_DIRS="$WS_PREFIX/share${XDG_DATA_DIRS:+:$XDG_DATA_DIRS}"
export LD_LIBRARY_PATH="$WS_PREFIX/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

meson setup builddir && ninja -C builddir

Run the app, a saddle daemon and a mock LLM in three shells (the mock's port must be reachable from the daemon; project checkouts live under SADDLE_PROJECTS_ROOT):

export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=change-me
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
./builddir/clanker-saddle 8080

../saddle/builddir/src/saddle --port 7100

python3 ../saddle/test/mock_llm.py --port 9999

Browse to http://localhost:8080, log in with the seeded admin account, register the saddle on /saddles with host localhost:7100, and create your first project and task. spry dev wraps the first shell (rebuild + restart on save); web-config.json (from spry keys) keeps sessions alive across restarts and is never packaged or baked into images.

The database is ./clanker-saddle.sqlite, or SPRY_DB_PATH when set.

Docker setup

Two multi-stage Alpine images build everything from source against musl (see ../Dockerfile.saddle and ../Dockerfile.clanker-saddle):

podman build -f Dockerfile.clanker-saddle -t localhost/clanker-saddle:0.1 ..
podman build -f Dockerfile.saddle -t localhost/saddle:0.1 ..

podman network create clanker-net
podman run -d --name clanker-web --userns=keep-id --network clanker-net \
    -p 8080:8080 -v "$PWD/data:/var/lib/clanker-saddle:Z" \
    -e SPRY_SEED_ADMIN_USERNAME=admin -e SPRY_SEED_ADMIN_PASSWORD=change-me \
    -e SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 \
    -e SADDLE_PROJECTS_ROOT=/workspace \
    -e SADDLE_CALLBACK_BASE_URL=http://clanker-web:8080 \
    localhost/clanker-saddle:0.1
podman run -d --name clanker-saddle --userns=keep-id --network clanker-net \
    -p 7100:7100 -v "$PWD/workspace:/workspace:Z" \
    localhost/saddle:0.1

Register the saddle with its network address (clanker-saddle:7100) — the web app talks to the daemon container-to-container, while the LLM base URL is dialled by the daemon itself (a mock can run inside the saddle container via podman exec -d clanker-saddle …, reachable at http://127.0.0.1:9999). SADDLE_CALLBACK_BASE_URL (or the web-config.json saddle_callback_base_url key) must be the app's address as the daemon dials it — every registered daemon pushes telemetry, task callbacks and notes there. Put the SQLite store on a volume (SPRY_DB_PATH defaults to /var/lib/clanker-saddle/clanker-saddle.sqlite) and mount git checkouts under /workspace. On SELinux hosts the bind mounts need :Z, and rootless podman needs --userns=keep-id for the mounts to be writable. For durable sessions mount web-config.json read-only and point ASTRALIS_CONFIG_PATH at it; otherwise the app runs with ephemeral keys.

Creating your first project and task

1. Project — /projects → name, git URL, default branch. The checkout must exist at $SADDLE_PROJECTS_ROOT/<name> on the saddle's filesystem (clone it into the workspace volume; the agent does not clone). Link related projects as co-developed (agent may modify) or reference (read-only) from the project page.
2. Task — /tasks → project, title, description, priority (1 is highest) and the plan-11 options ("auto-approve plan", "auto-enqueue agent-created subtasks", "auto-approve subtasks"). The task is created in draft: edit every field, attach rules and capabilities, then press Enqueue (dependencies still in draft block the enqueue with an error naming them; nothing depends on a draft before you may delete it). Chain work with dependencies: an enqueued task waits until everything it depends on is done.
3. Saddle — /saddles (admin) → name plus the daemon's host:port. Registration mints a per-saddle token and delivers it to the daemon's POST /register (with this app's base URL); a phase-1 daemon then asks for work whenever idle-awake and sleeps on a 204 answer, so a ready task reaches it within one dispatcher tick (wake → ask → allocate). Pre-flip daemons without /register still register — copy the token from the saddle card instead.

Tasks can also arrive from the agent itself: saddle subtask lines create child tasks (through the synchronous POST /api/saddle/subtask callback), and push-style daemons report through POST /api/saddle/results (round entries plus state transitions) and POST /api/saddle/chunks (raw PTY batches) — every /api/saddle/* call is authenticated by the per-saddle bearer token (removing the saddle revokes it).

Monitoring agent progress

• /tasks — live queue: status chips (draft → queued → assigned → working → blocked → done/failed), mode badges, goal links and the standalone/by-goal filter.
• /tasks/{id} — the control room; it refreshes on every state pulse while a saddle works: comments thread (yours and the agent's), blocks, goal context, session history, and the agent's final summary.
• /tasks/{id}/terminal — the full bash session log: the raw PTY stream the daemon pushes (chunk batches land in session_raw_log and stream over SSE byte-exact), with the typed entry view as the fallback for legacy sessions. The session picker honours ?session_id= across every run a task ever had.
• /goals/{id} — the goal control room: lifecycle head, locked brief, plan expander with revision history, task strip, the supervision narrative and the completion review.
• / — dashboard: the "needs your attention" pile (goal AND task blocks interleaved, newest-first) above the activity timeline, saddle cards and queue summary.

Answering block questions

When an agent needs a human decision it runs saddle block "<question>"; the daemon pushes the blocked state with the block payload, the task flips to blocked (the pulse bump lands the question on live pages without a reload) and the saddle parks. Type the answer, submit — answering IS allocating: the answered task becomes resume-offerable, the dispatcher wakes the affine saddle, and the daemon's ask returns the assignment carrying the verdict as its resume_notice (the stable wordings — "Human selected: …", "Plan approved. Proceed with implementation.", "Review approved. Continue with the task." &c.) with a fresh session id; the parked session closes as resumed. A goal agent's blocks attach to the GOAL (dual-rowed: the goal block plus a quiet mirror on the run task) — one answer resolves both and resumes the run. saddle plan parks with the goal's plan-review block: Approve answers "Plan approved. Proceed with implementation." and applies the markdown as the goal's plan; the feedback textarea returns the agent to planning to revise. The completion review answers "Review approved. Continue with the task." to complete the goal.

Resumed assignments also carry an optional context_update (markdown string): the delta of the instruction context that changed since the run's last round — the goal's plan approved in the meantime (the approved markdown rides along), rules created, edited or attached to the task/its projects, or new project notes. The daemon appends it as a TURN on resume (prompt-cache friendly) instead of refreshing the system prompt; the member is absent when nothing changed.

Browser notifications (Web Push)

The app speaks the real Web Push protocol (VAPID + RFC 8291 payload encryption, no third-party service). The VAPID keypair is generated at startup on first run and persisted in web-config.json under the vapid section (public_key / private_key, base64url P-256) — that file is the single home for the keys; delete the section to rotate (regenerate = restart). SADDLE_VAPID_SUBJECT overrides the JWT's mailto: contact (default mailto:clanker-saddle@localhost).

To enable notifications in a browser: open the dashboard and press Enable notifications — the page requests permission, registers the service worker (GET /sw.js, scope /) and subscribes with the app's VAPID key. "Turn off notifications" reverses it. Per task, Notify me / Don't notify me (on the task page's status card) opts you into that task's first start, failure and completion; task questions, RFCs and review requests notify every subscribed user, and goal blocks plus goal success/failure ALWAYS notify every subscribed user. Clicking a notification opens the task's or goal's page. Delivery is best-effort: failed sends retry on the dispatcher tick and endpoints answering 410/404 are pruned automatically.

Rules and project notes

Rules (/rules) are reusable instructions dispatched with every task that carries them: create them on the rules page, then attach/detach them on the project page (they lead every task of the project) and the task page (they lead the task's own set). A dispatch ships the effective set — task rules, then the primary project's, then the co-developed projects' in link order, deduplicated; reference projects contribute nothing.

Project notes are written by agents mid-task via saddle note and curated by humans on the project page (delete only — there is no manual create). Daemons POST notes to this app's POST /api/saddle/note callback, authenticated by the per-saddle bearer token the daemon persisted at registration (plan 10; the shared saddle-callback-… token is gone). The callback base URL daemons must dial is this app's own externally reachable address (it differs from the browser's when the daemon runs in a container) and is configured with the top-level saddle_callback_base_url key in web-config.json, falling back to the SADDLE_CALLBACK_BASE_URL environment variable and then http://localhost:8080:

{
  "statum": { "…": "…" },
  "saddle_callback_base_url": "http://clanker-web:8080"
}

Dispatches also carry the primary and co-developed projects' notes (up to 30 entries / 20 000 characters, oldest first) so agents start with the accumulated project knowledge.

Integration testing

test/integration_test.py drives BOTH real binaries — this app and a real saddle daemon over a real git checkout — through the full goal-era arc (83 checks, Statum-style PASS/FAIL): goal create → start → pre-plan question (one answer resolving the goal row AND the mirrored run-task row) → plan reject-with-feedback → revision → approval on the agreed verdict words → task fan-out through the real /api/saddle/task/* callbacks → the worker's ephemeral worktree (auto-pruned, branch kept) → the coalesced terminal-event resume → merge + branch delete + push-before-review → completion review accept → goal complete, plus the /status goal_branches cache, SSE over the agent's raw terminal, the goal_events narrative, the legacy ingest API surface and the stm-markdown renderer. Host run (the harness prepares a demo repo with a pushable bare origin when the workspace is empty; the database must be FRESH — the harness refuses a used one):

export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=adminpass123
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
export SADDLE_CALLBACK_BASE_URL=http://localhost:8080
export SPRY_DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite
export SADDLE_GOAL_RESUME_S=2   # shrink the coalescing window for tests
./builddir/clanker-saddle 8080 &

SADDLE_WORKSPACE_ROOT=/tmp/clanker-e2e/workspace \
    ../saddle/builddir/src/saddle --port 7100 &

WEB=http://localhost:8080 SADDLE=http://localhost:7100 \
WORKSPACE=/tmp/clanker-e2e/workspace \
DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite \
    python3 test/integration_test.py

The daemon's SADDLE_WORKSPACE_ROOT and the app's SADDLE_PROJECTS_ROOT point at the SAME directory (the goal clone the daemon cuts must anchor the worker worktrees — the shared-volume deployment assumption). For a container run point WEB/SADDLE at the published ports, pass SADDLE_HOST=clanker-saddle:7100, the container's SQLite file as DB_PATH, and start the in-container mock via MOCK_CMD='podman exec -d clanker-saddle python3 /usr/libexec/saddle/mock_llm.py --port 9999' (the test appends the same scripted responses to whatever command MOCK_CMD prefixes). The suite releases a daemon left blocked by an earlier run via /stop before it starts.

meson test additionally runs test/web_stack_test.py — a self-contained suite (no daemon needed) that spawns the app in a scratch directory plus an in-process fake saddle daemon speaking the wire contract, and asserts the web-stack features: plan-review body hiding, the effective rules array of assignments, project notes with the /api/saddle/note callback and the payload's project_notes/callback caps, the ask/allocation flip (concurrent asks race one winner, affinity pinning, the lost-wake guard), the phase-2 telemetry flip (pushed chunks idempotent and ordered into the terminal view, results pushes with blocked/suspended/done/failed state transitions, entry-redelivery idempotency, the one-probe staleness policy with its dead-daemon requeue, and the retired /output///log///start daemon routes 404ing while the web side never calls them), and the phase-3 capability awareness + UI (the assignment capabilities context, GET /api/saddle/capabilities, task-callback --capability validation with the valid-tag list in the error, the task page's top status block and newest-first blocking sections, the dashboard's pile and timeline, and the slow-ask lost-wake regression), and the plan-12 goal era (lifecycle, plan/completion reviews with dual-rowed goal blocks and the agreed verdict wordings, task callbacks, coalesced terminal-event resumes, pause queueing, cancel cascade, auto-enqueue-after, goal notifications and the goal context of worker assignments).

Realtime channels and the browser's SSE budget

Browsers cap ~6 concurrent connections per origin on HTTP/1.1, and SSE streams hold theirs forever: Statum's realtime channel (/_statum/channel, one per browser via its SharedWorker — one per tab where no SharedWorker is available) plus one /terminal-stream per open terminal tab. Deep tab-stacking on HTTP/1.1 can starve the pulse channel silently (task pages then miss live block updates). Mitigations built in: hidden terminal tabs close their stream until focused, and task pages refresh their state on focus in addition to each pulse. Serving the app over HTTP/2 removes the cap entirely.

Deployment assumptions (goal era)

• Shared workspace volume across saddles. A goal's clone workspace (<root>/goal-<id>, one clone per involved repo on the goal branch) anchors every task worktree (<root>/task-<task-id>, ephemeral branch task/<goal-id>-<task-id>). Whichever saddle runs a goal task must see the same SADDLE_WORKSPACE_ROOT volume as the saddle that runs the goal agent — today the fleet is single-saddle per goal by construction (one goal agent, one worker at a time on the one registered saddle); multi-saddle fleets must mount the same volume everywhere before fanning goal work out.
• Database tear-down-and-recreate. The plan-12 migrations flatten the schema again (a fresh M0001_Initial: goals, goal links/comments/ blocks/events, the flattened tasks). There is NO upgrade path from a plan-11 database — delete the SQLite file (or point SPRY_DB_PATH at a fresh path) and let the app recreate it.
• Restart both sides. The daemon (saddle) and this app must both restart onto the goal-era builds together — the wire contract (assignment kind/mode/creator/goal/ephemeral_branch, the task callbacks, goal-keyed blocks, /status goal_branches) is only honoured by matching pairs, and the daemon's registration/token persists across its restarts while the app re-seeds on a fresh DB.

Deployment (USM)

MANIFEST.usm + usm-scripts/ package the app as bin:clanker-saddle (usm manifest package), published to the Web-Stack repository:

usm manifest package
cd ../web-stack-repo && usm repository add ../clanker-saddle-0.1.usmc

web-config.json is never packaged (.usmignore); keys are mounted at runtime, never baked into images.