clanker-saddle/README.md

clanker-saddle

ClankerSaddle is an LLM task orchestration platform: a Statum + Spry web application that queues work for humans and dispatches it to saddle daemons — one-container agents that work on real git checkouts through bash sessions with full auditability. You design tasks during the day; the agent works while you sleep.

Architecture

Browser ←→ clanker-saddle (Spry app: SQLite queue, Statum UI, dispatcher)
                          ↕ HTTP (daemons push; asks/wakes only go out)
               saddle daemons (Vala: bash session + LLM loop, port 7100)
                          ↕ curl
               any OpenAI-compatible chat-completions endpoint

The web app owns projects, tasks, dependencies and the saddle registry. Work acquisition is ask-based (plan 10): daemons sleep when idle and ask POST /api/saddle/next-task (per-saddle bearer token); the allocation is one write transaction that pairs the asking saddle with the best offerable task — workspace affinity first (a task parked on this saddle is offered only to it), then capability-tag matching. The dispatcher wakes sleeping daemons when work becomes ready (POST /wake, the affine saddle first for parked tasks) and declares a saddle offline when no ask arrives within the lost-wake guard window. Telemetry is push-based (phase 2): daemons flush raw PTY batches to POST /api/saddle/chunks and round results with state transitions to POST /api/saddle/results — blocked raises the question as an answerable block and parks the saddle, suspended parks for the subtask-event resume, done/failed complete or fail the task. Nothing polls for output; GET /status against a daemon is exceptional only — the staleness probe (a silent runner owes one probe after SADDLE_STALE_S; a failed probe requeues its task and marks the saddle offline) and the manual refresh button. A recovered daemon's ask flips an offline row back to idle.

Component	Lives	Role
Web app	src/	pages, actions, services, dispatcher, saddle API
Saddle daemon	../saddle	bash PTY-free session, LLM loop, control server
Mock LLM	../saddle/test/mock_llm.py	scripted endpoint for testing

Orchestration behaviour to know:

Setting	Meaning
SADDLE_LLM_BASE_URL	LLM endpoint sent with every assignment; unset pauses dispatch
SADDLE_LLM_API_KEY / SADDLE_LLM_MODEL	Optional credentials/model for the above
SADDLE_LOST_WAKE_S	Lost-wake guard window (default 15s): no ask after a wake → the saddle goes offline and the next capable one is woken
SADDLE_STALE_S	Running-liveness window (default 90s): no chunks or results for this long → one exceptional GET /status probe; failure requeues the task and marks the saddle offline
SADDLE_PROJECTS_ROOT	Checkouts are addressed as root/<project-name> inside the saddle container (default /workspace)

Quick start (local)

export WS_PREFIX="$HOME/.local"
export PKG_CONFIG_PATH="$WS_PREFIX/lib64/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
export XDG_DATA_DIRS="$WS_PREFIX/share${XDG_DATA_DIRS:+:$XDG_DATA_DIRS}"
export LD_LIBRARY_PATH="$WS_PREFIX/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

meson setup builddir && ninja -C builddir

Run the app, a saddle daemon and a mock LLM in three shells (the mock's port must be reachable from the daemon; project checkouts live under SADDLE_PROJECTS_ROOT):

export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=change-me
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
./builddir/clanker-saddle 8080

../saddle/builddir/src/saddle --port 7100

python3 ../saddle/test/mock_llm.py --port 9999

Browse to http://localhost:8080, log in with the seeded admin account, register the saddle on /saddles with host localhost:7100, and create your first project and task. spry dev wraps the first shell (rebuild + restart on save); web-config.json (from spry keys) keeps sessions alive across restarts and is never packaged or baked into images.

The database is ./clanker-saddle.sqlite, or SPRY_DB_PATH when set.

Docker setup

Two multi-stage Alpine images build everything from source against musl (see ../Dockerfile.saddle and ../Dockerfile.clanker-saddle):

podman build -f Dockerfile.clanker-saddle -t localhost/clanker-saddle:0.1 ..
podman build -f Dockerfile.saddle -t localhost/saddle:0.1 ..

podman network create clanker-net
podman run -d --name clanker-web --userns=keep-id --network clanker-net \
    -p 8080:8080 -v "$PWD/data:/var/lib/clanker-saddle:Z" \
    -e SPRY_SEED_ADMIN_USERNAME=admin -e SPRY_SEED_ADMIN_PASSWORD=change-me \
    -e SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 \
    -e SADDLE_PROJECTS_ROOT=/workspace \
    localhost/clanker-saddle:0.1
podman run -d --name clanker-saddle --userns=keep-id --network clanker-net \
    -p 7100:7100 -v "$PWD/workspace:/workspace:Z" \
    localhost/saddle:0.1

Register the saddle with its network address (clanker-saddle:7100) — the web app talks to the daemon container-to-container, while the LLM base URL is dialled by the daemon itself (a mock can run inside the saddle container via podman exec -d clanker-saddle …, reachable at http://127.0.0.1:9999). Put the SQLite store on a volume (SPRY_DB_PATH defaults to /var/lib/clanker-saddle/clanker-saddle.sqlite) and mount git checkouts under /workspace. On SELinux hosts the bind mounts need :Z, and rootless podman needs --userns=keep-id for the mounts to be writable. For durable sessions mount web-config.json read-only and point ASTRALIS_CONFIG_PATH at it; otherwise the app runs with ephemeral keys.

Creating your first project and task

1. Project — /projects → name, git URL, default branch. The checkout must exist at $SADDLE_PROJECTS_ROOT/<name> on the saddle's filesystem (clone it into the workspace volume; the agent does not clone). Link related projects as co-developed (agent may modify) or reference (read-only) from the project page.
2. Task — /tasks → project, title, description, priority (1 is highest), optionally a design task (explore/plan/question only). Chain work with dependencies: a task waits until everything it depends on is done.
3. Saddle — /saddles (admin) → name plus the daemon's host:port. Registration mints a per-saddle token and delivers it to the daemon's POST /register (with this app's base URL); a phase-1 daemon then asks for work whenever idle-awake and sleeps on a 204 answer, so a ready task reaches it within one dispatcher tick (wake → ask → allocate). Pre-flip daemons without /register still register — copy the token from the saddle card instead.

Tasks can also arrive from the agent itself: saddle subtask lines create child tasks (through the synchronous POST /api/saddle/subtask callback), and push-style daemons report through POST /api/saddle/results (round entries plus state transitions) and POST /api/saddle/chunks (raw PTY batches) — every /api/saddle/* call is authenticated by the per-saddle bearer token (removing the saddle revokes it).

Monitoring agent progress

• /tasks — live queue: status chips (queued → assigned → working → blocked → done/failed), dependency states and ready flags.
• /tasks/{id} — the control room; it refreshes on every state pulse while a saddle works: comments thread (yours and the agent's), blocks, subtasks, session history, and the agent's final summary.
• /tasks/{id}/terminal — the full bash session log: the raw PTY stream the daemon pushes (chunk batches land in session_raw_log and stream over SSE byte-exact), with the typed entry view as the fallback for legacy sessions.
• / — dashboard: saddle cards, queue summary, recent activity.

Answering block questions

When an agent needs a human decision it runs saddle block "<question>"; the daemon pushes the blocked state with the block payload, the task flips to blocked (the pulse bump lands the question on live pages without a reload) and the saddle parks. Type the answer, submit — the task becomes resume-offerable, the dispatcher wakes the affine saddle, and the daemon's ask returns the assignment with the answer in its conversation; the task returns to working. Answers ride the resume assignment exactly once per question.

Rules and project notes

Rules (/rules) are reusable instructions dispatched with every task that carries them: create them on the rules page, then attach/detach them on the project page (they lead every task of the project) and the task page (they lead the task's own set). A dispatch ships the effective set — task rules, then the primary project's, then the co-developed projects' in link order, deduplicated; reference projects contribute nothing.

Project notes are written by agents mid-task via saddle note and curated by humans on the project page (delete only — there is no manual create). Daemons POST notes to this app's POST /api/saddle/note callback, authenticated by the per-saddle bearer token the daemon persisted at registration (plan 10; the shared saddle-callback-… token is gone). The callback base URL daemons must dial is this app's own externally reachable address (it differs from the browser's when the daemon runs in a container) and is configured with the top-level saddle_callback_base_url key in web-config.json, falling back to the SADDLE_CALLBACK_BASE_URL environment variable and then http://localhost:8080:

{
  "statum": { "…": "…" },
  "saddle_callback_base_url": "http://clanker-web:8080"
}

Dispatches also carry the primary and co-developed projects' notes (up to 30 entries / 20 000 characters, oldest first) so agents start with the accumulated project knowledge.

Integration testing

test/integration_test.py drives the whole loop against a running app + daemon pair (88 checks, Statum-style PASS/FAIL): explore task, dependency gating, then the Part 2 feature tour — saddle plan, saddle query with piped stdin, progress updates, a multiple-choice block answered through the web UI, an RFC block with a markdown body, the saddle review banner approved via the UI, a quoted user comment delivered over /notify, terminal SSE (replay + live), the bearer-token ingest API and the stm-markdown renderer. Host run:

mkdir -p /tmp/clanker-e2e/workspace && cd /tmp/clanker-e2e/workspace \
    && git init demo && cd demo && git commit --allow-empty -m one \
    && git commit --allow-empty -m two

WEB=http://localhost:8080 SADDLE=http://localhost:7100 \
WORKSPACE=/tmp/clanker-e2e/workspace \
DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite \
    python3 test/integration_test.py

For a container run point WEB/SADDLE at the published ports, pass SADDLE_HOST=clanker-saddle:7100, the container's SQLite file as DB_PATH, and start the in-container mock via MOCK_CMD='podman exec -d clanker-saddle python3 /usr/libexec/saddle/mock_llm.py --port 9999' (the test appends the same scripted responses to whatever command MOCK_CMD prefixes). The suite releases a daemon left blocked by an earlier run via /stop before it starts.

meson test additionally runs test/web_stack_test.py — a self-contained suite (no daemon needed) that spawns the app in a scratch directory plus an in-process fake saddle daemon speaking the wire contract, and asserts the web-stack features: plan-review body hiding, the effective rules array of assignments, project notes with the /api/saddle/note callback and the payload's project_notes/callback caps, the ask/allocation flip (concurrent asks race one winner, affinity pinning, the lost-wake guard), and the phase-2 telemetry flip (pushed chunks idempotent and ordered into the terminal view, results pushes with blocked/suspended/done/failed state transitions, entry-redelivery idempotency, the one-probe staleness policy with its dead-daemon requeue, and the retired /output///log///start daemon routes 404ing while the web side never calls them).

Realtime channels and the browser's SSE budget

Browsers cap ~6 concurrent connections per origin on HTTP/1.1, and SSE streams hold theirs forever: Statum's realtime channel (/_statum/channel, one per browser via its SharedWorker — one per tab where no SharedWorker is available) plus one /terminal-stream per open terminal tab. Deep tab-stacking on HTTP/1.1 can starve the pulse channel silently (task pages then miss live block updates). Mitigations built in: hidden terminal tabs close their stream until focused, and task pages refresh their state on focus in addition to each pulse. Serving the app over HTTP/2 removes the cap entirely.

Deployment (USM)

MANIFEST.usm + usm-scripts/ package the app as bin:clanker-saddle (usm manifest package), published to the Web-Stack repository:

usm manifest package
cd ../web-stack-repo && usm repository add ../clanker-saddle-0.1.usmc

web-config.json is never packaged (.usmignore); keys are mounted at runtime, never baked into images.