clanker-saddle

clanker-saddle

ClankerSaddle is an LLM task orchestration platform: a Statum + Spry web application that queues work for humans and dispatches it to saddle daemons — one-container agents that work on real git checkouts through bash sessions with full auditability. You design tasks during the day; the agent works while you sleep.

Architecture

Browser ←→ clanker-saddle (Spry app: SQLite queue, Statum UI, dispatcher)
                          ↕ HTTP (poll-based, 5s ticks)
              saddle daemons (Vala: bash session + LLM loop, port 7100)
                          ↕ curl
              any OpenAI-compatible chat-completions endpoint

The web app owns projects, tasks, dependencies and the saddle registry. Work acquisition is ask-based (plan 10): daemons sleep when idle and ask POST /api/saddle/next-task (per-saddle bearer token); the allocation is one write transaction that pairs the asking saddle with the best offerable task — workspace affinity first (a task parked on this saddle is offered only to it), then capability-tag matching. The dispatcher wakes sleeping daemons when work becomes ready (POST /wake, the affine saddle first for parked tasks) and declares a saddle offline when no ask arrives within the lost-wake guard window. During the phase-1 transition the poll-based telemetry stays: busy saddles are still polled for GET /output, GET /log and GET /status every five seconds (with the heal/fence layers), and pre-flip daemons that never ask keep their /answer and /start resume paths.

Component	Lives	Role
Web app	src/	pages, actions, services, dispatcher, saddle API
Saddle daemon	../saddle	bash PTY-free session, LLM loop, control server
Mock LLM	../saddle/test/mock_llm.py	scripted endpoint for testing

Orchestration behaviour to know:

Setting	Meaning
SADDLE_LLM_BASE_URL	LLM endpoint sent with every assignment; unset pauses dispatch
SADDLE_LLM_API_KEY / SADDLE_LLM_MODEL	Optional credentials/model for the above
SADDLE_LOST_WAKE_S	Lost-wake guard window (default 15s): no ask after a wake → the saddle goes offline and the next capable one is woken
SADDLE_PROJECTS_ROOT	Checkouts are addressed as root/<project-name> inside the saddle container (default /workspace)

Quick start (local)

export WS_PREFIX="$HOME/.local"
export PKG_CONFIG_PATH="$WS_PREFIX/lib64/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
export XDG_DATA_DIRS="$WS_PREFIX/share${XDG_DATA_DIRS:+:$XDG_DATA_DIRS}"
export LD_LIBRARY_PATH="$WS_PREFIX/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

meson setup builddir && ninja -C builddir

Run the app, a saddle daemon and a mock LLM in three shells (the mock's port must be reachable from the daemon; project checkouts live under SADDLE_PROJECTS_ROOT):

export SPRY_SEED_ADMIN_USERNAME=admin SPRY_SEED_ADMIN_PASSWORD=change-me
export SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 SADDLE_LLM_MODEL=mock
export SADDLE_PROJECTS_ROOT=/tmp/clanker-e2e/workspace
./builddir/clanker-saddle 8080

../saddle/builddir/src/saddle --port 7100

python3 ../saddle/test/mock_llm.py --port 9999

Browse to http://localhost:8080, log in with the seeded admin account, register the saddle on /saddles with host localhost:7100, and create your first project and task. spry dev wraps the first shell (rebuild + restart on save); web-config.json (from spry keys) keeps sessions alive across restarts and is never packaged or baked into images.

The database is ./clanker-saddle.sqlite, or SPRY_DB_PATH when set.

Docker setup

Two multi-stage Alpine images build everything from source against musl (see ../Dockerfile.saddle and ../Dockerfile.clanker-saddle):

podman build -f Dockerfile.clanker-saddle -t localhost/clanker-saddle:0.1 ..
podman build -f Dockerfile.saddle -t localhost/saddle:0.1 ..

podman network create clanker-net
podman run -d --name clanker-web --userns=keep-id --network clanker-net \
    -p 8080:8080 -v "$PWD/data:/var/lib/clanker-saddle:Z" \
    -e SPRY_SEED_ADMIN_USERNAME=admin -e SPRY_SEED_ADMIN_PASSWORD=change-me \
    -e SADDLE_LLM_BASE_URL=http://127.0.0.1:9999 \
    -e SADDLE_PROJECTS_ROOT=/workspace \
    localhost/clanker-saddle:0.1
podman run -d --name clanker-saddle --userns=keep-id --network clanker-net \
    -p 7100:7100 -v "$PWD/workspace:/workspace:Z" \
    localhost/saddle:0.1

Register the saddle with its network address (clanker-saddle:7100) — the web app talks to the daemon container-to-container, while the LLM base URL is dialled by the daemon itself (a mock can run inside the saddle container via podman exec -d clanker-saddle …, reachable at http://127.0.0.1:9999). Put the SQLite store on a volume (SPRY_DB_PATH defaults to /var/lib/clanker-saddle/clanker-saddle.sqlite) and mount git checkouts under /workspace. On SELinux hosts the bind mounts need :Z, and rootless podman needs --userns=keep-id for the mounts to be writable. For durable sessions mount web-config.json read-only and point ASTRALIS_CONFIG_PATH at it; otherwise the app runs with ephemeral keys.

Creating your first project and task

1. Project — /projects → name, git URL, default branch. The checkout must exist at $SADDLE_PROJECTS_ROOT/<name> on the saddle's filesystem (clone it into the workspace volume; the agent does not clone). Link related projects as co-developed (agent may modify) or reference (read-only) from the project page.
2. Task — /tasks → project, title, description, priority (1 is highest), optionally a design task (explore/plan/question only). Chain work with dependencies: a task waits until everything it depends on is done.
3. Saddle — /saddles (admin) → name plus the daemon's host:port. Registration mints a per-saddle token and delivers it to the daemon's POST /register (with this app's base URL); a phase-1 daemon then asks for work whenever idle-awake and sleeps on a 204 answer, so a ready task reaches it within one dispatcher tick (wake → ask → allocate). Pre-flip daemons without /register still register — copy the token from the saddle card instead.

Tasks can also arrive from the agent itself: saddle subtask lines create child tasks, and push-style daemons may report through POST /api/saddle/{id}/update (output, comment, block, subtask, done, status) and collect answers from GET /api/saddle/{id}/poll — every /api/saddle/* call is authenticated by the per-saddle bearer token (removing the saddle revokes it).

Monitoring agent progress

• /tasks — live queue: status chips (queued → assigned → working → blocked → done/failed), dependency states and ready flags.
• /tasks/{id} — the control room; it refreshes every three seconds while a saddle works: comments thread (yours and the agent's), blocks, subtasks, session history, and the agent's final summary.
• /tasks/{id}/terminal — the full bash session log: every command the agent ran with its combined output and exit code, plus its commentary lines (# …) and saddle events.
• / — dashboard: saddle cards, queue summary, recent activity.

Answering block questions

When an agent needs a human decision it runs saddle block "<question>"; the task flips to blocked and the question appears on its page with an answer form. Type the answer, submit — the dispatcher delivers it to the daemon within five seconds, the agent resumes with the answer in its conversation, and the task returns to working. Answers are delivered exactly once per question.

Rules and project notes

Rules (/rules) are reusable instructions dispatched with every task that carries them: create them on the rules page, then attach/detach them on the project page (they lead every task of the project) and the task page (they lead the task's own set). A dispatch ships the effective set — task rules, then the primary project's, then the co-developed projects' in link order, deduplicated; reference projects contribute nothing.

Project notes are written by agents mid-task via saddle note and curated by humans on the project page (delete only — there is no manual create). Daemons POST notes to this app's POST /api/saddle/note callback, authenticated by the per-saddle bearer token the daemon persisted at registration (plan 10; the shared saddle-callback-… token is gone). The callback base URL daemons must dial is this app's own externally reachable address (it differs from the browser's when the daemon runs in a container) and is configured with the top-level saddle_callback_base_url key in web-config.json, falling back to the SADDLE_CALLBACK_BASE_URL environment variable and then http://localhost:8080:

{
  "statum": { "…": "…" },
  "saddle_callback_base_url": "http://clanker-web:8080"
}

Dispatches also carry the primary and co-developed projects' notes (up to 30 entries / 20 000 characters, oldest first) so agents start with the accumulated project knowledge.

Integration testing

test/integration_test.py drives the whole loop against a running app + daemon pair (88 checks, Statum-style PASS/FAIL): explore task, dependency gating, then the Part 2 feature tour — saddle plan, saddle query with piped stdin, progress updates, a multiple-choice block answered through the web UI, an RFC block with a markdown body, the saddle review banner approved via the UI, a quoted user comment delivered over /notify, terminal SSE (replay + live), the bearer-token ingest API and the stm-markdown renderer. Host run:

mkdir -p /tmp/clanker-e2e/workspace && cd /tmp/clanker-e2e/workspace \
    && git init demo && cd demo && git commit --allow-empty -m one \
    && git commit --allow-empty -m two

WEB=http://localhost:8080 SADDLE=http://localhost:7100 \
WORKSPACE=/tmp/clanker-e2e/workspace \
DB_PATH=/tmp/clanker-e2e/web/clanker-saddle.sqlite \
    python3 test/integration_test.py

For a container run point WEB/SADDLE at the published ports, pass SADDLE_HOST=clanker-saddle:7100, the container's SQLite file as DB_PATH, and start the in-container mock via MOCK_CMD='podman exec -d clanker-saddle python3 /usr/libexec/saddle/mock_llm.py --port 9999' (the test appends the same scripted responses to whatever command MOCK_CMD prefixes). The suite releases a daemon left blocked by an earlier run via /stop before it starts.

meson test additionally runs test/web_stack_test.py — a self-contained suite (no daemon needed) that spawns the app in a scratch directory plus an in-process fake saddle daemon speaking the wire contract, and asserts the web-stack features: plan-review body hiding, the effective rules array of /start, project notes with the /api/saddle/note callback and the payload's project_notes/callback caps, and raw-log polling into the terminal view.

Realtime channels and the browser's SSE budget

Browsers cap ~6 concurrent connections per origin on HTTP/1.1, and SSE streams hold theirs forever: Statum's realtime channel (/_statum/channel, one per browser via its SharedWorker — one per tab where no SharedWorker is available) plus one /terminal-stream per open terminal tab. Deep tab-stacking on HTTP/1.1 can starve the pulse channel silently (task pages then miss live block updates). Mitigations built in: hidden terminal tabs close their stream until focused, and task pages refresh their state on focus in addition to each pulse. Serving the app over HTTP/2 removes the cap entirely.

Deployment (USM)

MANIFEST.usm + usm-scripts/ package the app as bin:clanker-saddle (usm manifest package), published to the Web-Stack repository:

usm manifest package
cd ../web-stack-repo && usm repository add ../clanker-saddle-0.1.usmc

web-config.json is never packaged (.usmignore); keys are mounted at runtime, never baked into images.