Incremental M0002 migration: upgrade serial-1 databases in place

The schema convention was one flattened M0001_Initial applied to a
tear-down-and-recreate database, which only worked while the app shipped
to fresh databases. This goal's tasks edited M0001 in place to add the
notes table, the stacks and stack_projects tables, and the saddles
prompt_note and workspace_root columns. A deployed database that already
recorded M0001 never re-runs it, so every page touching those tables or
columns returned 500 - the live instance's dashboard and goal pages.

M0002_BatchImprovements (serial 2) brings any serial-1 database up to
the current shape in place. It creates the three missing tables and
appends the two saddles columns, each guarded so a fresh database (where
M0001 already created them) skips the operation and records M0002 as a
no-op. The guards introspect the schema through a separate read-only
sqlite handle resolved from the same SPRY_DB_PATH main.vala uses,
because the migration builder only queues operations for later
execution inside a single transaction and offers no if-exists helpers;
the runner runs up() before that transaction opens, so the introspection
is safe. tasks.project_id is deliberately not altered: the goal-8
default-0 change is satisfied by the ORM always writing the column, and
the live constraint text (NOT NULL with the projects FK, no default) is
inert at runtime since foreign keys are never enforced.

Verified end to end on a copy of the live database: the app boots, M0002
records as serial 2, the home dashboard, goals, tasks, projects, stacks
and saddles pages plus every goal-detail page return 200, existing rows
are intact, and a dashboard note insert through the ORM lands in the
migrated notes table. A fresh database records both migrations and ends
with the same 25 tables, 7 indexes and column sets as the upgraded one.

The pages bundle now pins the recorded serials at exactly 1 and 2 and
asserts the new tables and columns exist, so a future flattened edit of
an applied migration fails the suite. The README documents that the
database is upgradeable from here on and applied migrations must never
be edited again.

README.md

@@ -101,6 +101,10 @@ restart on save); `web-config.json` (from `spry keys`) keeps sessions
 alive across restarts and is never packaged or baked into images.
 
 The database is `./clanker-saddle.sqlite`, or `SPRY_DB_PATH` when set.
+It is UPGRADEABLE: migrations are incremental from `M0002` on and apply
+automatically at startup, so a deployed database keeps its data across
+upgrades. An old single-migration database (serial 1 only) upgrades in
+place on first boot of a newer build.
 
 ## Container deployment (spry deploy / usm)
 
@@ -516,13 +520,21 @@ HTTP/2 removes the cap entirely.
   `POST /register`; the saddles page shows it), falling back to
   `SADDLE_PROJECTS_ROOT`, then the historical `/workspace`. The daemon
   owns its filesystem and rewrites paths onto its own root.
-- **Database tear-down-and-recreate.** The schema is a single flattened
-  `M0001_Initial` (goals with links/comments, the flattened tasks, and
-  the plan-16 deletions already applied: no `api_tokens`, no
+- **Database upgrades IN PLACE.** `M0001_Initial` remains the full
+  fresh-install schema (goals with links/comments, the flattened tasks,
+  and the plan-16 deletions already applied: no `api_tokens`, no
   `working_branch`, no saddle-row work cache — blocks, comments and
-  narrative ride the run task's own structures). There is NO upgrade
-  path from an earlier era's database — delete the SQLite file (or
-  point `SPRY_DB_PATH` at a fresh path) and let the app recreate it.
+  narrative ride the run task's own structures). From goal 8 on,
+  schema changes ship as INCREMENTAL migrations (`M0002_BatchImprovements`
+  added `notes`, `stacks`, `stack_projects` and the per-saddle
+  `prompt_note`/`workspace_root` columns) and run automatically at
+  startup: a deployed database is upgraded in place, no data loss, no
+  deleting the SQLite file. **Never edit an applied migration again** —
+  a database that already recorded it will not re-run it, and the pages
+  touching the new tables/columns 500 (exactly the live-instance break
+  this fixed). Add a new `M0003_...` with the next serial instead; the
+  pages bundle pins the recorded serials so a flattened edit fails the
+  suite.
 - **Restart both sides.** The daemon (`saddle`) and this app must both
   restart onto the goal-era builds together — the wire contract
   (assignment `kind`/`mode`/`creator`/`goal`/`ephemeral_branch`, the

meson.build

@@ -171,6 +171,7 @@ app_sources = files(
     'src/Format.vala',
     'src/Data/Entities.vala',
     'src/Data/Migrations/M0001_Initial.vala',
+    'src/Data/Migrations/M0002_BatchImprovements.vala',
     'src/Services/ProjectService.vala',
     'src/Services/TaskService.vala',
     'src/Services/GoalService.vala',

src/Data/Migrations/M0002_BatchImprovements.vala

@@ -0,0 +1,196 @@
+using InvercargillSql.Migrations;
+using Sqlite;
+
+namespace ClankerSaddle.Data.Migrations {
+
+    /**
+     * Goal 8: the incremental upgrade that un-flattens the schema.
+     *
+     * M0001 was historically treated as tear-down-and-recreate — every
+     * later change edited it in place. A deployed database (the live
+     * instance) carries `__migrations` row (1, "clanker-saddle", 1),
+     * so an in-place edit of M0001 never re-runs there: every page
+     * touching the new tables/columns 500s. This migration brings any
+     * serial-1 database up to the current M0001 shape IN PLACE.
+     *
+     * Adds (exactly what the goal-8 tasks folded into M0001):
+     *   - `stacks`, `stack_projects` (task 2 — reference-project bundles)
+     *   - `notes` (task 3 — the dashboard scratchpad)
+     *   - `saddles.prompt_note` (task 5 — per-saddle prompt preamble)
+     *   - `saddles.workspace_root` (task 6 — per-saddle workspace paths)
+     *
+     * NOT altered: `tasks.project_id`. Goal 8 task 4 changed its M0001
+     * definition (dropped the projects FK + index, added DEFAULT 0) for
+     * standalone tasks. A live serial-1 database already has the column
+     * — NOT NULL, no default, with the FK — and SQLite cannot ALTER a
+     * column in place. The difference is semantically inert at runtime:
+     * foreign keys are not enforced (no `PRAGMA foreign_keys=ON` is
+     * ever set — see the connection setup in main.vala), and the ORM
+     * always writes project_id explicitly (0 for standalone tasks), so
+     * the column default is never exercised. Both shapes work; the
+     * constraint text in sqlite_master simply differs.
+     *
+     * IDEMPOTENCE: the runner applies each migration exactly once (the
+     * `__migrations` insert is the guard), so up() only ever runs on a
+     * database this migration has not touched. But it must succeed BOTH
+     * on a serial-1 database (live) and, defensively, on a fresh one
+     * where M0001 already created all of this (e.g. a database created
+     * by this build then handed to an older build, or a partially
+     * upgraded copy). The framework offers no guarded create/add, and
+     * up() cannot catch a failure itself: the builder merely QUEUES
+     * operations, which the runner executes later inside a single
+     * transaction (any duplicate would abort the whole migration). So
+     * up() introspects first — the runner invokes up() BEFORE opening
+     * that transaction (verified against the InvercargillSql runner
+     * source: migration.up(builder) runs, then begin_transaction, then
+     * builder.execute) — and only queues the operations the schema
+     * still needs. A fresh database queues nothing and the migration
+     * records as a no-op.
+     */
+    public class M0002_BatchImprovements : Migration {
+
+        public override string name { get { return "Goal 8 batch improvements: stacks, notes, per-saddle columns"; } }
+        public override uint64 serial { get { return 2; } }
+        public override string migration_namespace { get { return "clanker-saddle"; } }
+
+        // The database this migration is upgrading. The runner never
+        // hands up() a connection (the builder only queues SQL), so the
+        // schema is read through a separate read-only handle. SPRY_DB_PATH
+        // is the same variable main.vala resolves the connection string
+        // from, so this always targets the database actually being
+        // migrated.
+        private string db_path() {
+            var path = Environment.get_variable("SPRY_DB_PATH") ?? "";
+            if (path.strip().length == 0) {
+                return "./clanker-saddle.sqlite";
+            }
+            return path;
+        }
+
+        private bool table_exists(string name) {
+            Database db;
+            int rc = Database.open_v2(db_path(), out db, Sqlite.OPEN_READONLY);
+            if (rc != Sqlite.OK) {
+                // Unopenable (locked, or missing — a database the runner
+                // is about to create): treat everything as absent and
+                // let the queued operations fail loudly rather than
+                // silently skipping an upgrade.
+                return false;
+            }
+            Statement statement;
+            rc = db.prepare_v2(
+                "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?",
+                -1, out statement);
+            if (rc != Sqlite.OK) {
+                return false;
+            }
+            statement.bind_text(1, name, -1);
+            if (statement.step() != Sqlite.ROW) {
+                return false;
+            }
+            return statement.column_int64(0) > 0;
+        }
+
+        private bool column_exists(string table, string column) {
+            Database db;
+            int rc = Database.open_v2(db_path(), out db, Sqlite.OPEN_READONLY);
+            if (rc != Sqlite.OK) {
+                return false;
+            }
+            Statement statement;
+            rc = db.prepare_v2(
+                "SELECT COUNT(*) FROM pragma_table_info(?) WHERE name = ?",
+                -1, out statement);
+            if (rc != Sqlite.OK) {
+                return false;
+            }
+            statement.bind_text(1, table, -1);
+            statement.bind_text(2, column, -1);
+            if (statement.step() != Sqlite.ROW) {
+                return false;
+            }
+            return statement.column_int64(0) > 0;
+        }
+
+        public override void up(MigrationBuilder b) {
+            // Stacks (goal 8 task 2) — same shapes M0001 now defines.
+            if (!table_exists("stacks")) {
+                b.create_table("stacks", (t) => {
+                    t.column<int64?>("id")
+                        .auto_increment()
+                        .primary_key();
+                    t.column<string>("name")
+                        .not_null()
+                        .unique();
+                    t.column<string>("summary");
+                    t.column<DateTime?>("created_at");
+                });
+            }
+            if (!table_exists("stack_projects")) {
+                b.create_table("stack_projects", (t) => {
+                    t.column<int64?>("id")
+                        .auto_increment()
+                        .primary_key();
+                    t.column<int64?>("stack_id")
+                        .not_null()
+                        .references("stacks", "id")
+                        .indexed();
+                    t.column<int64?>("project_id")
+                        .not_null()
+                        .references("projects", "id")
+                        .indexed();
+                    t.column<string>("relationship")
+                        .not_null()
+                        .default_value<string>("reference");
+                    t.column<DateTime?>("created_at");
+                    t.unique("uq_stack_projects_pair", {"stack_id", "project_id"});
+                });
+            }
+
+            // Dashboard scratchpad notes (goal 8 task 3).
+            if (!table_exists("notes")) {
+                b.create_table("notes", (t) => {
+                    t.column<int64?>("id")
+                        .auto_increment()
+                        .primary_key();
+                    t.column<string>("body")
+                        .not_null();
+                    t.column<bool>("done")
+                        .not_null()
+                        .default_value<bool>(false);
+                    t.column<DateTime?>("done_at");
+                    t.column<int64?>("goal_id")
+                        .references("goals", "id");
+                    t.column<DateTime?>("created_at");
+                });
+            }
+
+            // Per-saddle columns (tasks 5 and 6).
+            if (!column_exists("saddles", "prompt_note")) {
+                b.alter_table("saddles", (t) => {
+                    t.add_column<string>("prompt_note");
+                });
+            }
+            if (!column_exists("saddles", "workspace_root")) {
+                b.alter_table("saddles", (t) => {
+                    t.add_column<string>("workspace_root");
+                });
+            }
+        }
+
+        public override void down(MigrationBuilder b) {
+            // Column drops first (the two saddles additions), then the
+            // three tables. drop_column is not universal across SQLite
+            // builds older than 3.35, but the runner's rollback path is
+            // unused by the app (nothing calls rollback); down() exists
+            // to satisfy the abstract class.
+            b.alter_table("saddles", (t) => {
+                t.drop_column("workspace_root");
+                t.drop_column("prompt_note");
+            });
+            b.drop_table("notes");
+            b.drop_table("stack_projects");
+            b.drop_table("stacks");
+        }
+    }
+}

src/main.vala

@@ -43,6 +43,7 @@ int main(string[] args) {
             print(@"[ClankerSaddle] sqlite pragmas failed: $(pragma_error.message)\n");
         }
         db.add_migration<ClankerSaddle.Data.Migrations.M0001_Initial>();
+        db.add_migration<ClankerSaddle.Data.Migrations.M0002_BatchImprovements>();
         db.add_entity<Project>(Project.entity_mapping);
         db.add_entity<ProjectLink>(ProjectLink.entity_mapping);
         db.add_entity<TaskEntity>(TaskEntity.entity_mapping);

test/bundles/pages.py

@@ -810,3 +810,33 @@ def run(ctx):
           "an active task's Edit button is draft-gated (stm-if hides it)")
     check(editor_at < 0 or card_html.rfind('stm-if="task.editable_all"', 0, editor_at) > 0,
           "an active task's editor form sits inside the draft-gated branch")
+
+    # =====================================================================
+    # Goal 8 task 38: the schema is UPGRADEABLE now — M0002 is the
+    # incremental migration that un-flattened it. On this bundle's
+    # fresh boot both migrations ran: assert serial 2 is recorded and
+    # every object M0002 owns exists, so a future in-place edit of
+    # M0001 (the regression this goal shipped against) is caught by
+    # the serial staying 1 here while the schema drifts.
+    print("== migrations: incremental schema (M0002 recorded) ==")
+    conn = db()
+    serials = conn.execute(
+        "select serial from __migrations where namespace = 'clanker-saddle'"
+        " order by serial").fetchall()
+    conn.close()
+    check([s[0] for s in serials] == [1, 2],
+          "the clanker-saddle namespace carries exactly serials 1 and 2"
+          " (a fresh boot ran M0001 then the M0002 no-op)")
+    check(len(serials) == 2,
+          "no duplicate migration rows (M0002's guarded up() applied once)")
+
+    conn = db()
+    tables = {r[0] for r in conn.execute(
+        "select name from sqlite_master where type = 'table'")}
+    saddle_cols = {r[1] for r in conn.execute(
+        "pragma table_info('saddles')")}
+    conn.close()
+    check({"notes", "stacks", "stack_projects"} <= tables,
+          "M0002's tables (notes, stacks, stack_projects) exist")
+    check({"prompt_note", "workspace_root"} <= saddle_cols,
+          "M0002's saddles columns (prompt_note, workspace_root) exist")