Plan phase 1 of Vala lib

phase-1-plan.md

@@ -0,0 +1,144 @@
+# Statum — Astralis Runtime Integration + `pstm-` Pre-rendering Pipeline
+
+## Goal
+Wire Statum into Astralis/Inversion as two coupled subsystems:
+- **Subsystem A (runtime):** the HTTP endpoints from `model.md` (`entrypoint`, `slots`, `channel` + `channel/{id}`, `resource/{name}`), `StateService` as a singleton, realtime channel service, action conventions, and a `StatumConfigurator`.
+- **Subsystem B (`pstm-` pipeline):** a build-time generator that turns app HTML into precompressed Vala page classes (predicate-variant selection, template/fragment composition, guards, resource rewriting) + a resource generator + meson wiring.
+
+Behavioral reference: `frontend-demo/server.php` (PHP reference backend) + `model.md`. Pattern reference: Spry (`SpryModule`, `ComponentEndpoint`, `SseEndpoint`, `StaticResourceProvider`/`ConstantStaticResource`, `spry-mkssr`/`spry-mkconst` tools wired as meson `custom_target`).
+
+---
+
+## Confirmed Design Decisions (from interview)
+
+1. **No Gee.** Use `Invercargill.DataStructures` everywhere. ⇒ migrate existing `DirectiveDto` Gee usage (`Gee.List`/`Gee.ArrayList` → `Lot`/`Series`/`Enumerable`).
+2. **Entrypoint handler = type-based subclass.** `StatumEntrypoint` with `async Lot<DirectiveDto> handle()`; injected `StatumRequest` (uri, matched route params, held slots by `type_name`, query) + `StateService` + `ChannelService`. Same `StatumRequest`/header parsing reused by action handlers.
+3. **Held slots exposed by `type_name`, with decrypted `public` AND `private` data**, scoped to only what the client sent in `X-Statum-Slot` headers.
+4. **Slot keys are random/unpredictable (session-like).** Never app-chosen (auth state lives in slots = trust model). `update(key, state)` mutates in place keeping the key; client round-trips the slot; `StateService` is a short-lived cache for the `transmit_after` window. Server is effectively stateless (slot-as-session).
+5. **Realtime push is `StateService`-driven on mutation.** `StateService.update(key, state)` (and `clear_slot`) auto-push the resulting frame to any channels subscribed to that key; the push is effectively a no-op when no stream is subscribed. Selective fan-out emerges naturally from subscriptions (a counter with no subscribers → no push; a live slot with subscribers → push), matching `server.php` without manual fan-out calls. `update` returns the signed `FrameDto` so the handler reuses it for the response `set` directive (one sign → push + response); set-directive signature idempotency dedupes the acting tab if it is also subscribed. `ChannelService.push_set/push_clear` remain as the low-level mechanism (used internally; still callable for advanced cases).
+6. **`add_page<TPage, TEntrypoint>()`** registers BOTH the generated `StatumPage` (Endpoint for each `<pstm-uri>` route) AND the entrypoint handler in the framework's internal entrypoint route table (dispatched by `/_statum/entrypoint?uri=…`). Also `add_page<TPage>()` (static, no entrypoint) and `add_resource<T>()`. No standalone `add_entrypoint`.
+7. **`pstm-` variant selection is pure.** Page handler evaluates `pstm-if`/`pstm-else-if`/`pstm-else`/`pstm-guard` against the request's held slots only (no app/DB logic in the page handler). Entrypoints are separate handler classes that send state for the template's `type_name`s. Flow: template served → JS requests entrypoint → state hydrates. No SSR beyond guards/ifs.
+8. **Predicate roots:** each `type_name` binds to its held **public** data, or `null` if absent; plus a virtual **`.private`** member exposing the decrypted private data. e.g. `auth != null && auth.role == "admin" && auth.private.secret_rating > 30`. `.private` is server-only (never sent to client).
+9. **`pstm-guard`** has `predicate` + `require` (a `type_name` whose absence fails the guard). Failure → optional `redirect` (HTTP 302) or the app's globally-registered guard error page at `status` (default 403).
+10. **Template composition:** `<pstm-name>` names this file; `<pstm-template>` names the **parent** template to wrap with (by its `<pstm-name>`); templates chain upward (a template may itself declare `<pstm-template>`); `<pstm-content>` is the child outlet; resolved at compile time, **not** route-based. Title prefix/suffix merge up the chain.
+11. **`StatumPage` base in Subsystem A;** generator emits subclasses carrying variant byte arrays + compiled predicate/guard Vala. Author registers each page with one `add_page` call (routes read from the generated class).
+
+---
+
+## Architecture / Data Flow
+- Browser GETs `<pstm-uri>` (e.g. `/dashboard`) → Astralis router → generated `StatumPage` Endpoint → parses `X-Statum-Slot` headers into held slots (public+private) → evaluates guards/`pstm-if` matrix → serves the matching precompressed variant (encoding negotiation + ETag), or 302/error-page on guard failure.
+- Page HTML boots Statum JS → JS GETs `/_statum/entrypoint?uri=/dashboard` → `EntrypointEndpoint` matches `uri` in internal table → resolves scoped `StatumEntrypoint` → `handle()` returns `set`/`subscribe`/… directives (`application/vnd.statum+json`).
+- Actions (app-defined URIs) → `StatumAction` handlers → mutate state via `yield StateService.update(key, state)` (returns the signed frame; auto-pushes to subscribed channels), build the response `set` directive from that frame, return directives.
+- Stale slots → client POSTs frames to `/_statum/slots` → `restore_slot` + re-sign → refreshed `set` directives.
+- Realtime → `GET /_statum/channel` (SSE, first event `channel: <id>`); `PATCH /_statum/channel/{id}` `{subscribe,unsubscribe}`; server pushes `set`/`clear` events to subscribed channels.
+
+---
+
+## Subsystem A — Runtime (implement & validate first)
+
+### A1. Gee → Invercargill migration
+- `src/Model/DirectiveDto.vala`: drop `using Gee;`; change `serialize_array(Gee.List<DirectiveDto>)`/`deserialize_array` to Invercargill `Lot`/`Enumerable` + `Series`/`Iterate`. Update any callers.
+
+### A2. `StateService` changes (`src/StateService.vala`)
+- Register as **singleton** (done in `StatumModule`).
+- Inject `ChannelService` (singleton; one-way — `ChannelService` does NOT inject `StateService`, so no DI cycle).
+- Add `async FrameDto update(string key, State state) throws Error`: set `slot.current_state = state`; bump `last_touched`; sign (`sign_slot` logic) → `FrameDto`; serialize frame → `yield ChannelService.push_set(key, frame_json)` (no-op if no subscribers); return the frame (handler reuses it for the response `set` directive).
+- Add `async void clear_slot(string key)`: remove slot; `yield ChannelService.push_clear(key)` (no-op if no subscribers).
+- Make `get_slot` a pure read (stop mutating `last_touched` there — liveness is now driven by `transmit_after`/`invalid_after` and explicit `update`).
+- Fix `cleanup()`: currently removes recently-touched slots (`last_touched > cutoff` is backwards). Evict slots whose `transmit_after`/`invalid_after` has passed (client will resend the frame via `/_statum/slots`).
+
+### A3. Held-slot resolution service (new)
+- `HeldSlot` record: `key`, `Scope`, `public` (Properties), `private` (Properties, decrypted), `as_at`, `transmit_after?`, `invalid_after?`.
+- `HeldSlotResolver` (singleton): parses each `X-Statum-Slot` header:
+  - Reference form `key=…; as_at=…` → `StateService.get_slot(key)` → build `HeldSlot` from `current_state` (skip if absent/expired).
+  - Base64 frame form → `Base64.decode` → JSON `{content,signer,signature}` → `SigningProvider.verify_frame` → parse `content` as `SnapshotDto` → `EncryptionProvider.read_properties(slot.key, snapshot.private)` → build `HeldSlot`.
+  - Index result by `snapshot.slot.slot_type`.
+- Returns `ReadOnlyAssociative<string, HeldSlot>` (type_name → held).
+
+### A4. `StatumRequest` + handler base classes (new)
+- `StatumRequest`: `uri`, `route_params` (`ReadOnlyAssociative<string,string>`), `held` (type_name → `HeldSlot`), `query` (`Catalogue<string,string>`).
+- `StatumEntrypoint` (abstract): injected `StatumRequest`, `StateService`, `ChannelService`; `abstract async Lot<DirectiveDto> handle()`.
+- `StatumAction` (abstract, for app action endpoints): same + decrypts `X-Statum-Private` via `EncryptionProvider.read_properties("action-private", blob)` → action-private Properties; parses form (`FormDataParser`) + query; `abstract async Lot<DirectiveDto> handle()`.
+- Directive-response helper: `StatumDirectives.to_result(Lot<DirectiveDto>)` → `HttpStringResult(json)` with `Content-Type: application/vnd.statum+json` + `Cache-Control: no-store`.
+
+### A5. Framework endpoints (new, registered in `StatumModule`)
+- `EntrypointEndpoint` (GET `/_statum/entrypoint`, scoped): read `uri` query; match against `EntrypointRouteTable` (singleton list of `(EndpointRoute, Type)`) using `EndpointRoute`-style segment matching; build `StatumRequest` (register scoped); resolve + `handle()`; return directive result. 404 if no match.
+- `SlotPostEndpoint` (POST `/_statum/slots`, scoped): read JSON array of frames; per frame `verify_frame` + `restore_slot`; re-sign (`sign_slot`) → `set` directive; return directives (matches `server.php` refresh).
+- `ChannelEndpoint` (GET `/_statum/channel`, **singleton**, extends `SseEndpoint`): `new_connection` generates random channel id, sends `event: channel\ndata: <id>`, tracks `id→SseStream` + `id→Set<string>` subs; exposes `push_set(key, frame_json)` / `push_clear(key)` sending `event: set`/`event: clear` to subscribed streams; `retry_interval` ≈ 3000.
+- `ChannelSubscriptionEndpoint` (PATCH `/_statum/channel/{id}`, scoped): inject `ChannelEndpoint`; parse `{subscribe:[], unsubscribe:[]}`; update subs; on each newly-subscribed key, if `StateService` has the slot cached, push its current frame (`get_slot`+`sign_slot`) via `push_set`; return `HttpEmptyResult(204)`.
+- `ResourceEndpoint` (GET `/_statum/resource/{name}`, scoped): mirror Spry `StaticResourceProvider` — `inject_all<StatumResource>`; pick best encoding by `Accept-Encoding`; `ETag`/`If-None-Match`; serve. *(Pin singular vs plural path: `model.md`/demo use `/_statum/resource/{name}`; user example said `resources`. Default to `resource`.)*
+
+### A6. Resource types (new)
+- `StatumResource` interface (mirror Spry `StaticResource`): `name`, `get_best_encoding`, `get_etag_for`, `to_result`.
+- `ConstantStatumResource` base (mirror Spry): `name`/`file_hash`/`content_type`, `get_best_encoding`, `get_encoding`, byte-array consts.
+
+### A7. `StatumModule` + `StatumConfigurator` (`src/Statum.vala`)
+- `StatumModule.register_components`: singletons `StateService`, `SigningProvider`, `EncryptionProvider`, `ChannelEndpoint` (as `ChannelService`), `EntrypointRouteTable`, `HeldSlotResolver`; framework endpoints A5 with `EndpointRoute`s (ChannelEndpoint as singleton endpoint; rest scoped).
+- `StatumConfigurator` (like `SpryConfigurator`):
+  - `add_page<TPage>()` / `add_page<TPage,TEntrypoint>()`: read `TPage`'s `<pstm-uri>` route(s); register `TPage` as scoped `Endpoint` per route; if `TEntrypoint`, add `(route, typeof(TEntrypoint))` to `EntrypointRouteTable`.
+  - `add_resource<T>()`: register `T` as `StatumResource`.
+
+### A8. Validate Subsystem A
+- Port `frontend-demo/server.php` semantics to a Vala example (counter/products/cart/user/notice + live broadcast) exercising entrypoint, slots pre-flight, channel subscribe/push, actions, `X-Statum-Private`. Confirm against `js/statum.js` + `js/statum-worker.js`.
+
+---
+
+## Subsystem B — `pstm-` Compile-time Pipeline
+
+### B1. `statum-mkpstm` generator tool (new, Vala executable)
+- Deps: `astralis`, `invercargill`, `invercargill-json`, `libxml2`, `inversion` (uses Astralis compressors + Invercargill expressions + libxml2 HTML parsing).
+- Input: app HTML files — **pages** (have `<pstm-uri>`), **templates** (`<pstm-name>` + `<pstm-content>`, no uri), **fragments** (`<pstm-name>` + optional `<pstm-input>`s, no uri).
+- Output: one Vala `StatumPage` subclass per page.
+
+### B2. Composition (compile-time)
+- Resolve `<pstm-template>` chain upward (page → template → …); inline child body into each ancestor's `<pstm-content>` outlet.
+- Apply `pstm-materialise-as` (default `div`) when embedding a fragment/template-child `<body>`.
+- Inline `<pstm-fragment name="…">` includes; map attributes matching the fragment's `<pstm-input>`s to `stm-def-<x>-as` attributes on the materialised element.
+- Compose `<pstm-title prefix|suffix>` up the chain → final `<title>`.
+- Rewrite `pstm-res-<x>="name"` → `<x>="/_statum/resource/<name>"`.
+- Strip all `pstm-*` directives/elements from emitted HTML (they must not reach the browser — see `model.md` `pstm-` warning).
+
+### B3. Predicate variant matrix + codegen
+- Collect `pstm-if`/`pstm-else-if`/`pstm-else` chains; enumerate the cross-product ⇒ one rendered HTML variant per combination (each branch kept, others removed).
+- For each variant: fully compose (B2), then compress with identity/gzip(9)/zstd(19)/brotli(11); store as `IDENTITY/GZIP/ZSTD/BR` byte arrays.
+- Emit compiled Vala: a `select_variant(held)` that (a) evaluates `pstm-guard`s (presence `require` + `predicate` → `GuardFailure(redirect|status)`), then (b) evaluates the `pstm-if` matrix against the held context → variant index → returns that variant's bytes.
+- **Predicate codegen:** parse each predicate with Invercargill `ExpressionParser` (AST) and emit Vala reading the held context — `type_name` root → held public Properties or `null`; `.private` → held private Properties (virtual); member access → Properties lookup w/ coercion; `!= null` → presence; `==`/`!=`/`>`/etc., `&&`/`||`/`!`, literals.
+  - **Risk:** requires the Invercargill expression AST to be walkable for source emission. **Fallback:** emit Vala that calls the runtime evaluator (`ExpressionParser.parse(str).evaluate(ctx)`) — functional but not "compiled ifs". Decide during B3 based on AST introspectability; document the supported predicate subset either way (root type access, `.private`, null checks, member access, comparisons, logical ops, literals).
+
+### B4. Generated `StatumPage` subclass shape
+- Extends `StatumPage` (Subsystem A) which provides header parsing → held slots (public+private), encoding negotiation, ETag, guard redirect/error-page serving.
+- Subclass provides: variant byte arrays (B3), compiled `select_variant`, and a static list of its `<pstm-uri>` routes (for `add_page`).
+
+### B5. `statum-mkres` resource generator (new, mirror `spry-mkssr`)
+- File → Vala `ConstantStatumResource` subclass with `IDENTITY/GZIP/ZSTD/BR` byte arrays + `get_best_encoding`/`get_encoding`/hash/content-type.
+
+### B6. Meson wiring
+- `tools/meson.build`: build `statum-mkpstm` + `statum-mkres` executables (install).
+- App `meson.build`: `custom_target` per page HTML → generated `.vala`; `custom_target` per resource via `statum-mkres`; include generated sources in the app executable. (Mirror Spry `demo/meson.build` `docs_css_resource` pattern.)
+
+### B7. Validate Subsystem B
+- Example app: a homepage with `pstm-if="auth != null"` (signup CTA vs dashboard variants), a `pstm-guard require="auth"` protected page (redirect on failure), a template chain (`<pstm-template>`), a fragment with `<pstm-input>`, and a `pstm-res-src` image. Verify: correct variant served by held state, hidden branches absent from response source, guard redirect, resource served compressed.
+
+---
+
+## Cross-cutting
+
+### Storage & crypto keys (v1 scope)
+- **In-memory** `StateService` store; **ephemeral** `SigningProvider`/`EncryptionProvider` keys (random per process). ⇒ all outstanding slots/sessions invalidate on restart. SQL persistence (`invercargill-sql`/`sqlite3` deps already wired) + key persistence = **out of scope / future**.
+
+### Naming consistency
+- Mirror `stm-*` for `pstm-*` control flow: `pstm-if`, `pstm-else-if`, `pstm-else` (user wrote `pstm-if-else` once; align to `stm-else-if`).
+
+---
+
+## Risks / Open Items
+- **Predicate→Vala codegen** depends on Invercargill expression AST walkability (B3); runtime-eval fallback documented.
+- **Channel SSE concurrency:** `SseEndpoint` base broadcasts to all streams; per-channel subscription filtering must be implemented in the `ChannelEndpoint` subclass (iterate `get_open_streams()`, filter by subs, `send_event`). Validate send failures close streams.
+- **Singleton endpoint injecting scoped services:** Channel/Resource/Entrypoint endpoints must inject only singletons (`StateService`, `ChannelService`, `HeldSlotResolver`, `EntrypointRouteTable`) — never scoped `Scope`/`HttpContext` at construction (those are per-request). `StatumRequest` is built per-request and registered scoped inside the endpoint.
+- **Resource path** singular vs plural (default `/_statum/resource/{name}`).
+- **`<pstm-uri>` templated routes** (e.g. `/cats/{breed}/info`) must register identically for both the page Endpoint and the entrypoint table so `uri` matching agrees.
+
+## Validation (overall)
+- Subsystem A: `server.php`-equivalent Vala example passes against the real JS client (entrypoint hydration, slots pre-flight after `transmit_after`, channel live push across tabs, action `X-Statum-Private`, directive content-type).
+- Subsystem B: variant-selection/guard/template-chain/fragment/resource example produces correct, source-minimized, precompressed responses.
+- Build: `meson compile` succeeds with generated sources; tools install.