usm/LFS_BOOTSTRAP_FEASIBILITY.md

USM Distro Bootstrap — Feasibility & Effort Report

Investigation report for goal 15 ("a USM based Linux distro"), covering every item of the brief: a scaffold repository for LFS bootstrap packages, a bootstrap container, a USM repository of LFS packages, a DCBA repository of LFS build artifacts, a USM-driven chroot build, and a container of the final root with USM installed.

Verdict up front: feasible, with an explicitly staged architecture, and conditioned on the build host. USM's existing mechanics (staged installs via USM_DESTDIR/USM_CONFIGDIR, destination-local resource resolution, file:// repositories, the DCBA build-artifact flow) map well onto an LFS bootstrap — but USM's resolver is single-pass, while LFS is a three-pass toolchain construction (temporary → chroot → final), so the bootstrap cannot be one USM resolution. It must be staged: a handful of driver invocations, each a valid single-pass resolution, chained by an orchestrator. The dominant risks are environmental, not architectural: device nodes and /proc inside a capability-restricted chroot, systemd's build requirements, and self-hosting USM itself (valac + glib and several friends are not in the LFS book — see Stage 5).

Everything below was verified against the USM checkout at commit b046afe ("Merge branch 'goal/dcba' of Tilo15/usm into master", branch master) and the LFS book checkout at branch 13.1 (LFS 13.1, trunk working copy), plus live probes of this build sandbox. File/line citations refer to those checkouts.

---

1. Executive summary

• Feasible. Every one of the six brief items maps onto USM mechanics that exist on master today; none requires new USM core features. What must be built is an orchestration layer (a generator that turns the LFS book into USM scaffolds, stage drivers, and OCI assembly scripts) plus a small beyond-LFS package set so USM can build itself.
• The architecture must be staged. Resolver.resolve (src/lib/Resolver.vala:233) processes manage+build+runtime phases in one pass and produces one topological order (src/lib/Resolver.vala:422-428) — a single resolution cannot express LFS's ch5/6 temporary tools → ch7 chroot tools → ch8 final system sequence, where later packages must be built with the products of earlier ones. The bootstrap is therefore N driven stages, each internally a normal USM install; the stages are ordered by an orchestrator script, not by the resolver.
• Host-conditioned. On a sandbox without CAP_SYS_ADMIN/CAP_MKNOD (this one), chroot builds work but /dev nodes are plain files and /proc//sys are empty. That is fine for the great majority of LFS packages (proven: bison 3.8.2 configured and compiled inside a host-root clone with exactly that setup) but endangers the systemd build and anything reading /proc/mounts. Final artifacts should be produced on a full host (or a rootful container); the sandbox remains excellent for development of the pipeline itself.
• Disk layout matters: / (and /tmp) is a 17 GB overlay while /workspace has ~11 TB free — all LFS scratch trees must live under /workspace. (The host also lacks g++, patch and makeinfo — three chapter-2 requirements — but the sandbox has root and dnf5, so stage 0's image recipe closes that with one dnf install line; see §2.3.)
• Effort. Roughly 7–10 engineer-weeks for the pipeline (generator + stage drivers + OCI scripts + key management + CI), of which the single largest item is the book-to-scaffold generator (~90 packages, ~640 command blocks, needing curated per-package handling). Wall-clock for one full bootstrap run on this 28-core sandbox: ~3–5 hours without test suites, ~8–14 hours with (SBU-derived; see §7). Artifacts published per run: ~90 .usmc source packages (one per package-pass) and as many .dcba build artifacts, plus the signed listings.

2. Verified platform facts

Every probe below was run live in this session (2026-09-23) unless marked "inherited from the goal-15 investigation"; re-running any of them is a one-liner.

2.1 Capabilities and kernel

• grep CapEff /proc/self/status → 00000000800405fb; capsh --decode → cap_chown, cap_dac_override, cap_fowner, cap_fsetid, cap_kill, cap_setgid, cap_setuid, cap_setpcap, cap_net_bind_service, cap_sys_chroot, cap_setfcap. cap_sys_chroot present; cap_mknod, cap_sys_admin, cap_net_admin absent (CapBnd equals CapEff — they cannot be gained). Consequences: chroot(2) works; mknod(2) fails (Operation not permitted); no mounts, no user/pid namespaces, no overlayfs.
• Kernel 7.0.12-101.fc43.x86_64 (uname -r) — comfortably above the book's minimum kernel requirement (the book specifies >= 5.10 for glibc's --enable-kernel reasoning; chapter02/hostreqs.xml).
• chroot re-proven this session: a minimal root (/usr/bin/true + its ldd closure + the dynamic loader) executed via chroot $W /usr/bin/true → "CHROOT EXEC OK".
• Inherited from the goal-15 investigation (not re-run): a clone of the host root with a plain-file /dev/null and empty /proc configured and compiled GNU bison 3.8.2 with gcc inside chroot; the only fix needed was PATH reconstruction under env -i. Also inherited: tar cannot carry device nodes into this sandbox (Cannot mknod: Operation not permitted).

2.2 Resources

• 28 cores (nproc).
• Storage: /workspace on /dev/sdc1 — 19 T total, 11 T free; / is an overlay with only 17 G free (so /tmp is tight — LFS scratch trees must live under /workspace).
• Network: outbound HTTPS works (curl -sI https://ftp.gnu.org/gnu/ → HTTP/1.1 200 OK).

2.3 Toolchain versions and version soundness

• Host gcc 16.2.1 20260819 (Red Hat 16.2.1-2), host glibc 2.43 (ldd --version), Python 3.14.7, perl 5.42.3, bison 3.8.2.
• Book targets (reference/LFS-Stable packages.ent): gcc 16.2.0, glibc 2.44, binutils 2.47, sysvinit 3.14 / systemd 261.3 (both revisions shipped in the book).
• Soundness for a native (non-cross) bootstrap:
  • host glibc 2.43 < book glibc 2.44 → the temporary tools never accidentally link against a newer libc than they are building;
  • host gcc 16.2 matches the book's 16.2.0 — and crucially the book warns (chapter02/hostreqs.xml) that newer gcc/binutils than the book's are untested and likely to break the build; equal is the intended case;
  • kernel 7.0 > minimum → glibc headers configure cleanly.
• Host tool gaps (audited this session against the book's ch2 list): g++/cc1plus absent (gcc -x c++ fails: "cannot execute 'cc1plus'"), patch absent, and makeinfo/texi2any (Texinfo) absent — all three are ch2 requirements (Texinfo-5.0+ is on the list; the glibc/gcc/perl builds invoke makeinfo if present and skip documentation otherwise, but the book expects a compliant host). /usr/bin/yacc is also absent (the book wants yacc as a bison link — bison 3.8.2 itself is present, and the ch5 bison rebuild provides /usr/bin/yacc inside the toolchain, so this only matters for the stage-0 host surface). Everything else on the ch2 list is present: bash 5.3.9, as/ld (binutils 2.46.1), bison, coreutils (incl. nproc), diffutils (diff/cmp), findutils, gawk, gcc, grep, gzip, m4, make, perl, python3, sed, tar, xz. dnf5 5.4.3.0 is present and the missing items are plain dnf installs (gcc-c++, patch, texinfo) → the Stage-0 image build must dnf install gcc-c++ patch texinfo (plus a ch2 re-check script at image-build time).

2.4 Container runtimes

• podman, docker, buildah, skopeo, umoci: all absent. So containers must be assembled (tar layers + config + manifest, i.e. an OCI image layout) by script here; they can be consumed anywhere a runtime exists. usm deploy's podman path (§4.6) remains available on hosts that have podman.

3. USM mechanics the bootstrap relies on (verified on master)

These are the load-bearing behaviours; each was read at the cited location in this session.

1. Staged installs. Paths.override_from_environment (src/lib/Paths.vala:189-207) overrides destination, prefix, every dir, and usm_config_dir from USM_DESTDIR, USM_PREFIX, USM_BINDIR, … USM_CONFIGDIR. Manifest.run_build then calls paths.set_envs() (src/lib/Manifest.vala:289-314), exporting DESTDIR/PREFIX/BINDIR/LIBDIR/… to the build script, and Manifest.run_install clones the paths and overrides destination with the per-package install path (src/lib/Manifest.vala:441-475) before spawning the install script. scripts/dcba-e2e.sh drives two fully isolated roots this way end-to-end (its header documents the recipe: <root>/config via USM_CONFIGDIR, <state>/{lists,packages,installed} pre-created, usm.config with is_managed=true + managed.state_path).
2. ResourceFinder local presence. ResourceFinder.locate_resource (src/lib/ResourceFinder.vala:13-64) resolves refs mostly relative to paths.destination (locate_path, locate_res, locate_cfg, locate_bin walks $PATH under destination, locate_libres, locate_pc including PKG_CONFIG_PATH entries, locate_vapi, locate_typelib, …), so inside a chroot (destination /) resolution sees only the staged root. Caveat (a real constraint, not a blocker): several locators are host-absolute — locate_lib consults ldconfig -p and falls back to absolute /lib /usr/lib /lib64 /usr/lib64 (src/lib/ResourceFinder.vala:75-104), and locate_inc, locate_libexec, locate_gir, locate_info/man/locale, locate_app use absolute /usr/... paths. Inside a chroot that is correct (destination is /); outside a chroot with USM_DESTDIR set, lib:/include: refs can be satisfied by the host — the stage drivers must run usm inside the target chroot, or audited for it.
3. Resolver is single-pass. Resolver.resolve (src/lib/Resolver.vala:233-287) picks roots, does ONE batched SPM query, processes all packages, and resolution_phases folds manage+build+runtime (src/lib/Resolver.vala:422-428) into one dependency graph with one topological install order. Missing resources resolve: (a) local presence, (b) system packages, (c) USM packages — exactly the semantics a stage needs ("what the previous stage left in the root counts as present"), but it cannot reorder the world between two packages in one transaction.
4. file:// repositories. Repository.get_client (src/lib/Repository/Repository.vala:34-53) routes file → GioRepositoryClient, http/https → CurlRepositoryClient (curl -fsSL --max-time). All repositories in the bootstrap can be plain directories on the same filesystem — no server needed.
5. DCBA flow. usm dcba repository init/add (src/cli/Dcba.vala): init generates an ed25519 keypair under keys/ (private 0600) and a .dcbar descriptor; add <file>.dcba validates tar.xz, derives name/version from the filename via DcbaArtifact.parse_filename (src/lib/Repository/DcbaArtifact.vala:22-56) and rebuilds a signed ARTIFACTS.usml (SHA-512 over entry lines, ed25519 signatures — DcbaListingWriter.write, src/lib/Repository/DcbaArtifact.vala:82+). The artifact itself is literally the package cache's build.tar.xz renamed: scripts/dcba-e2e.sh copies <state>/packages/<n>-<v>/build.tar.xz to <n>-<v>.dcba (its §3), produced by CachedPackage.archive_build (src/lib/State/CachedPackage.vala:101-131, digest-gated re-archive). A consumer with the .dcbar configured gets the artifact as its build cache ("Retrieved pre-built artifact"), records provenance in origin-info ("dcba" field), and a tampered artifact is rejected with a warning + fallback to source build — all asserted by the e2e script (assertions a–f).
6. usm deploy. usm manifest deploy (src/cli/Deploy.vala:223-473) generates a Containerfile-based context and (unless --no-build) builds with podman and saves an xz-compressed image. Default base is Fedora (src/cli/Deploy.vala:16); --base overrides. This is the "container from the chroot" path on hosts with podman; here we script OCI assembly instead (Stage 6).
7. Scaffolds. usm scaffold <name> <template> [attrs] (src/cli/Scaffold.vala) generates MANIFEST.usm + usm-scripts/ for templates basic/makefile/meson with attributes vala/c/acquire (the last adds a wget/tar acquire script + deps). Scaffolded manifests express deps as resource refs (bin:make, pc:glib-2.0.pc, …), the same vocabulary the LFS scaffolds will use — but LFS scaffolds will be generated, not hand-scaffolded, and their scripts are bespoke per package.

4. The six brief items — analysis

4.1 Item 1: a repository of USM scaffolds containing all packages needed to bootstrap LFS

Mapping. One git repository ("the scaffold repo") holding one directory per LFS package, each a USM scaffold: MANIFEST.usm (name/version/licences/ provides/depends/execs) + usm-scripts/{acquire.sh,build.sh,install.sh} + vendored patches. usm manifest package turns each into a .usmc; the collection is published with usm repository init/add/build-list. Nothing here needs new USM features — Scaffold.vala already writes this shape; the generator will write it directly (scaffold templates don't fit LFS's per-package configure flags anyway).

What's missing / must be built — the generator. The book is DocBook XML; the machine-readable part is excellent (packages.ent: 718 entities — names, versions, URLs, md5s, SBU/disk estimates as text), but the ~640 build steps are prose. Counts from this checkout:

• userinput remap="…" command blocks across ch5–8: 563 — 105 configure, 120 make, 180 install, 76 pre, 76 test, 3 adjust, plus 3 one-offs (clean, locale-test, locale-full); plus 83 un-remapped <userinput> (chapter setup commands) → 646 <userinput> elements in total across ch5–8.
• Package pages (procedural pages like kernfs/chroot/createfiles excluded): ch5 5, ch6 17, ch7 8, ch8 85 → 115 pages, 91 distinct packages (some are multi-pass: gcc appears as ch5 pass1/pass2 plus libstdc++ and ch8; binutils as ch5/ch6 pass1/pass2 and ch8). Chapter 3 lists 93 downloadable URLs (~85 of them source tarballs with md5s, the rest docs/bootscripts/udev-lfs/kernel); the ~90 package-pass scaffolds in the executive summary exclude docs/extras and count multi-pass packages once.

A naive <userinput>-to-shell-literal extraction produces almost right scripts — but not right: entities need expansion, make -j1 / make -k check conventions need policy, ch5/6 scripts must hard-code /tools prefixes and env (LFS, LFS_TGT, PATH=/tools/bin:...), ch8 scripts assume the chroot, and a few packages need hand-massaging (gcc's three passes plus its ulimit -s -H unlimited test preamble, glibc's long multi-line configure block, systemd's meson -D* policy, perl's and python's non-./configure build commands). So: a generator (Python, ~1.5-2k lines) with a curated overrides table for the ~15 hairy packages, emitting both the scaffolds and a stage-manifest.json the orchestrator consumes. Generated scaffolds are checked into git, reviewed once, then regenerated only on book bumps.

Effort: generator 2-3 engineer-weeks (including the overrides table and the ch5/6/ch7/ch8 script envs), scaffold review pass 2-4 days.

4.2 Item 2: build a container into this bootstrap environment

Mapping. The bootstrap container is the build host for stages 1-4: a root filesystem with a chapter-2-compliant toolchain, the dnf-installed gaps closed, usm + its runtime/build closure (curl, git, libsodium, glib, libgee, json-glib, vala, gobject-introspection, and the two invercargill libraries — all via dnf on the Fedora base; see §5 Stage 5) preinstalled, and the LFS sources either pre-fetched or fetchable. Two ways to produce it:

• (a) dnf-based (recommended for stage 0). Start from a Fedora container base (or dnf --installroot into a directory on this host — no runtime needed), dnf install the ch2 checklist + gcc-c++ patch texinfo + usm's runtime closure, then tar it up. Fully reproducible-ish, fast.
• (b) host-clone (proven here). Clone the sandbox host root, add the missing pieces via dnf, prune to the ch2 list. Faster to stand up (the bison proof used exactly this), less hermetic; fine for development, not for the published artifact.

What's missing / must be built. An OCI image assembler script: tar the rootfs into one (or a few) layers, write oci-layout, index.json, blobs/sha256/… config+manifest. ~200-300 lines of Python/shell; standard format, no runtime required to produce. (On podman hosts, usm manifest deploy --no-build's context or a plain Containerfile is the alternative production path.)

Effort: image build recipe + assembler 3-5 days, including a verify-by-chroot harness (unpack the image, chroot in, run the ch2 version-check script from the book).

4.3 Item 3: a USM repository of the LFS packages

Mapping. Exactly usm repository init lfs && usm repository add *.usmc && usm repository build-list (src/cli/Repository.vala): per-package .usmc from usm manifest package, a PACKAGES.usml signed listing (ed25519 keys under keys/, same mechanics as DCBA init), served as a file:// (or later https) repository. Consumers usm add-repo it. The only structural question is granularity of stages: because the resolver is single-pass (§3.3), the repo should be one repository with stage tags — e.g. manifests carry a stage extra-property (or separate repos lfs-tmp0, lfs-tmp1, lfs-final; tags are lighter). The orchestrator installs stage by stage; each stage's install is one ordinary usm install <pkg>... transaction whose dependencies resolve against what earlier stages already left in the root (local presence, §3.2).

What's missing / must be built. The stage tags (generator-emitted); a provides vocabulary decision for LFS (e.g. lib:libc.so.6 from glibc, bin:gcc from gcc, pc:… where the book installs .pc files); discipline that ch5/6 temporary packages self-contain under /tools and do not provides-collide with ch8 finals (they must not — same package name, different version suffix, e.g. gcc-tmp-16.2.0 vs gcc-16.2.0).

Effort: 1 week, mostly the provides/depends modelling and validating resolution with the real resolver (the itemised UNSATISFIABLE reports make this pleasantly debuggable).

4.4 Item 4: a DCBA repository of the LFS packages

Mapping. Direct use of the DCBA flow (§3.5): after a build stage, each package cache's build.tar.xz becomes <name>-<version>.dcba (cp, exactly as scripts/dcba-e2e.sh §3 does), usm dcba repository add rebuilds the signed ARTIFACTS.usml, and consumers with the .dcbar configured install with pre-built artifacts ("Retrieved pre-built artifact") — turning later stages (and any third party) from compiles into cache-restores. This is precisely the artifact-distribution model the brief asks for, and it is what makes Stage 4 (final chroot build) cheap enough to iterate on.

What's missing / must be built. A small publish driver (loop over the state dir, copy, add, verify — dcba repository verify exists). Real considerations: (i) the artifact is only valid for one stage's environment — a ch5 build.tar.xz is useless in ch8 — so artifacts must be namespaced (separate DCBA repos per stage, or name-embedded stage suffixes mirroring the -tmp package names); (ii) signing keys per repo with keys/ 0600 and the public key distributed in the .dcbar; (iii) .dcba files are multi-hundred-MB for gcc — disk budgeting in the publish step (11 TB free here; a full set is ~15-25 GB xz).

Effort: 2-4 days including the verify-by-fresh-root test (reuse the dcba-e2e.sh pattern).

4.5 Item 5: build a chroot using USM from those repos

Mapping. Stage 4 in the architecture below: enter the stage-0 container root (chroot on this sandbox; a container anywhere else), configure USM's USM_CONFIGDIR/state inside it, usm add-repo the file:// USM repo and usm dcba add-repo the DCBA repo(s), then run the orchestrator's stage installs — each usm install <stage-packages> resolves against the root's current contents, restores .dcba caches where available, builds where not, and stages outputs into the root. The result after the final stage is an LFS root built by USM transactions — the actual "USM distro" claim.

Constraints found. (i) The chroot needs /dev/null (plain file is enough for most builds — proven) and ideally /dev/zero, /dev/random, /dev/tty; without CAP_MKNOD they stay plain files, so anything doing ioctl on them (systemd's build probes, some test suites) is at risk. (ii) /proc empty breaks packages that read /proc/mounts or /proc/self/… at build time; systemd again, plus procps tests. (iii) The ResourceFinder host-absolute locators (§3.2) are correct inside a chroot (destination /), so run usm inside the chroot for the build stages — which is also the honest end-to-end test of self-hosting. (iv) USM never creates the state root itself (dcba-e2e.sh header): the orchestrator pre-creates <state>/{lists,packages,installed} per root.

What's missing / must be built. The stage orchestrator (~300-500 lines): create root, populate /dev workaround files, run per-stage installs with the right env, checkpoint between stages (tar snapshot for fast iteration), and the sysvinit-vs-systemd switch (recommend building sysvinit revision first — no /proc dependency at build time — and systemd as a follow-up on a full host).

Effort: 1-1.5 weeks including debugging the first full run.

4.6 Item 6: a container based on that chroot with USM installed

Mapping. The final root already is a USM-managed root (stages 4-5 installed everything through USM transactions, and USM itself is installed by stage 5). Producing the container = the same OCI assembly as item 2 (§4.2): tar the final root as one layer, write config (cmd /bin/bash, env PATH=/usr/bin:/bin, labels recording the repo commits + listing digests), manifest, oci-layout. On a podman host, usm manifest deploy of a meta-package (or just podman import) is the one-liner alternative.

What's missing / must be built. Nothing structural. A "distro identity" decision (name, /etc/os-release — trivially a scaffold package), and the same assembler script with a different config blob. Verification: podman run on a capable host, or chroot-in-the-unpacked-image on this sandbox (bash + usm --version + usm list).

Effort: 2-3 days, including the smoke-test matrix.

5. The staged architecture

One diagram, then per-stage detail. H is the build host (this sandbox or a full machine); every stage's output feeds the next.

Stage 0            Stage 1        Stage 2/3                 Stage 4              Stage 5        Stage 6
bootstrap image ← scaffolds  →   USM repo + DCBA repo  →   final LFS root   →   +beyond-LFS    → OCI image
(dnf + ch2 tool-   (generated     (built INSIDE a stage-0    (built by staged     + USM itself    (assembled by
 chain + usm)       from book)     chroot, staged installs)  usm transactions)   (self-host)     script)

Stage 0 — bootstrap container/image

Purpose: a hermetic chapter-2-compliant build host. Recipe (dnf-based):

1. dnf --installroot=$ROOT --releasever=44 install @build-tools or an explicit list from the book's ch2 table, plus gcc-c++, patch and texinfo (the ch2 gaps found on this sandbox — see §2.3; bison provides yacc only via a separate symlink the image recipe must create), plus usm's runtime closure: curl git libsodium glib2 json-glib libgee vala gobject-introspection (Fedora names).
2. Install usm itself (any build that runs on the host glibc — the Fedora dnf set is fine; this stage-0 copy never needs to be the self-hosted one).
3. Fetch/verify the ~93 LFS downloads listed in ch3 (URLs+md5 from packages.ent; ~85 are source tarballs, the rest docs/extras), or vendor them into the image; pre-seed the file:// repo layout.
4. Assemble the OCI image (script) — or just keep it as a directory tree for chroot use on this sandbox.

Outputs: $ROOT (chroot-able), stage0.img (OCI layout).

Stage 1 — the scaffold repository

The generator (§4.1) reads packages.ent + chapter XML from the LFS checkout and emits one scaffold dir per package-pass:

• MANIFEST.usm — name (with -tmp suffixes for ch5/6 passes), version, licence (from the book's licence notes), depends (manage: bin:bash, bin:coreutils…; build: the tools the book's commands call; runtime: the libs they install), provides (bin:gcc, lib:libc.so.6, …), execs (acquire/build/install/test), extra-property stage ∈ {tmp0, tmp1, chroot, final}.
• usm-scripts/build.sh — the book's configure/make commands with PATH/LFS_TGT env set by the stage driver, make -j$(nproc).
• usm-scripts/install.sh — the book's install commands; DESTDIR-style staging is NOT what LFS does (it installs into ///tools inside the chroot). USM runs install scripts with a per-package install dir as DESTDIR (Transaction.vala passes create_install_directory() into run_install, which overrides destination — Manifest.vala:441-455); LFS scaffolds must therefore either install into that DESTDIR and let the stage driver move its contents into place, or use the book's direct paths (///tools) from inside the chroot. Per-scaffold choice, recorded in the generator's overrides.
• vendored patches from the book.

Output: the scaffold git repo (reviewed, committed) → *.usmc via usm manifest package → the USM repo (§4.3).

Stage 2/3 — build the USM + DCBA repositories inside a stage-0 chroot

Why inside: the packages must be built by the stage-0 toolchain in the stage-0 environment, and USM must be able to see its own staged outputs (local presence) — which is exactly "run usm inside the chroot". Driver:

1. chroot into stage-0's $ROOT (plain-file /dev/null workaround; empty /proc acknowledged).
2. usm add-repo the stage-1 USM repo; for each stage s ∈ {tmp0, tmp1, chroot, final}: usm install <stage-s packages> — each install resolves against the root as-left-by-stage-(s-1).
3. After each stage: publish .dcba artifacts (copy build.tar.xz → <name>-<version>.dcba, usm dcba repository add) into per-stage DCBA repos.

Outputs: PACKAGES.usml-signed USM repo; per-stage ARTIFACTS.usml-signed DCBA repos.

Stage 4 — USM-driven chroot build of the final root

Fresh root $LFS; populate /dev workaround files + empty /proc; chroot; configure a new USM state inside it; add the stage-1 USM repo + the per-stage DCBA repos; run the same stage sequence. Because the DCBA caches restore, this stage is CPU-cheap (cache restores + final-system gcc/glibc builds are the only heavy items). End state: a complete LFS system whose every file was placed by a USM transaction.

Stage 5 — beyond-LFS self-hosting set + USM itself

The book does not contain USM's build deps. The gap set, re-derived this session from USM's own MANIFEST.usm (which is the authoritative dependency list) plus its direct library deps: the book HAS pcre2 (chapter08/pcre.xml), libffi, python, meson, ninja, pkgconf, zlib, xz, util-linux (libmount/libblkid), elfutils/libelf. libselinux is NOT in the book, yet USM's manifest names lib:libselinux.so.1 as a runtime ref (it arrives via glib's linkage on a typical distro). Beyond-LFS set, from the manifests:

• glib (glib/gobject/gio/gmodule; pulls libffi+pcre2+util-linux from the book, plus libselinux which must be added beyond-LFS — or glib rebuilt -Dselinux=disabled, in which case USM's lib:libselinux.so.1 runtime ref needs relaxing),
• libgee, json-glib, valac,
• invercargill and invercargill-json (USM's own base libraries — lib:libinvercargill-1.so, pc:invercargill-1.pc, pc:invercargill-json.pc are hard runtime/build refs of USM; both are vala/meson packages needing only glib+gobject, so they slot after json-glib),
• curl (http(s) repositories at runtime; usm dcba add-repo for remote repos too) (+ optional git),
• libsodium (USM's ed25519 signing links libsodium; build dep via vapi, runtime lib:libsodium.so.26),
• gobject-introspection — USM's build deps include pc:gobject-introspection-1.0.pc (optional in principle but currently required by the manifest); it needs glib and Python, both present by this point in stage 5.

So the honest count is ~10 beyond-LFS packages (glib, libselinux or a glib flag change, libgee, json-glib, gobject-introspection, valac, invercargill, invercargill-json, libsodium, curl), not the ~6 the plan's evidence suggested. Order: glib (+libselinux) → libgee → json-glib → gobject-introspection → valac → invercargill → invercargill-json → libsodium → curl → USM. Then usm install usm (from a .usmc of USM's own repo — USM's MANIFEST.usm exists at the repo root) is the self-hosting milestone: the distro's package manager installs itself.

Stage 6 — OCI image assembly

Scripted (no runtime here): tar $LFS (+ stage-5 additions) as one gzip/xz layer, emit oci-layout, index.json, config JSON with cmd ["/bin/bash"], env, and labels carrying the repo commit + listing digests. Optionally split /usr vs /etc/var into two layers for cache-friendlier future updates. On podman hosts: podman load/run for verification, or skip assembly entirely and use usm manifest deploy.

6. Why staged (the resolver argument, precisely)

Resolver.resolve (src/lib/Resolver.vala:233) computes ONE resolved set with ONE topological order over manage+build+runtime refs (src/lib/Resolver.vala:422-428). LFS needs three orders interleaved: binutils-p1 → gcc-p1 → (ch5 libs) → gcc-p2 must complete before ch7's chroot tools build against them, and ch8's final glibc/gcc must replace the temporary ones — i.e. the same resource (lib:libc.so.6, bin:gcc) is provided by different packages at different times, with deliberate replacement semantics. USM's resolver treats one name-version per package and picks providers once; replacement-across-stages is a different problem (it has usm downgrade/transaction machinery, but not "same provides, different package, by stage"). Hence: stage boundaries are orchestrator-level, stage interiors are USM resolutions. This is not a USM defect — it is the honest boundary between a package manager and a distro builder, and it keeps every stage debuggable with stock usm commands and stock resolver error reports.

7. Effort estimates

7.1 Wall-clock (build time)

SBU data is machine-readable in packages.ent (entities named <pkg>-<tmp|tmpp1|tmpp2|fin>-sbu). Summed this session (separate test-suite entities excluded, "less than X" counted as X/2, e2fsprogs' spinning-disk figure taken):

• ch5/6 temporary tools (-tmp*-sbu entities: 33 of them, counting binutils/gcc pass1+pass2): ~21 SBU (gcc-p2 5.2, gcc-p1 4.3, tcl 2.9, gettext 1.5, glibc-tmp 1.3, …).
• ch8 final system (-fin-sbu, 82 entries): ~100 SBU (gcc 53, glibc 11, vim 3.3, python 2.7, e2fsprogs 2.4 on spinning disk / 0.4 on SSD, …). The gcc entity is explicitly "53 SBU (with tests)" — the book publishes no tests-free gcc figure, so the sum double-counts gcc's tests as if they were build time; treat ~100 as an upper bound for build-only; the no-tests estimate below keeps that headroom.
• Linux kernel ("0.4 - 32 SBU, typically about 2.5 SBU") is not part of the bootstrap (no boot requirement for a container/chroot distro); adding it costs ~2.5 SBU typical / up to ~32 SBU worst-case.

One SBU = one core's binutils-p1 time, typically ~1-2 min on modern hardware (the book: gcc ~5 min "on the fastest systems"). With ~120 SBU total and make -j28, the parallelisable make phases compress roughly 10-20x, but configure phases, install phases and single-threaded steps (glibc's make install, gcc's early stages) do not. Honest estimate for one full stage-0→4 pass on this 28-core sandbox:

• without test suites: ~3-5 hours (gcc dominates; discounting a third to a half of its 53 SBU as test time still leaves ≈ 1-2 h alone at j28 with serial configure/install — inside this range).
• with the book's recommended test suites: ~8-14 hours (tests are mostly serial and several are multi-SBU).
• Stage 5 (beyond-LFS, ~10 packages + USM): ~45-75 min (glib ~0.5 SBU, valac ~2 SBU, gobject-introspection ~1-2 SBU, curl ~1 SBU, the two invercargills and USM itself well under 1 SBU each at j28).
• A DCBA-cached re-run of stage 4 (iteration loop): ~20-40 min — this is the number that matters for daily development.

Disk per full run: sources ~1 GB, build trees ~40-80 GB, artifacts (build.tar.xz set) ~15-25 GB, final root ~4-8 GB. Budget ~200 GB per working copy; 11 TB free comfortably supports several parallel runs, and zero runs if any of it lands on / (17 GB — see §2.2).

7.2 Engineering time

Per-item figures from §4; consolidated (one engineer, familiar with both USM and LFS):

• Stage 0 recipe + OCI assembler + verify harness: 3-5 days
• Generator (book → scaffolds, with overrides table): 2-3 weeks
• USM repo modelling (stages, provides vocab, resolution validation): 1 week
• DCBA publish driver + per-stage repos: 2-4 days
• Stage orchestrator (chroot mgmt, /dev workarounds, checkpoints): 1-1.5 weeks
• Beyond-LFS set (~10 packages, valac + gobject-introspection ordering) + USM self-host: 4-6 days
• OCI final image + smoke tests + documentation: 2-3 days
• Integration/debug of the first full run + book-drift tooling (regeneration diff): 1 week

Total: ~7-10 engineer-weeks (summing the lines above at 5-day weeks: 36-50 working days; the spread is almost entirely the generator: ~90 packages × curated overrides vs mostly-mechanical). Uncertainty: a first full run will surface 5-15 per-package script bugs (extraction artefacts, env differences) — that's inside the integration line above; a systemd-revision build adds ~3-5 days if pursued (and needs a full host for /proc reasons, §8).

8. Risk register

• Device nodes in /dev (no CAP_MKNOD). Plain-file /dev/null works for most builds (proven with bison); /dev/zero, /dev/random, /dev/tty as plain files break anything doing ioctls. Mitigations: stage-0 image carries pre-created device nodes (built where mknod is allowed, e.g. the human's machine or a rootful builder — the image is just a tarball, so nodes survive into it; only creating them here is impossible); chroots unpacking that image inherit working /dev.
• Empty /proc and /sys (no CAP_SYS_ADMIN to mount). Most packages fine; systemd's build/configure and readers of /proc/mounts at risk. Mitigations: sysvinit revision first (no /proc at build time); systemd build on a full host; or bind-mount-free stub files where the book's commands tolerate them.
• systemd vs sysvinit. Both revisions are in the book (261.3 vs 3.14). Recommend shipping sysvinit first, systemd as a follow-up goal.
• ResourceFinder host-absolute locators (ldconfig -p, /usr/include, /usr/libexec, gir/man/info/locale — §3.2). Outside a chroot, staged roots can be "satisfied" by host files. Mitigation: run build-stage usm inside the chroot (destination /, so absolute == correct); for outside-chroot tooling, audit or later patch ResourceFinder with a strict-destination mode (candidate small USM improvement, out of scope here).
• Acquire phase needs curl/git/wget — but the bootstrap's acquire scripts can instead use vendored sources already inside the .usmc or the stage-0 image, removing the network dependency from stages 2-5 entirely (recommended; the book's md5s gate integrity).
• Book drift. LFS 13.1 → 13.2+ bumps package versions and commands. Mitigation: the generator is the only thing that reads the book; regeneration diff (git diff of scaffolds) makes drift reviewable.
• valac bootstrap ordering. USM needs valac + glib + libgee + json-glib + invercargill ×2 (+ libsodium, gobject-introspection at build time). valac's own build (per its LFS-blfs recipe and the dnf package) needs glib (libgee is a common consumer, not a valac prerequisite). Fixed linear order in stage 5 (glib → libgee → json-glib → gobject-introspection → valac → invercargill → invercargill-json → libsodium → curl → USM); no cycle exists because valac's compilation can use a stage-0-hosted valac (dnf-installed) — the dnf set already includes one, so USM's first self-host can even avoid valac-from-source if desired.
• ed25519 key management. keys/ 0600, private keys never leave the producer; .dcbar/.usmr carry the public key. Risk: losing the key orphanes a repo — keep one offline master per distro release line, and document rotation.
• Disk. / here is 17 GB — nothing large may touch /tmp; all scratch under /workspace (11 TB). Checkpoint tars double usage transiently; clean between stages.
• gcc/binutils newer than the book are explicitly unsupported by LFS (ch2 warning). This host's gcc 16.2.1 sits on the book's 16.2.0 line and its binutils 2.46.1 is older than the book's 2.47 — fine; any host bump past the book needs a book bump first. Relatedly, the sandbox host lacks g++/patch/makeinfo (ch2 requirements) — with root and dnf5 this is a non-event: stage 0 installs them, and a ch2-check script fails fast on any host still missing items.

9. Recommended task breakdown (for the implementation goal)

Ordered; each is independently verifiable and lands value on its own:

1. stage0 recipe + OCI assembler + ch2 check (3-5 d) — produce a bootstrap image from dnf; verify by chroot + book ch2 script.
2. Generator v1: packages.ent → scaffold skeletons (3-5 d) — manifest fields, URLs/md5s, acquire scripts; build.sh placeholders; no commands yet. Deliverable: ~90 scaffolds packaging cleanly (usm manifest package all green).
3. Generator v2: chapter XML → build/install scripts + overrides table (1.5-2 wk) — ch5/6 (tmp passes, /tools env), ch7, ch8; curated handling for gcc/binutils/glibc/systemd|sysvinit/perl/python.
4. USM repo modelling + stage tags + resolution dry-runs (1 wk) — resolver-validated dependency graph per stage, zero UNSATISFIABLE.
5. Stage orchestrator + stage-2/3 build of tmp/chroot stages (1 wk) — first real chroot builds inside stage-0; publish tmp/chroot .dcba repos.
6. Stage 4: final-system build via USM (3-5 d) — full ch8 through usm transactions with DCBA caches.
7. Stage 5: beyond-LFS set (~10 packages) + USM self-host (4-6 d) — the milestone: usm install usm inside the final root.
8. Stage 6: final OCI image + smoke tests (2-3 d) — assembled image, chroot/podman verification matrix, docs.
9. (Follow-up, separate goal) systemd revision, kernel build, https repo hosting, ResourceFinder strict-destination mode.

10. Conclusion

The goal is feasible with current USM master: the staged architecture uses only documented, e2e-tested mechanics (staged installs, file:// repos, DCBA artifact distribution), the LFS book is machine-minable enough for a generator with a modest overrides table, and the two hard environmental constraints of this sandbox (no device-node creation, no mounts) are worked around for every stage except a systemd build. The recommended first implementation milestone is tasks 1+2: a verified bootstrap image and ~90 packaging-clean scaffolds — roughly a week of work that de-risks everything downstream.