# Universal Source Manifest ## Manifest format (MANIFEST.usm) ```json { "name": "my-package", "version": "1.0.5+2", "summary": "A sample package", "licences": [ {"name": "GPLv3", "category": "libre", "text": "src/LICENSE"} ], "provides": { "bin:hello-world": "hello-world", "app:hello-world.desktop": "hello-world.desktop", "res:usr/share/icons/hicolor/scalable/hello-world.svg": "icons/hello-world.svg" }, "depends": { "runtime": [ "lib:libgtk-4.so.1", ], "build": [ "bin:valac", "bin:meson", "inc:gtk-4.0" ], "manage": [ "bin:bash" ] }, "flags": [ "simpleBuildEnvironment" ], "execs": { "install": "usm-scripts/install", "remove": "usm-scripts/remove", "build": "usm-scripts/build", "rebuild": "usm-scripts/rebuild", "test": "usm-scripts/test" } } ``` ### Optional extended properties ```json { "md": "src/DESCRIPTION.md", "url": "https://my.package.com", "screenshots": [ "hello-world-screenshot.png" ], "icon": "hello-world.svg", "git": { "origin": "https://git.my.package.com/hello-world", "commit": "8d9f3b198e3a767d41f3c33fac2fbd8496f23ed2" }, "metainfo": "hello-world.appdata.xml", "extras": { "customData": "whatever" } } ``` ## Manifest flags USM supports several flags that modify the build and installation behavior: - `"buildInSourceTree"`: Tell USM to not create a separate build directory, and compile the package "in place". - `"setManifestPropertyEnvs"`: Sets environment variables based on manifest properties (TODO: document specifics). - `"simpleBuildEnvironment"`: Copy the full source tree to the build directory before running the build exec, and execute all build-related scripts (build, install, rebuild, test, postInstall) from the build directory instead of the source directory. This provides a clean, isolated build environment. - `"dataPackage"`: Marks the package as a data package; see [Data packages](#data-packages) below. ## Resource types - "rootpath": Generic resource: file on root filesystem without leading "/" - "path": Generic resource: file name locatable /usr directory - "opt": Optional software: file name locatable /opt directory - "res": Generic resource: resource under /usr/share directory - "cfg": Configuration file: file name locatable /etc directory - "bin": Binary executable: file name locatable in `$PATH` - "sbin": Binary executable: file name locatable in /usr/sbin or /sbin - "lib": Shared library: file name locatable by LD - "libexec": Binary executable: file name locatable in /usr/libexec - "libres": Generic resource: file name locatable /usr/lib, /usr/lib64, /lib, or /lib64 directories - "info": Information page: file name locatable /usr/share/info - "man": Manual page: file name locatable /usr/share/man - "locale": Locale: file name locatable /usr/share/locale - "app": Desktop entry: file name locatable in /usr/share/applications - "inc": Include: file name in /usr/include - "pc": Package config: file name locatable in `$PKG_CONFIG_PATH` - "vapi": Vala API: file name in / - "gir": GObject introspection: file name locatable in /usr/share/gir - "typelib": GObject typelib: file name locatable in /usr/lib64/girepository-1.0, /usr/lib/girepository-1.0, /lib64/girepository-1.0, or /lib/girepository-1.0 - "gio": GIO plugin module: module filename locatable in /usr/lib64/gio/modules, /usr/lib/gio/modules or `/usr/lib//gio/modules` - "tag": A USM system tag, dot seperated with ".tag" appended, i.e. "hello.world" would be locatable in /usr/share/usm-tags/hello/world.tag ## Source Repositories (Repo.usmr) ```json { "name": "my-repo", "summary": "My software repository", "uris": [ "https://my.software/repository", ], "key": "EDXLsUvOZEne+xcv+huvSaqNBs8TTldCv6hd69GdmYw=" } ``` ## Source Repository listing (PACKAGES.usml) Entries in JSON lines format: ```json { "type": "usmc", "manifest": {}, // Manifest object from MANIFEST.usm "path": "my-package.usmc", // Relative path to the package "sha512": "0+6QpQ5Wst2bPmAPYKO/RKrFYuq93rcaRs7tqt9lCo9zu8v6jpJsDqd7yPIqCWDt0KpOQ3uLK64Tmn9FYNw3eQ==" } ``` Last line consists of the signatures for the repository ```json { "type": "signatures", "signatures": [ { "key": "EDXLsUvOZEne+xcv+huvSaqNBs8TTldCv6hd69GdmYw=", "signature": "0rHQiDGKgumMK2S3B6D9jjU3NLnLLjijYJRvaB5vLbp4zxzjlixfyX8yfvvJUCCd34CC1STzyzawDtb9ZammCTE4zFHFdpnO8eseDS5OY+aKgAyL13oBGizvy/V0zGHVCdv83GQ7QUtgSWje0RKKk1zPGDAIF/prkurlFDgd3Vg=" } ] } ``` ## Complete package (package.usmc) Complete package is simply a tar.xz file containing a MANIFEST.usm file and all the requisite files that would be acquired if the acquire script was run. That is to say, a USMC file could be created by simply running `usm manifest acquire` followed by `tar -cJf ../package.usmc .`. ## .usmignore `usm manifest package` consults an optional `.usmignore` file at the root of the package (next to `MANIFEST.usm`) to prune files and directories from the produced archive. Semantics: - Blank lines and lines starting with `#` are skipped. - A pattern containing `/` matches the FULL path relative to the package root; a pattern without `/` matches any path suffix (equivalently, the basename at any depth). - A trailing `/` marks a directory-only pattern: it matches the directory itself and everything beneath it. - `*` and `?` are wildcards and never match `/`. - There is no negation: `!` is a literal pattern character and cannot re-include an ignored path. - The root `.usmignore` and `MANIFEST.usm` files are always packaged; they can never be ignored. - The `.git` directory is always ignored. When no `.usmignore` file exists, `.git` is the only default ignore. ``` # .usmignore # Build trees at any depth (directory-only: the directory and all beneath it) builddir/ # Database artifacts by basename at any depth *.sqlite # A specific generated tree, anchored to the package root docs/generated/ # Wildcards; '*' and '?' never cross a path separator src/secret.* notes/?draft.md ``` ## Data packages A manifest with the `"dataPackage"` flag ships static files: fonts, icons, shared data sets and the like. Data packages define no build or install machinery: - Validation rejects a `dataPackage` manifest that defines any lifecycle executable (`build`, `install`, `rebuild`, `test`, `remove`, `postInstall`); only `acquire` is permitted. - `usm manifest build` is a no-op: no build directory and no build artifacts are created. - `usm manifest install` and `usm manifest validate` copy (or check) each `provides` entry directly from the source tree — path bases are treated as `source:` regardless of how they are written — into the destination's suggested path. Staging a data install works the same as any other package, e.g. `USM_DESTDIR=/tmp/stage usm manifest install`. - `usm manifest acquire` and resource bookkeeping on install/remove are unchanged. ```json { "name": "example-fonts", "version": "1.0.0", "summary": "The Example font family", "licences": [], "flags": ["dataPackage"], "provides": { "res:fonts/Example-Regular.ttf": "source:fonts/Example-Regular.ttf", "res:fonts/Example-Bold.ttf": "source:fonts/Example-Bold.ttf" }, "depends": { "runtime": [], "build": [], "manage": ["bin:bash"] } } ``` ## USM Directory /var/usm - /packages - /packagename-0.1.1 - /origin-info - /package.usmc - /build-log (when built) - /build.tar.xz (when built) - /build (when building) - /installed - /packagename-0.1.1 (symlink to /var/usm/packages/packagename-0.1.1) - /lists - /repositoryname - /2020-03-20T14:28:23.382748.usml - /2020-03-20T14:34:42.382748.usml - /transactions - 2020-03-21T14:28:23.382748 - 2020-03-23T14:23:43.382748 ### Origin info format ```json { "repository": "repositoryname", "listfile": "2020-03-20T14:34:42.382748.usml", "original_path": "packagename-0.1.1.usmc", "signature_verified": true } ``` ## Resource Ref Additions - Add version expressions (can copy from MPK), i.e. == >= <= <> - Add support for version expressions to *some* resource types, namely pkg and pc Below here are draft ideas that may or may not get implemented ## Transactions - Create backup BTRFS snapshot - Create transaction BTRFS snapshot from backup snapshot - Apply difference between backup and transaction snapshots to live filesystem - Delete transaction snapshot JSONL transaction journal format ```json { "operation": "begin-transaction", "timestamp": "2020-03-20T14:34:42.382748", "snapshot": "path/to/backup/snapshot", "subvolume": "path/to/transaction/subvolume" } ``` ```json { "operation": "begin-transaction", "timestamp": "2020-03-20T14:34:42.382748", } ``` ## Scriptable installation - USM will not manage the installation if an install exec is present - Compare backup and transaction snapshots for validation - Scripts can optionally handle installation but it must be verified that the expected files (and only the expected files) are installed - Add post-install and maybe post-transaction execs that are not subject to validations ## USM Commands ### Manifest Commands - `usm manifest build `: Build the package - `usm manifest install `: Build and install the package to the system - `usm manifest remove`: Remove the package from the system - `usm manifest acquire`: Run the acquire script to download sources - `usm manifest package`: Create a package archive - `usm manifest deploy [--exec CMD] [--base IMAGE] [--repository FILE]... [--no-build] [--installer-url URL]`: Build a container image from the manifest directory; see [Container deployment](#container-deployment) - `usm manifest autoprovides [--replace] [--debug] [build path]`: Scan installation and generate provides section - `usm manifest test [build path]`: Run package tests - `usm manifest validate [build path]`: Validate package by building and checking resources ### Repository Commands - `usm repository init `: Create the canonical repository layout in the current directory: `.usmr`, `keys/` (signing keys, private key mode 0600 and gitignored via `.gitignore`), and an empty `public/` packages directory - `usm repository add `: Validate the package (manifest + checksum), copy it into `public/`, then rebuild and re-sign `PACKAGES.usml`; a package with the same name and version is rejected unless identical - `usm repository remove `: Delete the matching `.usmc` from `public/` (exact filename, or `-` prefix; an ambiguous name lists the candidates and asks for the filename), then rebuild and re-sign - `usm repository list [repo.usmr]`: Print name, version and summary of every package in the signed listing (from the repository root, or fetched from the given `.usmr`'s URI) - `usm repository verify `: Check the listing signature, then deep-verify every entry's sha512 against the `.usmc` fetched from the repository URI; exits non-zero on any failure - `usm repository publish `: Copy the `.usmr` and `PACKAGES.usml` to `` (always overwriting) and each `public/*.usmc` only when the destination copy is missing or older by mtime - `usm repository new ` / `usm repository build-list`: Legacy commands using loose `public-key`/`private-key` files in the working directory All commands except `init`, `new` and `verify` locate the repository root by searching the current directory and its ancestors for a directory containing a `*.usmr` file, a `keys/` directory and a `public/` directory. Signing keys load from `keys/public-key` and `keys/private-key` in the repository root, falling back to the legacy loose `public-key` and `private-key` files. ### Validation Command The `validate` command performs comprehensive validation of USM packages: ```bash usm manifest validate [build path] ``` The validation process includes: - Manifest syntax validation - Dependency checking - License verification (with warning if missing) - Build execution with progress reporting - Install execution to temporary directory - Resource validation based on path base type (as-expected, build:, source:, install:) - Unexpected resource detection Exit codes indicate specific validation failures, making it suitable for CI/CD environments. For detailed information, see `slopdocs/utility.usm.manifest.validate.md`. ## System package manager integration USM can delegate missing resources to the system package manager (DNF and friends) before falling back to USM repositories. The integration is configured in `usm.config` and talks to a helper executable through two stable contracts; five reference helpers ship under `spm/` (installed to `bindir` as `usm-spm-`): | Helper | Distro(s) | Language | Container bootstrap | Limitations | |---|---|---|---|---| | `spm/dnf/usm-spm-dnf` | Fedora | Python 3 + dnf4 | `dnf install -y python3-dnf && dnf clean all` | — | | `spm/apt/usm-spm-apt` | Debian, Ubuntu | Python 3 + python3-apt | `apt-get update && apt-get install -y curl python3 python3-apt apt-file && apt-file update && apt-get clean` | file queries go through **apt-file only** (no naming heuristics), so the contents index downloaded by `apt-file update` (~100 MB) is mandatory; paths follow Debian multiarch (`/usr/lib/x86_64-linux-gnu`, both pkgconfig dirs) | | `spm/apk/usm-spm-apk` | Alpine | POSIX shell (busybox ash) | `apk add --no-cache bash curl` | dependency-free — no python3, no jq; queries ride apk's native `pc:`/`so:` provides and the `/lib` (not lib64) busybox layout | | `spm/emerge/usm-spm-emerge` | Gentoo | Python 3 + portage | none — the stage3 base already carries portage + python | offline ebuilds expose no file lists: file queries for not-yet-installed packages resolve only via installed files, virtuals and a small package-name table, otherwise land in `not-found`; dependency counts come from `emerge --pretend` (see the `usm-spm-emerge` docstring for the full limits) | | `spm/pacman/usm-spm-pacman` | Arch | POSIX shell (bash) | `pacman -Sy --noconfirm curl && pacman -Fy --noconfirm && rm -rf /var/cache/pacman/pkg/*` | dependency-free like the apk helper: bsdtar (a pacman dependency) parses the sync databases for soname provides and the **files databases** for file queries, so the `pacman -Fy` refresh (~40 MB) is mandatory — without it only installed files resolve (`pacman -Qo`), no naming heuristics; dependency counts come from a `--print` run against a throwaway `--dbpath`; merged-usr `/usr/lib` layout | All five implement the identical JSON/JSONL contract below; `usm deploy --spm` (see [Container deployment](#container-deployment)) wires the matching one into generated images. ### Configuration ```json "system_package_manager": { "query": ["usm-spm-dnf","query"], "install": ["usm-spm-dnf","install"] } ``` Both keys or neither (validation error otherwise); argv arrays, no shell. ### Query contract ` …` → STDOUT single JSON object, no side effects: ```json { "not-found": ["pc:foo.pc"], "packages": [ { "name": "vala", "resources": ["bin:valac","vapi:gtk-4.0.vapi"], "dependency-count": 5, "installed-dependency-count": 3 } ] } ``` `dependency-count` = packages the manager would install for this package alone incl. itself. Exit 0 with non-empty `not-found`; non-zero only on query failure. Version-constrained refs (`lib:…>=…`, `pc:…==…`) never reach the helper — the resolver skips them in the batched query (helpers match names exactly), so they resolve from local presence or USM packages only. ### Plan contract ` …` → STDOUT single JSON object, no side effects: the material install set — the chosen names plus every dependency of a dependency — in the manager's own transaction order: ```json { "packages": ["libgpm2", "libncurses6", "sl"] } ``` Used to preview and drive progress for the whole native transaction. Any resolution failure exits non-zero with the message on STDERR (the caller degrades to the input names). ### Install contract ` …` → STDOUT JSONL events: `{"type":"begin","total":N}` / `{"type":"package","name":…,"current":n,"total":N,"progress":f}` / `{"type":"package-complete","name":…}` / `{"type":"complete","status":"ok","installed":N}` / terminal `{"type":"error","message":…}` before non-zero exit. ### Exit codes | Code | Meaning | |---|---| | 0 | Query completed (refs may still be listed in `not-found`), or install transaction finished | | 1 | Unexpected helper failure, after emitting a terminal `error` event | | 2 | Usage error (invalid arguments) | | 3 | Query failure, or install marking/resolution failure | | 4 | Package download failure | | 5 | rpm transaction failure | The query subcommand never modifies system state (at most it refreshes package-metadata caches). The install subcommand requires root, follows the system's dnf GPG settings, and never prompts interactively: a missing GPG key or unresolved dependency fails instead. ## Container deployment `usm manifest deploy` builds a single-stage container image for the manifest in the current directory: the package is created with the `usm manifest package` logic, a deploy context is generated into `.usm-deploy/`, the image is built with podman (build happens INSIDE the container — the image is not cross-compiled on the host), and the result is saved as an xz-compressed archive. `usm deploy ` is the package-then-deploy convenience taking either a manifest directory (delegates directly) or a `.usmc` archive (extracted to a temporary directory first; the finished artifact is moved next to the invocation directory). ```bash usm manifest deploy # defaults below usm manifest deploy --exec "my-app 8080" # entrypoint words usm manifest deploy --base quay.io/fedora/fedora:43 # base image override usm manifest deploy --spm apt # wire the apt SPM helper usm manifest deploy --repository web-stack.usmr # repeatable; replaces the set usm manifest deploy --no-build # stop after generating the context usm deploy ./example-app --repository web-stack.usmr # directory form usm deploy example-app-1.0.0.usmc # package form ``` | Flag | Meaning | |---|---| | `--exec CMD` | Container command, split on whitespace into the exec-form ENTRYPOINT. Default: the package's single `bin:` provide as `/usr/bin/`; zero or several `bin:` provides without `--exec` is an error reported before anything is generated | | `--base IMAGE` | Base image (default `registry.fedoraproject.org/fedora:43`, declared as `DEPLOY_DEFAULT_BASE_IMAGE` in `src/cli/Deploy.vala`) | | `--spm dnf\|apt\|apk\|emerge\|pacman\|none` | System package manager wired into the image: the generated `usm.config` points `system_package_manager` at the `usm-spm-` helper the installer ships and sets the install `lib` directory per target (`lib` for the musl/Alpine, Debian-multiarch and Arch merged-usr layouts, `lib64` otherwise), and the Containerfile gains that SPM's bootstrap RUN (dnf installs `python3-dnf`; apt installs `curl python3 python3-apt apt-file` and refreshes the apt-file index; apk adds `bash curl`; emerge needs nothing — stage3 carries portage; pacman installs `curl` and refreshes the files databases with `pacman -Fy`; the curl installs cover base images that ship no fetcher for the USM installer). Default **none**: no `system_package_manager` section and no bootstrap RUN, so the image resolves everything from USM repositories alone (repo-only resolution — anything the repositories cannot provide fails the build) | | `--repository FILE` | Use exactly the given `.usmr` files (repeatable) instead of the machine-configured repositories (`$USM_CONFIGDIR/repos.d`, by default `/etc/usm/repos.d`) | | `--no-build` | Stop after generating the context | | `--installer-url URL` | Override the canonical USM installer source; the `file://` form is the sanctioned local-testing path (see below) | The generated context contains a `Containerfile`, a minimal `usm.config` (managed state under `/var/usm`; with `--spm ` also the SPM wired to `/opt/usm/bin/usm-spm-`), `repos/` with repository descriptors, `repo-trees/` for `file://` repositories, and `package/package.usmc`. The image pre-seeds the package into the USM cache and runs `usm install` in-container: the SPM (when wired) provides platform and toolchain resources, USM repositories provide the rest, and failures fail the build loudly. Only repository PUBLIC keys ever enter a context or image; application secrets such as `web-config.json` are never packaged and belong at runtime, e.g. `podman run --rm -v ./web-config.json:/etc/my-app/web-config.json:ro `. Add the context and artifacts to the project's `.usmignore` so they never get packaged: ``` .usm-deploy/ *.image.tar* ``` ### Installer URL The Containerfile installs USM from the URL held in the `Usm.Installer.CANONICAL_URL` constant in `src/cli/Deploy.vala` (default of the `USM_INSTALLER_URL` build ARG). The current value is a placeholder on a reserved documentation domain that fails the build loudly until replaced with the real hosted installer. The installer is downloaded to a file and run as `sh -y` rather than `curl | sh`, because it extracts its payload archive relative to `$0`. For local testing, point `--installer-url` at a `file://` URL; the referenced script is carried into the context automatically: ```bash usm manifest deploy --installer-url file:///path/to/install-usm.sh ``` ### file:// repository provisioning A `.usmr` whose URI is `file://` references a tree on the build host that the container cannot see. Deploy copies that tree into the context (`repo-trees//`) and rewrites the copied descriptor's URI to the in-image location `file:///usr/share/usm-repos/`, so in-container `usm install` resolves it without the host. Key material named `keys/` or `private-key*` is never copied. Remote (`https://` and friends) descriptors are copied verbatim and fetched at image build time. ### Building, loading and running ```bash usm manifest deploy # podman build -t : + save + xz podman load -i -.image.tar.xz podman run --rm : ``` The artifact is `-.image.tar.xz` next to the deploy context; `podman load -i` accepts the xz-compressed archive directly. Tag characters outside `[A-Za-z0-9_.-]` (usm release suffixes like `+`, for instance) are mapped to dashes in the tag; the artifact filename keeps the unsanitised version.