usm

Universal Source Manifest

Manifest format (MANIFEST.usm)

{
  "name": "my-package",
  "version": "1.0.5+2",
  "summary": "A sample package",
  "licences": [ {"name": "GPLv3", "category": "libre", "text": "src/LICENSE"} ],
  "provides": {
    "bin:hello-world": "hello-world",
    "app:hello-world.desktop": "hello-world.desktop",
    "res:usr/share/icons/hicolor/scalable/hello-world.svg": "icons/hello-world.svg"
  },
  "depends": {
    "runtime": [
      "lib:libgtk-4.so.1",
    ],
    "build": [
      "bin:valac",
      "bin:meson",
      "inc:gtk-4.0"
    ],
    "manage": [
      "bin:bash"
    ]
  },
  "flags": [
    "simpleBuildEnvironment"
  ],
  "execs": {
    "install": "usm-scripts/install",
    "remove": "usm-scripts/remove",
    "build": "usm-scripts/build",
    "rebuild": "usm-scripts/rebuild",
    "test": "usm-scripts/test"
  }
}

Optional extended properties

{
  "md": "src/DESCRIPTION.md",
  "url": "https://my.package.com",
  "screenshots": [ "hello-world-screenshot.png" ],
  "icon": "hello-world.svg",
  "git": { "origin": "https://git.my.package.com/hello-world", "commit": "8d9f3b198e3a767d41f3c33fac2fbd8496f23ed2" },
  "metainfo": "hello-world.appdata.xml",
  "extras": { "customData": "whatever" }
}

Manifest flags

USM supports several flags that modify the build and installation behavior:

• "buildInSourceTree": Tell USM to not create a separate build directory, and compile the package "in place".
• "setManifestPropertyEnvs": Sets environment variables based on manifest properties (TODO: document specifics).
• "simpleBuildEnvironment": Copy the full source tree to the build directory before running the build exec, and execute all build-related scripts (build, install, rebuild, test, postInstall) from the build directory instead of the source directory. This provides a clean, isolated build environment.
• "dataPackage": Marks the package as a data package; see Data packages below.

Resource types

• "rootpath": Generic resource: file on root filesystem without leading "/"
• "path": Generic resource: file name locatable /usr directory
• "opt": Optional software: file name locatable /opt directory
• "res": Generic resource: resource under /usr/share directory
• "cfg": Configuration file: file name locatable /etc directory
• "bin": Binary executable: file name locatable in $PATH
• "sbin": Binary executable: file name locatable in /usr/sbin or /sbin
• "lib": Shared library: file name locatable by LD
• "libexec": Binary executable: file name locatable in /usr/libexec
• "libres": Generic resource: file name locatable /usr/lib, /usr/lib64, /lib, or /lib64 directories
• "info": Information page: file name locatable /usr/share/info
• "man": Manual page: file name locatable /usr/share/man
• "locale": Locale: file name locatable /usr/share/locale
• "app": Desktop entry: file name locatable in /usr/share/applications
• "inc": Include: file name in /usr/include
• "pc": Package config: file name locatable in $PKG_CONFIG_PATH
• "vapi": Vala API: file name in /
• "gir": GObject introspection: file name locatable in /usr/share/gir
• "typelib": GObject typelib: file name locatable in /usr/lib64/girepository-1.0, /usr/lib/girepository-1.0, /lib64/girepository-1.0, or /lib/girepository-1.0
• "gio": GIO plugin module: module filename locatable in /usr/lib64/gio/modules, /usr/lib/gio/modules or /usr/lib/<multiarch-triple>/gio/modules
• "tag": A USM system tag, dot seperated with ".tag" appended, i.e. "hello.world" would be locatable in /usr/share/usm-tags/hello/world.tag

Source Repositories (Repo.usmr)

{
  "name": "my-repo",
  "summary": "My software repository",
  "uris": [
    "https://my.software/repository",
  ],
  "key": "EDXLsUvOZEne+xcv+huvSaqNBs8TTldCv6hd69GdmYw="
}

Source Repository listing (PACKAGES.usml)

Entries in JSON lines format:

{
  "type": "usmc",
  "manifest": {}, // Manifest object from MANIFEST.usm
  "path": "my-package.usmc", // Relative path to the package
  "sha512": "0+6QpQ5Wst2bPmAPYKO/RKrFYuq93rcaRs7tqt9lCo9zu8v6jpJsDqd7yPIqCWDt0KpOQ3uLK64Tmn9FYNw3eQ=="
}

Last line consists of the signatures for the repository

{
  "type": "signatures",
  "signatures": [
    {
      "key": "EDXLsUvOZEne+xcv+huvSaqNBs8TTldCv6hd69GdmYw=",
      "signature": "0rHQiDGKgumMK2S3B6D9jjU3NLnLLjijYJRvaB5vLbp4zxzjlixfyX8yfvvJUCCd34CC1STzyzawDtb9ZammCTE4zFHFdpnO8eseDS5OY+aKgAyL13oBGizvy/V0zGHVCdv83GQ7QUtgSWje0RKKk1zPGDAIF/prkurlFDgd3Vg="
    }
  ]
}

Complete package (package.usmc)

Complete package is simply a tar.xz file containing a MANIFEST.usm file and all the requisite files that would be acquired if the acquire script was run. That is to say, a USMC file could be created by simply running usm manifest acquire followed by tar -cJf ../package.usmc ..

.usmignore

usm manifest package consults an optional .usmignore file at the root of the package (next to MANIFEST.usm) to prune files and directories from the produced archive.

Semantics:

• Blank lines and lines starting with # are skipped.
• A pattern containing / matches the FULL path relative to the package root; a pattern without / matches any path suffix (equivalently, the basename at any depth).
• A trailing / marks a directory-only pattern: it matches the directory itself and everything beneath it.
• * and ? are wildcards and never match /.
• There is no negation: ! is a literal pattern character and cannot re-include an ignored path.
• The root .usmignore and MANIFEST.usm files are always packaged; they can never be ignored.

• The .git directory is always ignored. When no .usmignore file exists, .git is the only default ignore.

  # .usmignore
  
  # Build trees at any depth (directory-only: the directory and all beneath it)
  builddir/
  
  # Database artifacts by basename at any depth
  *.sqlite
  
  # A specific generated tree, anchored to the package root
  docs/generated/
  
  # Wildcards; '*' and '?' never cross a path separator
  src/secret.*
  notes/?draft.md
  

Data packages

A manifest with the "dataPackage" flag ships static files: fonts, icons, shared data sets and the like. Data packages define no build or install machinery:

• Validation rejects a dataPackage manifest that defines any lifecycle executable (build, install, rebuild, test, remove, postInstall); only acquire is permitted.
• usm manifest build is a no-op: no build directory and no build artifacts are created.
• usm manifest install and usm manifest validate copy (or check) each provides entry directly from the source tree — path bases are treated as source: regardless of how they are written — into the destination's suggested path. Staging a data install works the same as any other package, e.g. USM_DESTDIR=/tmp/stage usm manifest install.

• usm manifest acquire and resource bookkeeping on install/remove are unchanged.

  {
  "name": "example-fonts",
  "version": "1.0.0",
  "summary": "The Example font family",
  "licences": [],
  "flags": ["dataPackage"],
  "provides": {
  "res:fonts/Example-Regular.ttf": "source:fonts/Example-Regular.ttf",
  "res:fonts/Example-Bold.ttf": "source:fonts/Example-Bold.ttf"
  },
  "depends": { "runtime": [], "build": [], "manage": ["bin:bash"] }
  }
  

USM Directory /var/usm

• /packages
  • /packagename-0.1.1
  • /origin-info
  • /package.usmc
  • /build-log (when built)
  • /build.tar.xz (when built)
  • /build (when building)
• /installed
  • /packagename-0.1.1 (symlink to /var/usm/packages/packagename-0.1.1)
• /lists
  • /repositoryname
  • /2020-03-20T14:28:23.382748.usml
  • /2020-03-20T14:34:42.382748.usml
• /transactions
  • 2020-03-21T14:28:23.382748
  • 2020-03-23T14:23:43.382748

Origin info format

{
    "repository": "repositoryname",
    "listfile": "2020-03-20T14:34:42.382748.usml",
    "original_path": "packagename-0.1.1.usmc",
    "signature_verified": true
}

Resource Ref Additions

• Add version expressions (can copy from MPK), i.e. == >= <= <>
• Add support for version expressions to some resource types, namely pkg and pc

Below here are draft ideas that may or may not get implemented

Transactions

• Create backup BTRFS snapshot
• Create transaction BTRFS snapshot from backup snapshot
• Apply difference between backup and transaction snapshots to live filesystem
• Delete transaction snapshot

JSONL transaction journal format

{
  "operation": "begin-transaction",
  "timestamp": "2020-03-20T14:34:42.382748",
  "snapshot": "path/to/backup/snapshot",
  "subvolume": "path/to/transaction/subvolume"
}

{
  "operation": "begin-transaction",
  "timestamp": "2020-03-20T14:34:42.382748",
}

Scriptable installation

• USM will not manage the installation if an install exec is present
• Compare backup and transaction snapshots for validation
• Scripts can optionally handle installation but it must be verified that the expected files (and only the expected files) are installed
• Add post-install and maybe post-transaction execs that are not subject to validations

USM Commands

Manifest Commands

• usm manifest build <build path>: Build the package
• usm manifest install <build path>: Build and install the package to the system
• usm manifest remove: Remove the package from the system
• usm manifest acquire: Run the acquire script to download sources
• usm manifest package: Create a package archive
• usm manifest deploy [--exec CMD] [--base IMAGE] [--repository FILE]... [--no-build] [--installer-url URL]: Build a container image from the manifest directory; see Container deployment
• usm manifest autoprovides [--replace] [--debug] [build path]: Scan installation and generate provides section
• usm manifest test [build path]: Run package tests
• usm manifest validate [build path]: Validate package by building and checking resources

Repository Commands

• usm repository init <name>: Create the canonical repository layout in the current directory: <name>.usmr, keys/ (signing keys, private key mode 0600 and gitignored via .gitignore), and an empty public/ packages directory
• usm repository add <pkg.usmc>: Validate the package (manifest + checksum), copy it into public/, then rebuild and re-sign PACKAGES.usml; a package with the same name and version is rejected unless identical
• usm repository remove <name|filename>: Delete the matching .usmc from public/ (exact filename, or <name>-<version> prefix; an ambiguous name lists the candidates and asks for the filename), then rebuild and re-sign
• usm repository list [repo.usmr]: Print name, version and summary of every package in the signed listing (from the repository root, or fetched from the given .usmr's URI)
• usm repository verify <repo.usmr>: Check the listing signature, then deep-verify every entry's sha512 against the .usmc fetched from the repository URI; exits non-zero on any failure
• usm repository publish <path>: Copy the .usmr and PACKAGES.usml to <path> (always overwriting) and each public/*.usmc only when the destination copy is missing or older by mtime
• usm repository new <name> / usm repository build-list: Legacy commands using loose public-key/private-key files in the working directory

All commands except init, new and verify locate the repository root by searching the current directory and its ancestors for a directory containing a *.usmr file, a keys/ directory and a public/ directory. Signing keys load from keys/public-key and keys/private-key in the repository root, falling back to the legacy loose public-key and private-key files.

Validation Command

The validate command performs comprehensive validation of USM packages:

usm manifest validate [build path]

The validation process includes:

• Manifest syntax validation
• Dependency checking
• License verification (with warning if missing)
• Build execution with progress reporting
• Install execution to temporary directory
• Resource validation based on path base type (as-expected, build:, source:, install:)
• Unexpected resource detection

Exit codes indicate specific validation failures, making it suitable for CI/CD environments.

For detailed information, see slopdocs/utility.usm.manifest.validate.md.

System package manager integration

USM can delegate missing resources to the system package manager (DNF and friends) before falling back to USM repositories. The integration is configured in usm.config and talks to a helper executable through two stable contracts; five reference helpers ship under spm/ (installed to bindir as usm-spm-<spm>):

Helper	Distro(s)	Language	Container bootstrap	Limitations
spm/dnf/usm-spm-dnf	Fedora	Python 3 + dnf4	dnf install -y python3-dnf && dnf clean all	—
spm/apt/usm-spm-apt	Debian, Ubuntu	Python 3 + python3-apt	apt-get update && apt-get install -y curl python3 python3-apt apt-file && apt-file update && apt-get clean	file queries go through apt-file only (no naming heuristics), so the contents index downloaded by apt-file update (~100 MB) is mandatory; paths follow Debian multiarch (/usr/lib/x86_64-linux-gnu, both pkgconfig dirs)
spm/apk/usm-spm-apk	Alpine	POSIX shell (busybox ash)	apk add --no-cache bash curl	dependency-free — no python3, no jq; queries ride apk's native pc:/so: provides and the /lib (not lib64) busybox layout
spm/emerge/usm-spm-emerge	Gentoo	Python 3 + portage	none — the stage3 base already carries portage + python	offline ebuilds expose no file lists: file queries for not-yet-installed packages resolve only via installed files, virtuals and a small package-name table, otherwise land in not-found; dependency counts come from emerge --pretend (see the usm-spm-emerge docstring for the full limits)
spm/pacman/usm-spm-pacman	Arch	POSIX shell (bash)	pacman -Sy --noconfirm curl && pacman -Fy --noconfirm && rm -rf /var/cache/pacman/pkg/*	dependency-free like the apk helper: bsdtar (a pacman dependency) parses the sync databases for soname provides and the files databases for file queries, so the pacman -Fy refresh (~40 MB) is mandatory — without it only installed files resolve (pacman -Qo), no naming heuristics; dependency counts come from a --print run against a throwaway --dbpath; merged-usr /usr/lib layout

All five implement the identical JSON/JSONL contract below; usm deploy --spm (see Container deployment) wires the matching one into generated images.

Configuration

"system_package_manager": { "query": ["usm-spm-dnf","query"], "install": ["usm-spm-dnf","install"] }

Both keys or neither (validation error otherwise); argv arrays, no shell.

Query contract

<query> <usm-ref>… → STDOUT single JSON object, no side effects:

{ "not-found": ["pc:foo.pc"],
  "packages": [ { "name": "vala", "resources": ["bin:valac","vapi:gtk-4.0.vapi"], "dependency-count": 5, "installed-dependency-count": 3 } ] }

dependency-count = packages the manager would install for this package alone incl. itself. Exit 0 with non-empty not-found; non-zero only on query failure.

Version-constrained refs (lib:…>=…, pc:…==…) never reach the helper — the resolver skips them in the batched query (helpers match names exactly), so they resolve from local presence or USM packages only.

Plan contract

<plan> <native-name>… → STDOUT single JSON object, no side effects: the material install set — the chosen names plus every dependency of a dependency — in the manager's own transaction order:

{ "packages": ["libgpm2", "libncurses6", "sl"] }

Used to preview and drive progress for the whole native transaction. Any resolution failure exits non-zero with the message on STDERR (the caller degrades to the input names).

Install contract

<install> <native-name>… → STDOUT JSONL events:

{"type":"begin","total":N} / {"type":"package","name":…,"current":n,"total":N,"progress":f} / {"type":"package-complete","name":…} / {"type":"complete","status":"ok","installed":N} / terminal {"type":"error","message":…} before non-zero exit.

Exit codes

Code	Meaning
0	Query completed (refs may still be listed in not-found), or install transaction finished
1	Unexpected helper failure, after emitting a terminal error event
2	Usage error (invalid arguments)
3	Query failure, or install marking/resolution failure
4	Package download failure
5	rpm transaction failure

The query subcommand never modifies system state (at most it refreshes package-metadata caches). The install subcommand requires root, follows the system's dnf GPG settings, and never prompts interactively: a missing GPG key or unresolved dependency fails instead.

Container deployment

usm manifest deploy builds a single-stage container image for the manifest in the current directory: the package is created with the usm manifest package logic, a deploy context is generated into .usm-deploy/, the image is built with podman (build happens INSIDE the container — the image is not cross-compiled on the host), and the result is saved as an xz-compressed archive. usm deploy <path> is the package-then-deploy convenience taking either a manifest directory (delegates directly) or a .usmc archive (extracted to a temporary directory first; the finished artifact is moved next to the invocation directory).

usm manifest deploy                                  # defaults below
usm manifest deploy --exec "my-app 8080"             # entrypoint words
usm manifest deploy --base quay.io/fedora/fedora:43  # base image override
usm manifest deploy --spm apt                        # wire the apt SPM helper
usm manifest deploy --repository web-stack.usmr      # repeatable; replaces the set
usm manifest deploy --no-build                       # stop after generating the context
usm deploy ./example-app --repository web-stack.usmr # directory form
usm deploy example-app-1.0.0.usmc                    # package form

Flag	Meaning
--exec CMD	Container command, split on whitespace into the exec-form ENTRYPOINT. Default: the package's single bin: provide as /usr/bin/<name>; zero or several bin: provides without --exec is an error reported before anything is generated
--base IMAGE	Base image (default registry.fedoraproject.org/fedora:43, declared as DEPLOY_DEFAULT_BASE_IMAGE in src/cli/Deploy.vala)
--spm dnf\|apt\|apk\|emerge\|pacman\|none	System package manager wired into the image: the generated usm.config points system_package_manager at the usm-spm-<spm> helper the installer ships and sets the install lib directory per target (lib for the musl/Alpine, Debian-multiarch and Arch merged-usr layouts, lib64 otherwise), and the Containerfile gains that SPM's bootstrap RUN (dnf installs python3-dnf; apt installs curl python3 python3-apt apt-file and refreshes the apt-file index; apk adds bash curl; emerge needs nothing — stage3 carries portage; pacman installs curl and refreshes the files databases with pacman -Fy; the curl installs cover base images that ship no fetcher for the USM installer). Default none: no system_package_manager section and no bootstrap RUN, so the image resolves everything from USM repositories alone (repo-only resolution — anything the repositories cannot provide fails the build)
--repository FILE	Use exactly the given .usmr files (repeatable) instead of the machine-configured repositories ($USM_CONFIGDIR/repos.d, by default /etc/usm/repos.d)
--no-build	Stop after generating the context
--installer-url URL	Override the canonical USM installer source; the file:// form is the sanctioned local-testing path (see below)

The generated context contains a Containerfile, a minimal usm.config (managed state under /var/usm; with --spm <spm> also the SPM wired to /opt/usm/bin/usm-spm-<spm>), repos/ with repository descriptors, repo-trees/ for file:// repositories, and package/package.usmc. The image pre-seeds the package into the USM cache and runs usm install in-container: the SPM (when wired) provides platform and toolchain resources, USM repositories provide the rest, and failures fail the build loudly. Only repository PUBLIC keys ever enter a context or image; application secrets such as web-config.json are never packaged and belong at runtime, e.g. podman run --rm -v ./web-config.json:/etc/my-app/web-config.json:ro <image>.

Add the context and artifacts to the project's .usmignore so they never get packaged:

.usm-deploy/
*.image.tar*

Installer URL

The Containerfile installs USM from the URL held in the Usm.Installer.CANONICAL_URL constant in src/cli/Deploy.vala (default of the USM_INSTALLER_URL build ARG). The current value is a placeholder on a reserved documentation domain that fails the build loudly until replaced with the real hosted installer. The installer is downloaded to a file and run as sh <file> -y rather than curl | sh, because it extracts its payload archive relative to $0.

For local testing, point --installer-url at a file:// URL; the referenced script is carried into the context automatically:

usm manifest deploy --installer-url file:///path/to/install-usm.sh

file:// repository provisioning

A .usmr whose URI is file:// references a tree on the build host that the container cannot see. Deploy copies that tree into the context (repo-trees/<name>/) and rewrites the copied descriptor's URI to the in-image location file:///usr/share/usm-repos/<name>, so in-container usm install resolves it without the host. Key material named keys/ or private-key* is never copied. Remote (https:// and friends) descriptors are copied verbatim and fetched at image build time.

Building, loading and running

usm manifest deploy                    # podman build -t <name>:<version> + save + xz
podman load -i <name>-<version>.image.tar.xz
podman run --rm <name>:<version>

The artifact is <name>-<version>.image.tar.xz next to the deploy context; podman load -i accepts the xz-compressed archive directly. Tag characters outside [A-Za-z0-9_.-] (usm release suffixes like +, for instance) are mapped to dashes in the tag; the artifact filename keeps the unsanitised version.