1# rust-frontier — self-hosting the builder's rust toolchain
2
3The last frontier of `selfhost-verify` variante (b): replace the Alpine
4`rustc/cargo 1.91.1` in `/toolchain` with a hammer-built rust, via the **purist
5mrustc chain** (no prebuilt rustc binary). Path:
6
7    mrustc → rustc 1.90.0 (builds bundled LLVM) → 1.91.0 → 1.91.1
8
9These scripts + patch are the working artifacts. The mrustc tree itself lives in
10`.scratch/mrustc` (gitignored — it's a large third-party clone, upstream master
11`be69c74`); only our deltas are tracked here.
12
13## Files
14
15- **`run_rustc.patch`** — patch against mrustc `be69c74` for `run_rustc/Makefile`
16  and `run_rustc/rustc_proxy.sh`. Apply from the mrustc root:
17  `git -C .scratch/mrustc apply /path/to/run_rustc.patch`
18- **`build-stage0-musl.sh`** — rebuild the mrustc stage0 `rustc`+`cargo` with a
19  **musl host triple** (`output-1.90.0-x86_64-unknown-linux-musl/`). Run inside
20  the Alpine devfs sandbox (gcc, MRUSTC_TARGET_VER=1.90). Reuses the already-built
21  LLVM and `bin/mrustc`.
22- **`run-runrustc.sh`** — run a `run_rustc` make target inside the devfs sandbox,
23  pointed at the musl-host stage0.
24- **`bootstrap.toml`** — rust's own bootstrap config for hop 1 (1.90.0 → 1.91.0):
25  `build`/`host`/`target = x86_64-unknown-linux-musl`, stage0 `rustc`/`cargo` =
26  the run_rustc musl-host 1.90.0 prefix, external LLVM 20.1.8 reused via
27  `llvm-config` (clears bootstrap's `>=19` gate, no LLVM rebuild), `crt-static =
28  false` (dynamic link against the devfs system musl, like Alpine and the
29  run_rustc toolchain), `rpath = true`. Copy it into the rust source root.
30- **`bootstrap-1.91.1.toml`** — config for hop 2 (1.91.0 → 1.91.1). Same as hop 1
31  except: stage0 `rustc = /stage0/bin/rustc` (the installed hop-1 toolchain, bound
32  read-only via `STAGE0`), stage0 `cargo` = the run_rustc 1.90.0 cargo (1.90
33  satisfies bootstrap's "minor-1" check for a 1.91.x source), `extended = true` +
34  `tools = ["cargo"]` so the install yields a hammer-built cargo, and
35  **`cargo-native-static = true`** — the hermetic devfs has no system
36  OpenSSL/curl/libgit2, so this makes bootstrap build cargo with the `all-static`
37  feature set (vendored openssl/libgit2 + static curl/libz, compiled from the
38  vendored C sources via gcc). Without it the cargo tool fails on `openssl-sys`.
39- **`swap-rust-into-toolchain.sh`** — overlay the hammer-built rust 1.91.1
40  (`.scratch/rust-1.91.1-prefix`) onto a toolchain dir (default `.dev-fs/alpine`)
41  so `hammer build` / `bootstrap stage1` compile the 4/4 rust inputs (arje-zero,
42  hammerd) with hammer-rust. Additive + reversible: hammer's `x86_64-unknown-linux-musl`
43  rustlib and hash-suffixed `.so`s don't collide with Alpine's
44  `x86_64-alpine-linux-musl`, so only `/usr/bin/{rustc,cargo}` are replaced (Alpine
45  backed up to `*.alpine`). `--restore` undoes it. Wired into `selfhost-verify.sh`
46  as `SWAP_RUST=1` (applies before the stage1 build, restores on exit via trap).
47  **This swap changes `of_tree`** (rustc emits the binaries), so the criterion is
48  auto-consistency against a NEW `RUST_EXPECT_REF`, not the Alpine `9adefb82`.
49- **`run-xpy.sh`** — run rust's `x.py` inside the devfs sandbox, binding the
50  source at `/src`, the install prefix at `/out`, and the mrustc tree at `/mrustc`
51  (so the stage0 + `llvm-config` paths in `bootstrap.toml` resolve). Set the
52  `STAGE0` env var to also bind a prior hop's installed prefix read-only at
53  `/stage0`. Usage:
54  `run-xpy.sh <SRCDIR> <PREFIXDIR> -- build --stage 2 compiler/rustc library/std`.
55
56> The runners hardcode `/home/sergio/hammer` and assume the prepared devfs at
57> `.dev-fs/alpine` (256-TLS-key musl loader + g++; see the bootstrap-devfs notes).
58
59## Why the patches exist (the snags)
60
61The lab is **musl-only** (Alpine devfs); the 4/4 inputs (arje-zero/hammerd) are
62built `--target x86_64-unknown-linux-musl`. Three coupled problems had to be fixed
63to bootstrap a usable rust there:
64
651. **stage-1 std built for gnu** (`mmap64`/`open64` undefined at link).
66   `minicargo` was invoked without `--target`, so it used mrustc's default **gnu**
67   triple — and gnu std references glibc's LFS64 symbols that musl 1.2.5 doesn't
68   export. Fix: pass `--target $(RUSTC_TARGET)` to minicargo (Makefile:162) and to
69   the stage-1 smoke-test (Makefile:172). A correct musl std uses the **base** libc
70   symbols (`open`/`mmap`/…), matching Alpine's reference std.
71
722. **self-contained CRT absent.** musl's default link is `-static-pie` against a
73   `lib/rustlib/<t>/lib/self-contained/` CRT dir that the mrustc sysroot never
74   populates (also needs static `-lunwind`). Fix: `-C target-feature=-crt-static
75   -C link-self-contained=no` so the Alpine gcc supplies crt1/crti/crtn +
76   crtbeginS/crtendS and links dynamically against the system musl (RUSTFLAGS in
77   Makefile:111, plus **both** hello_world smoke-test rules: the stage-1 one at
78   Makefile:172 *and* the final fully-bootstrapped one at Makefile:262 — the latter
79   only surfaces once stage-2/3 actually completes, i.e. after snag #3 is fixed).
80
813. **host-triple = gnu on a musl system** (the deep one). cargo compiles
82   build-scripts and **proc-macros** for the *host*; mrustc's stage0 rustc has its
83   host triple baked as `x86_64-unknown-linux-gnu` (not runtime-overridable). A
84   proxy `--target musl` inject fixes build-script *bins*, but **proc-macros are
85   dlopen'd and must match the host triple structurally** — injection makes them
86   target artifacts that rustc rejects. Aliasing musl rlibs as gnu also fails:
87   rustc reads the triple from rlib *metadata* (`E0461`), not the path.
88
89   **The only fix is a stage0 rustc whose host triple is musl** — that's what
90   `build-stage0-musl.sh` produces (`CFG_COMPILER_HOST_TRIPLE` via `RUSTC_TARGET`,
91   `--target` via `MRUSTC_TARGET`). Then host==target==musl: cargo doesn't
92   cross-compile, proc-macros load natively, and `rustc_proxy.sh` only needs the
93   CRT flags (no `--target` inject).
94
95   Gotcha: do **not** pass `OUTDIR_SUF` on make's command line — a command-line var
96   can't be reassigned by the makefile, so the `-musl` suffix append is skipped and
97   the build contaminates the gnu output dir. Let it default.
98
994. **final cargo needs vendored OpenSSL.** The fully-bootstrapped `prefix/bin/cargo`
100   rule (Makefile:235) builds cargo with default features, so `openssl-sys` looks
101   for a system OpenSSL via pkg-config and fails (the hermetic musl devfs has none,
102   and no `openssl.pc`). Fix: add `--features vendored-openssl` to that cargo build
103   so `openssl-src` compiles OpenSSL from source — same approach the stage0 cargo
104   build uses. Real cargo (unlike minicargo) handles the OUT_DIR fine, so snag #4
105   doesn't recur here. Only surfaces after the final hello_world (snag #2/262) passes.
106
1075. **minicargo OUT_DIR host/target split** (only when building the musl stage0's
108   cargo). With `RUSTC_TARGET` set, minicargo runs each cargo build-script under
109   `cargo-build/host/build_X/`, but the *target* crate's `include!` /
110   `include_bytes!(concat!(env!("OUT_DIR"), "/.."))` resolves `OUT_DIR` to
111   `cargo-build/build_X/` (no `host/`, empty) → mrustc aborts with signal 6. Hit
112   `libsqlite3-sys` (`bindgen.rs`) and the `cargo` binary (`man.tgz`). The gnu
113   build never split (single triple → single dir). Since host==target the script
114   outputs are bit-identical, so `build-stage0-musl.sh` symlinks the target dir to
115   the host dir (`symlink_outdirs`). Late sys-crates (cargo, curl, openssl,
116   libgit2-sys, libssh2-sys) run their scripts near the end, so it's applied in a
117   retry loop around the incremental cargo build.
118
119## Workflow
120
121```sh
122# 1. apply the run_rustc patch into the mrustc clone
123git -C .scratch/mrustc apply scripts/rust-frontier/run_rustc.patch
124
125# 2. rebuild stage0 rustc+cargo with a musl host triple (~hours, reuses LLVM)
126.scratch/build-stage0-musl.sh            # -> output-1.90.0-x86_64-unknown-linux-musl/
127
128# 3. build the optimised rustc + cargo via run_rustc (stage 1/2/3)
129.scratch/run-runrustc.sh all
130
131# 4. climb hop 1 — 1.90.0 -> 1.91.0 (stage0 = the run_rustc musl-host 1.90.0)
132cp scripts/rust-frontier/bootstrap.toml .scratch/rustc-1.91.0-src/bootstrap.toml
133.scratch/run-xpy.sh .scratch/rustc-1.91.0-src .scratch/rust-1.91.0-prefix -- \
134    build --stage 2 compiler/rustc library/std
135.scratch/run-xpy.sh .scratch/rustc-1.91.0-src .scratch/rust-1.91.0-prefix -- \
136    install --stage 2 rustc library/std        # -> relocatable 1.91.0 (rpath)
137
138# 5. climb hop 2 — 1.91.0 -> 1.91.1 (stage0 rustc = the installed 1.91.0)
139cp scripts/rust-frontier/bootstrap-1.91.1.toml .scratch/rustc-1.91.1-src/bootstrap.toml
140STAGE0=.scratch/rust-1.91.0-prefix .scratch/run-xpy.sh \
141    .scratch/rustc-1.91.1-src .scratch/rust-1.91.1-prefix -- \
142    build --stage 2 compiler/rustc library/std src/tools/cargo
143STAGE0=.scratch/rust-1.91.0-prefix .scratch/run-xpy.sh \
144    .scratch/rustc-1.91.1-src .scratch/rust-1.91.1-prefix -- \
145    install --stage 2 rustc library/std cargo  # -> hammer rustc+cargo 1.91.1
146
147# 6. swap rust 1.91.1 into /toolchain + auto-consistency verify (in-VM)
148#    PRESEED=hammerd es OBLIGATORIO con SWAP_RUST: arje-zero (monorepo tawasuyu) vendorea
149#    ~1973 crates; en un rebuild in-VM completo (PRESEED=all) ese vendor desborda el
150#    rootfs en RAM de la VM → ENOSPC. Con PRESEED=hammerd se preseedea el arje-zero
151#    host-built (hammer-rust, locked) y sólo hammerd se reconstruye in-VM.
152SWAP_RUST=1 KVM=1 MEM=16384 PRESEED=hammerd ./scripts/selfhost-verify.sh
153```
154
155**DONE (2026-06-16):** both hops built. `rust-1.91.1-prefix/bin/{rustc,cargo}` run
156standalone (rpath), `rustc 1.91.1 (ed61e7d7e)` host `x86_64-unknown-linux-musl`
157LLVM 20.1.8, `cargo 1.91.1`. Alpine's exact rust version, fully self-hosted from
158the mrustc chain (mrustc → 1.90.0 → 1.91.0 → 1.91.1). Each hop ≈45 min compile
159(LLVM reused throughout); hop 2's cargo needed the `cargo-native-static` fix.
160
161**✓ AUTO-CONSISTENCY VERIFICADA IN-VM (2026-06-18):** `SWAP_RUST=1 … PRESEED=hammerd`
162→ el host construye stage1 con hammer-rust (of_tree `7fa6cb4e`), la VM lo reconstruye
163con hammer-rust y reproduce `7fa6cb4e` **bit a bit** (`stage1' == stage1`, DRIVER_RC=0).
164`RUST_EXPECT_REF=7fa6cb4e` (≠ el `9adefb82` de Alpine — rustc emite los bytes). El drift
165de arje-zero quedó cerrado fijando su `Cargo.lock` en tawasuyu (`recipes/arje-zero.toml`
166→ commit `9967b02c`): build `--locked` ⇒ bit-reproducible (`lock1 == lock2`).
167
168Full diagnostic history is in the `selfhost-verify-rust-frontier` memory.