FrankenRobotics

Spark: GPU backends and the kernel toolchain

12,693 words26 subsections21 tables5 listingsPlan version 1.1

On this page26
21.1 What Spark owns21.2 Position relative to suite doctrine21.3 Decisions21.4 Mechanisms and their tiers21.5 The kernel library21.6 Two layers of source21.7 The kernel IR21.8 Kernel classes: floating, exact-integer and guarded21.9 Backends21.10 The device arena, leases and the binary kernel cache21.11 Execution model: shape classes, variants, buckets and the step plan21.12 The placement planner and its receipts21.13 The CPU-versus-device check21.14 Precision21.15 Determinism on devices21.16 The hardware fingerprint21.17 Division of labour between the machines, and the fleet scheduler21.18 Specialization for survivors only; the fused step kernel21.19 Routes considered and not taken21.20 Rejected mechanisms, with the calculation that refutes each21.21 Refusals21.22 Invariants and proof obligations21.23 Known answers and negative controls21.24 Gates and kill gates21.25 Open questions21.26 Counts beyond 2³² without 64-bit atomics

21.1 What Spark owns

Spark owns everything between a derived kernel and a device result: the kernel library and its manifests, the kernel intermediate representation and its emitters, the two device backends, the device arena and its leases, the placement planner, the binary kernel cache, the hardware fingerprint, the CPU-versus-device check with its divergence ledger, and the device side of the division of labour between the two machines. It does not own the physics of a stage (§16, §18), the meaning of a family of bodies (§17), the determinism classes (§35) or the performance targets (§36). It implements them and reports against them.

Item Where
Crates frob-kir (layer 1), frob-kernels (layer 3), annexes derive/frob-derive, spark/frob-spark-cuda, spark/frob-spark-metal
Verbs spark.devices, spark.fingerprint, spark.probe, spark.check, spark.precompile, spark.place, spark.cache; fleet.status, fleet.plan
Records Kernel manifest frob.kernel/1, kernel IR frob.kir/1, step plan frob.stepplan/1, fingerprint frob.fingerprint/1, placement receipt frob.placement/1
Admission INT-CUDA-01 (cudarc) and INT-FT-01 (ft-kernel-metal), both at M0 (§33)

21.2 Position relative to suite doctrine

FACT: FrankenSim's accelerator doctrine states "FrankenSim does not ship an accelerator backend", requires a "Permanent CPU reference", and defines twelve evidence fields (AE-01 to AE-12) for any device run: device identity, driver and runtime, compiler and backend, kernel source hash, kernel and workload identity, CPU axes, reduction and determinism policy, CPU-versus-device comparison, the retained CPU reference, cancellation outcome, phase attribution, and a go or no-go decision (frankensim/docs/ACCELERATOR_DOCTRINE.md). In-process foreign kernels are forbidden in FrankenSim's production paths. The suite's working GPU code lives outside FrankenSim's workspace in what its manifests call the engine-kit pattern, and in FrankenTorch's ft-kernel-metal, which FrankenTorch calls its sanctioned unsafe boundary (FACT: frankentorch/crates/ft-kernel-metal/src/lib.rs).

FACT: the same doctrine names a falsifier. A device pilot is refused when the top three kernels jointly account for less than 50% of end-to-end wall time, or when no candidate suitable after transfer and synchronization costs holds at least 15% of it. Spark applies both thresholds to its own optimization funding (§36.8).

ADR-SPARK-01. FrankenRobotics has an explicit owner mandate for CUDA and low-level Metal. It meets the mandate without weakening the core:

  • The core workspace is unsafe_code = "deny" with Franken-only runtime dependencies and never depends on a GPU crate.
  • GPU backends live in annex workspaces under spark/, each with its own lockfile and a registry of unsafe capsules with a SAFETY.md each. They build on T, the one toolchain of the build graph (§33.13); the boundary exists for the unsafe and foreign-interface policy (§34.1).
  • Binaries that use a GPU are built from an apps/ workspace that links the core and an annex.
  • Every device run emits a receipt with FrankenSim's twelve evidence fields, unchanged. FrankenRobotics adds fields beside them (placement per stage, fingerprint digest, determinism class, capacities); it never renames or drops one of the twelve.
  • The CPU reference is permanent (§16).

Owner ratification is OPEN-SPARK-1; the fail-closed default is that GPU results are labeled exploratory until ratified.

What is claimed. Compiling one kernel source for several backends is prior art (Warp for CPU and CUDA; Genesis's compiler for CUDA, Metal and others; FACT, Appendix H) and is not claimed as new. The claim Spark may make, once GATE-SPARK-02 has passed, is narrower: kernel arithmetic is derived once and emitted for a CPU reference, CUDA and Metal, and the measured cross-backend disagreement is stored per kernel.

21.3 Decisions

ADR Decision Reason
ADR-SPARK-01 Annex workspaces, unchanged evidence fields, permanent CPU reference Above
ADR-SPARK-02 One shared C-dialect source for skeletons plus derived arithmetic stays the route. The kernel IR gains bounded loops and resource metadata and keeps dimension typing. It has no sine, cosine, fused multiply-add or fast mode The pattern already works in the suite (§21.6); a second source language would double the surface that must stay inside a determinism class
ADR-SPARK-03 Three kernel classes: floating, exact-integer, guarded. A strict predicate or a digest is never computed in bare floating point. Representation is chosen before placement A recorded pilot mislabeled 23,890 of 65,536 near-boundary cases with a bare f32 comparison and none with integers (§21.8)
ADR-SPARK-04 A batch is a bucket of one shape class. Each world indexes one of K model variants and one of K scene variants inside padded capacities. No shape changes inside a running environment A population of designs is the studio's main workload; static shapes keep launch graphs static (§21.11)
ADR-SPARK-05 Placement is planned and receipted per stage. A strict backend request fails before any scoring if a stage has no kernel there An agent must be able to tell acceleration from a silent fallback (§21.12)
ADR-SPARK-06 One arena per bucket, allocated once, read by the host only under a lease. Compiled binaries live in a content-addressed cache keyed by the hardware fingerprint No allocation in the hot path; no stale binary after a driver change (§21.10)
ADR-SPARK-07 The machines have roles. Only seeds, recipes, design differences, weights, receipts and summaries cross between them Regeneration from a seed is exact and cheaper than moving fields (§21.17)
ADR-SPARK-08 Per-design kernel specialization is built for survivors only, only if our own profile justifies it, and only bit-identical to the generic kernel The incumbent step trace caps recursion-only specialization near 1.2× (§21.18); sparsity in constraint assembly and solve is carried as data and is measured separately (§16)
ADR-SPARK-09 Metal is reached through ft-kernel-metal. Missing gateway features are upstream work items done in FrankenTorch's repository (FD-FT-01), never a local patch or a second binding; until they land Spark works inside the gateway's present limits. Kernels never use a grid-wide barrier The gateway is changed where it lives (§33.8); the suite's own renderer stalls on such a barrier (§21.9)
ADR-SPARK-10 Double-float accumulators use the split product, not a fused multiply-add, and every device proves at start-up that its compiler kept the error-free transformations A fused operation is banned in batched kernels (§35.6); a folded transformation fails silently

21.4 Mechanisms and their tiers

The spine is S. A row marked F or M names its pilot, its ceiling and its kill gate in the section cited.

Mechanism Tier Section
Annex backends, receipts, CPU reference, kernel manifests S §21.2, §21.5
Hand-written skeletons in the shared dialect S §21.6
Derived arithmetic through the kernel IR F, KILL-KIR §21.6, §21.7
Exact-integer kernels; guarded single precision with exact fallback S §21.8
CUDA backend, launch graphs, stream-ordered pools S §21.9
Metal backend as a rollout tier F, KILL-METAL §21.9
Arena, leases, binary kernel cache S §21.10
Per-world parameter variants S §21.11
Topology variants in one bucket; design-family buckets F, KILL-BUCKET §21.11
Persistent environments with slot dispatch F, KILL-SLOT §21.11
Placement planner and receipts; strict backends S §21.12
CPU-versus-device check and divergence ledger S §21.13
Bit equality of floating kernels across CPU, CUDA and Metal HYPOTHESIS, KILL-D1 §21.15
Hardware fingerprint S §21.16
Division of labour as policy S §21.17
Fleet scheduler across the two machines F, KILL-FLEET §21.17
Specialized kernels for survivors F, KILL-SPECIALIZE §21.18
Fused per-world step kernel M, KILL-MEGA §21.18
Fixed-point types for an embedded target M, reserved §21.7

21.5 The kernel library

A kernel is a manifest, its arithmetic, and its sources. The manifest holds: KernelId (content hash), its class (§21.8), inputs and outputs with shapes and physical dimensions, the determinism class it claims, the identities it declares (the certificates of §12 attach here), whether a derivative program exists (D-8), and the symbol of its CPU reference. It also holds a stage signature (DESIGN), because a device kernel is only as safe as what the host believes about its buffers:

  • the read footprint and the write footprint, per buffer, as index ranges in terms of capacities;
  • aliasing rules (which arguments may be the same slab; none by default);
  • layout: element type, extents, stride unit (bytes or elements) and stride signedness, logical origin and allocation extent;
  • the ownership epoch the kernel reads and the epoch it writes (§21.10);
  • resource bounds taken from the IR: live values, stack bytes, loop bounds, buffer argument count;
  • the numerical contract: precision profile, error bound if one is certified, neutral elements used for padding.

FACT to recheck at INT-FNP-01 and INT-FT-01: the pinned FrankenNumPy uses signed byte strides and the pinned FrankenTorch unsigned element strides. The smallest case that breaks a naive hand-off: a 2 × 2 contiguous f32 view has strides (8, 4) under the first convention and (2, 1) under the second. Copying (8, 4) verbatim as element strides onto a 16-element buffer reads element 8 + 4 = 12 at index [1, 1], where element 3 belongs, and both reads are in bounds, so nothing faults. A view crossing that boundary therefore carries its stride unit and signedness; a view that the receiver cannot represent is materialized and the copy charged to the receipt, or refused under a zero-copy contract (FROB-REF-617). This is an admission test and an upstream work item (FD-FNP-01, Appendix F).

Families at M2: kinematics (K-fk-*, K-twist-*), dynamics (K-crba-*, K-rnea-*), actuators (K-act-*, including the within-world reduction for a shared bus current limit, K-act-bus), collision (K-geom-*, K-bp-*, K-np-*, K-compact), constraints and solve (K-jac-*, K-cone-*, K-hess, K-scale, K-chol, K-linesearch, K-refine), integration (K-cay-step, K-normalize, K-anchor), sensing (K-ray-*, K-sense-*, with exposure-time and rolling-shutter bins), learning (K-gemm, K-act-fn, K-adam), utility (K-scan, K-sort, K-seg-reduce, K-reduce, K-variant-gather, K-commit, K-snapshot), and the exact family (K-x-pred, K-x-digest, K-x-hist, K-x-philox, K-g-pred, K-canary). One stable integer-key radix sort (K-sort) and one count, exclusive-scan, write compaction (K-compact) serve every stage; no stage carries its own sort.

21.6 Two layers of source

Skeletons are written by hand, twice. The control structure of a kernel (loops, gathers, scans, sorts, barriers inside a thread group) is written once in Rust for batch-cpu and once in the shared dialect: the common subset of CUDA C++ and Metal Shading Language, made portable by a per-backend prelude of macros for address-space qualifiers, kernel signatures and thread-group barriers. This is the pattern already working in frankensim/apps/instrument-lab/render-gpu-piano (one kernel.h, a prelude in cuda.rs, a prelude in metal.rs, and a CPU emulation that is the oracle; FACT). Dialect rules:

  1. No backend intrinsic outside the prelude.
  2. No floating-point atomics. Integer atomics are allowed only for operations whose result does not depend on arrival order: maximum, minimum, bitwise or, and wrapping addition.
  3. No built-in transcendental functions and no reciprocal-square-root built-in. Elementary functions are det_* functions written in the dialect itself, ported from fs-math's deterministic set.
  4. Floating-point contraction off; no fast-math relaxations; no explicit fused multiply-add.
  5. Explicit loop bounds that are literals or capacity constants of the shape class, unsigned 32-bit indices, no recursion, no pointer arithmetic beyond buffer plus index.
  6. No grid-wide barrier. A stage that needs every world to finish before the next begins is two launches.
  7. Integer arithmetic in exact kernels is unsigned and 32 bits wide; shift counts are masked. Signed overflow is undefined in both source languages and is therefore unreachable by construction. Sixty-four-bit integers and atomics are used only where the fingerprint records them for every device family in use. A count that can pass 2³² follows the two-limb rule of §21.26.
  8. A mask is a select, never an arithmetic blend: m·a + (1 − m)·b turns an infinity in the unselected arm into a NaN.
  9. Padding trails the active lanes and holds the neutral element of the operation that will consume it (§35.5).

FACT: two habits of the suite's piano renderer are deliberately not inherited. It compiles its CUDA source with multiply-add fusion on and builds its double-float product on fmaf and fma (cuda.rs, metal.rs), and it steps a whole stretch in one persistent launch whose thread groups meet at an atomic grid barrier, which on Metal "very rarely" leaves one group unscheduled, "about once per few hundred launches", so the host reruns the stretch (metal.rs). Rules 4 and 6 remove both.

Arithmetic is derived, once. The mathematical body of a kernel (a Cayley step, an adjoint transport, a cone projection, a Jacobian row) is produced at build time by derive/frob-derive using FrankenSymPy, lowered to a program in frob-kir, and emitted as Rust and as dialect text, together with its derivative programs and its certificates (C-14). Generated source is checked in with its digest; CI regenerates and compares; generated code is never edited by hand. Derivation is per kernel body, never per whole robot, which keeps expressions small: a published attempt to expand a humanoid mass matrix symbolically ran for over an hour and crashed (FACT, Appendix H). If FrankenSymPy hangs on a derivation it runs in an isolated worker with a time and memory cap.

✖ KILL-KIR Trigger: at M2 the derivation toolchain cannot express or emit more than 20% of the kernel arithmetic, or emitted kernels miss BET-01's speed bound after one optimization pass. Fallback: arithmetic is hand-written in Rust and the dialect and bound by differential tests; certificates fall back per KILL-LEMMA. The skeleton pattern is unaffected.

21.7 The kernel IR

frob-kir is typed static single assignment with bounded loops. It merges two requirements: every value carries a physical dimension, and every program carries the resource facts a backend needs before it compiles anything.

DESIGN: frob.kir/1, intent not syntax.

types      f32 | f64 (CPU targets only) | f32dd (double-float pair) | u32 | bool
           vec3<T> | mat3<T> | se3<T> | sym6<T> | array<T, N>          N is a literal
           q16_16 | q32_32                                              reserved: parsed, never lowered to a GPU
dimension  every floating value has an exponent vector over the fs-qty base dimensions; sums must be homogeneous
statements single assignment
           for i in 0..N carry (a = a₀, …) { …; yield (a′, …) }       N a literal or a shape-class capacity; a value that crosses iterations is a loop argument, so every name is still assigned once
           select(pred, a, b)                                          branchless
operators  add sub mul div sqrt neg abs min max clip compare           min and max are defined on NaN and on ±0
           and or xor shl shr addw mulw mulhi                           u32, wrapping, shift counts masked
           load(slot, lane, offset) store(slot, lane, offset, v)       offset a literal
excluded   sin cos exp log pow rsqrt fma, any approximate mode
metadata   content hash (BLAKE3); math_mode in {Ieee754Safe, ExactInteger, FixedPoint (reserved)};
           max_regs; max_stack_bytes; operation counts by type; loop bounds; buffer argument count;
           class claimed; CPU reference symbol; derivative program; proof_digest (optional)
  • Dimension typing is T-09: a program type-checks only if every sum is homogeneous, decided exactly.
  • Bounded loops replace strict straight-line form where a body repeats over links at one depth, rows or lanes. A loop bound is a literal or a capacity, so every program still has a finite, known operation count and a derivative program by the same transformation.
  • Elementary functions are det_* subprograms written in the IR from +, −, ×, ÷, √ and selects, so they inline as ordinary arithmetic and inherit the class of those operators. The design needs few of them: a revolute coordinate is a unit complex number, and group updates are Cayley maps (§11, §13).
  • Rewrites allowed are those that are exact in IEEE arithmetic: bounded common-subexpression elimination, dead-code removal, commutation of + and ×, x·1, x + (−0), double negation, select simplification, and folding of exact constant operations. Reassociation and distribution are not rewrites; a Horner form is chosen at derivation time, before the program is fixed. Scheduling by live range to cap register pressure reorders independent operations only. Extraction over an e-graph, if the M2 profile ever asks for it, is restricted to the same exact rules under a node budget.
  • Resource metadata is a proof obligation, not a comment (PO-SPARK-03): the emitter refuses a program whose bounds exceed a backend limit, for example the 31 buffer arguments of a Metal compute function (§21.9).
  • Fixed-point types q16_16 and q32_32 are reserved words with defined wrapping and saturation semantics. No kernel in the library uses them through M5. They exist so that a later embedded target M (§31) lowers the same programs without a second IR; the exported artifact would carry the no-authorization flags of §12.

21.8 Kernel classes: floating, exact-integer and guarded

Class Arithmetic Used for Determinism
float f32 (and f32dd accumulators) under Ieee754Safe The physics step, sensing, learning D2 by design; D1 a HYPOTHESIS (§21.15)
exact Unsigned 32-bit integers under ExactInteger Strict predicates, digests, counts and scans, sort keys, histogram bins, random streams, phase accumulators D0 on every backend
guarded An f32 residual with a certified error bound, plus an exact fallback Strict predicates whose inputs live on the device as floats and whose cases are mostly far from the boundary D0 in its decisions
stat Any of the above, across worlds Statistics rings and histograms Never feeds state (INV-SPARK-04)

Why a class and not a habit. The smallest case that breaks a bare comparison: is 16,777,216 less than 16,777,217? In integers, yes. In f32 the second number is not representable and rounds to the first, so the strict comparison is false. A requirement verdict, a termination test or a region boundary that is decided this way is wrong exactly where it matters.

FACT (a recorded pilot, Appendix K; reproduced at GATE-SPARK-08 before any gate cites it): a one-inequality boundary classifier with inputs on a grid of 2⁻²⁸ ran as a real Metal command buffer on this M5 Ultra and as a CUDA kernel on each RTX 4090 of trj, with readback and independent integer checking. On a 65,536-row grid dense near the boundary, the bare f32 sign of the residual gave 452 false inclusions and 23,438 false exclusions, identically on every device. An exact unsigned-integer kernel returned every label correctly on all three GPUs with identical bytes. A guarded f32 kernel that deferred to exact arithmetic whenever the absolute residual was at most 2⁻²¹ (the conversion and addition sequence bounds the error by 5·2⁻²⁵) decided none of the dense grid on the device and 58,254 of 65,536 rows of a more widely spaced grid, with no error after fallback. The same pilot is a negative placement result: end to end, the host integer loop was faster than any device (about 34 µs against about 290 µs on Metal, of which under 13 µs was device time, and over 1.2 ms on one RTX 4090), and two-card sharding was slower than one card.

DESIGN rules that follow:

  1. Representation before placement. First choose a representation that preserves the strict inequality, then place the kernel where its data and its consumer already live.
  2. A guard is proved, not tuned (PO-SPARK-02). frob-derive bounds the rounding error ε of the residual from its actual operation sequence. If |r̂| > ε the sign of r̂ is the sign of the exact residual and the kernel decides; otherwise it emits the case to the exact kernel. The fallback fraction is recorded in the receipt.
  3. A small host-resident job stays on the host. The planner (§21.12) moves a predicate to a device only when its inputs are already resident there; otherwise dispatch overhead dominates.
  4. Digests are integer mixes. K-x-digest folds the raw bit patterns of the canonical state slabs of one world into four u32 words in slab order. A run digest is the wrapping sum of per-world digests keyed by address, which is independent of slot order. The device digest is a tripwire classed Tested; identities in the record (EpisodeId) are BLAKE3 hashes computed by the host over committed snapshots (§29).
  5. Random streams are exact. Philox is integer arithmetic, so a draw is D0 everywhere; the conversion u = (x ≫ 8)·2⁻²⁴ to a uniform f32 in [0, 1) is exact (§35.9).

21.9 Backends

KernelBackend is a core trait in frob-kernels: create buffers, compile a library, build a step plan, launch it, read back, and return a receipt.

  • CPU. batch-cpu is itself a backend whose kernels are Rust symbols, so one harness drives every tier. The strict batch-cpu profile of §16 (transactional, event-located) is a different step plan on the same backend.
  • CUDA (spark/frob-spark-cuda). cudarc pinned to an exact version with the driver, NVRTC and dynamic loading features, as in FrankenSim's render-gpu (FACT: cudarc = "=0.19.9" with features driver, nvrtc, dynamic-loading). The driver and NVRTC load at run time, so the build needs no CUDA toolkit. Dialect source compiles per device architecture to a cached binary; frob spark precompile sm_89 fills the cache on a host without a GPU (the same recipe as render-gpu --precompile, FACT). Compile flags disable multiply-add fusion. Each device has one context and one host thread. With two RTX 4090s, worlds are sharded across devices and only small statistics cross through the host: the cards have no NVLink (FACT: the driver on trj reports a PCI host bridge between two cards of 24,564 MiB each) and peer-to-peer transfer is unverified (OPEN-SPARK-6).
  • Metal (spark/frob-spark-metal). FrankenTorch's ft_kernel_metal::compute::Gateway, described below.
  • A third backend through wgpu F is possible for other hardware because the dialect is small, but it is not planned and is not admitted as a dependency.

CUDA mechanisms (DESIGN).

  • Launch graphs. The step plan of a bucket is captured once as a launch graph spanning one control interval. Published measurements of another simulator show throughput falling to roughly a quarter to a third when the graph is re-captured every step (FACT, Appendix H), so nothing in a rollout may change a shape. Replaying a warm graph has near-constant host cost, independent of its node count (HYPOTHESIS; measured by the fingerprint, §21.16).
  • A control block. Everything that varies between replays of the same graph (step counter for random streams, epoch, cancel word, decimation phase) lives in a small device buffer that the first kernel of the graph advances. The graph's arguments never change, so it is never re-captured.
  • Stream-ordered pools. The arena is one allocation made at environment creation. Transient device memory (snapshot staging, readback) comes from a stream-ordered pool, so the allocations of a cancelled stream are released in stream order and cannot leak. If the pinned cudarc lacks a binding for graph capture or for the asynchronous pool calls, the capsule declares the missing driver symbols itself under its SAFETY.md; if that is refused, transient buffers are preallocated in the arena and the hot path is unchanged (OPEN-SPARK-4).
  • No synchronization inside a control interval. The host synchronizes once per chunk (§21.11).

The Metal gateway as it exists. FACT, read from frankentorch/crates/ft-kernel-metal version 0.1.3 (src/compute.rs, src/lib.rs): Gateway::open returns the process-wide device and command queue; library compiles Metal Shading Language under MathMode::Safe, which disables fast math (library_with selects otherwise); Library::pipeline builds a compute pipeline and exposes max_threads_per_threadgroup and thread_execution_width; buffers are StorageModeShared allocations made by buffer_f32, buffer_u32 and buffer_zeroed, read by read_f32, read_u32 and read_u8 and rewritten in place by write_f32 and write_u32; Gateway::batch encodes several kernels in order into one serial compute encoder of one command buffer, checks read and write declarations for aliases, borrows every output exclusively until submit returns, and returns a Receipt with the kernel count and the device's execution time. The gateway also reports max_buffer_length, recommended_working_set_bytes, allocated_bytes and max_threadgroup_memory. State is resident: a buffer persists across command buffers.

Limit today (FACT) Consequence for Spark (DESIGN) Upstream work item
Completion is synchronous: dispatch and submit block until the command buffer finishes One Batch per chunk of control intervals; one blocked host thread per device; host work overlaps device work only on other threads FD-FT-01
At most 31 buffer arguments per kernel (MAX_ARGUMENTS) Arrays are packed into a few slabs; kernels index through an offset table in a parameter buffer, as the piano renderer's P[P_*] slots do FD-FT-01
Buffers bind at offset zero Slab-relative indexing inside kernels; no sub-buffer views FD-FT-01
Shared storage only; no private storage, blit or heap Readback is a host copy after completion none needed
Host data is copied in and out; typed helpers for f32, u32 and bytes out Scene fields and tables upload once; 16-bit fields are packed into u32 words; rings are read by copy FD-FT-01
Compilation from source only; no function constants; no binary archive Shape-class constants are #define lines prepended to the dialect text; a warm start recompiles from cached source FD-FT-01
One device and queue per process; the queue holds 64 command buffers One Metal worker per process; no uncommitted command buffer is ever held none needed
The wait cannot be interrupted A watchdog marks the device stalled and the supervisor restarts the worker FD-FT-01
A failed batch is reported and not re-run; its outputs are undefined The commit fence of rule R2 below none needed
The receipt has a kernel count and device seconds only Per-kernel timing needs one batch per kernel in profiling mode FD-FT-01
The metal 0.33 binding is deprecated by its maintainers Pinned as FrankenTorch pins it; the migration is part of FD-FT-01 FD-FT-01

Until the upstream work of FD-FT-01 lands in FrankenTorch's repository (Appendix F), Spark works inside these limits. It does not patch the gateway locally and does not bind Metal a second time beside it (OPEN-SPARK-5).

Metal readback rules (DESIGN).

  • R1. The host reads a device slab only after the command buffer that last wrote it has completed and returned its receipt. Under the gateway today that is a compile-time fact, since a batch holds each output by exclusive borrow until submit returns (FACT: compile-fail tests in compute.rs).
  • R2. A terminal state other than Completed invalidates every slab the batch could write. This is not hypothetical: "when another process hangs the GPU, macOS restarts it and aborts every command buffer in flight" (FACT: lib.rs). The unit of commit is therefore the chunk, which is one command buffer. A chunk reads its checkpointable state from the current one of two committed slabs and ends with K-commit, which writes the new state to the other. Only a Completed receipt makes the other slab current. On failure the current slab was never written, so the arena restores from it and resubmits, at most four attempts with growing waits, then refuses with FROB-REF-607. Every retry is counted in the receipt. A replay from the committed slab is bit-identical because no kernel has data-dependent scheduling.
  • R3. All storage is shared today, so no blit and no managed-resource synchronization exist to be forgotten. When private storage arrives through FD-FT-01, results are blitted to a shared snapshot slab inside the same command buffer and read under R1; managed storage is not used.
  • R4. A run is labeled native on a device only if its receipt carries a completed command buffer with start and end times from that device (FACT: Receipt.device_seconds spans Metal's GPUStartTime to GPUEndTime).
  • R5. Timers cover completion, never enqueue alone.
  • R6. The device's thread execution width is read from the pipeline, never assumed, and thread-group sizes are multiples of it.

21.10 The device arena, leases and the binary kernel cache

Arena (DESIGN). Every per-world quantity lives in preallocated structure-of-arrays slabs aligned to 256 bytes: state (configuration, velocity, anchors, warm starts, actuator and battery state), kinematic and dynamic cache, rows, contacts, variant and scene tables, rollout storage, policy weights and optimizer state, the statistics ring, the two committed slabs, the snapshot slab and the control block. Slabs are packed into few device buffers so that every kernel stays under the argument limit, and no slab exceeds the device's largest buffer (§36.10). After environment creation there is no device allocation (INV-SPARK-03).

Leases. A slab is touched by the host only under an epoch-fenced lease:

// DESIGN: intent, not signatures.
enum Access { Device, HostRead, HostWrite }
struct SlabLease { slab: SlabId, epoch: u64, access: Access }   // valid only while `epoch` is the committed epoch

A host-read lease exists only on the current committed slab, the snapshot slab and the previous slot of the double-buffered ring, so diagnostics read epoch e − 1 while the device writes epoch e. A host-write lease (resets by the host, admission of a model variant into a free slot) exists only between chunks. On Metal the lease is the borrow the gateway already enforces; on CUDA it is a runtime epoch check around stream-ordered copies. A violation is FROB-REF-614. spark.place prints the residency map: where each slab lives, its size, its last committed epoch and who may read it.

Binary kernel cache. FACT: the piano renderer caches CUBINs under a key built from Rust's default hasher over source and options plus the architecture name, written aside and renamed (cuda.rs); a compile takes about 5 s on an idle host and can take 15 minutes on a heavily loaded one (render-gpu/README.md). DESIGN:

  • The key is BLAKE3 over the expanded dialect text, compile options, prelude version, compiler identity, device architecture and driver version. The default hasher is not stable across toolchains and is not used.
  • The value is a CUBIN on CUDA. On Metal it is the expanded source and its digest, because the gateway compiles from source only; a precompiled archive is part of upstream work item FD-FT-01.
  • A hit is checked against its digest before use; a corrupt entry is a miss. An offline host that cannot recompile refuses with FROB-REF-612.
  • The directory is content-addressed and may be copied between hosts. Its size is capped (4 GiB by default, DESIGN); eviction prefers entries that are cheap to recompile.
  • A change of compiler or driver changes the key, so the old binary is never loaded, and it also marks the CPU-versus-device check stale (§21.13).

21.11 Execution model: shape classes, variants, buckets and the step plan

Environment creation allocates every buffer, compiles or loads every kernel, and builds a static step plan: an ordered list of launches with fixed grid sizes. A rollout is policy forward, then the step plan repeated for the control decimation, then observation, all on the device. The host receives a small statistics ring and snapshots on request.

Shape class and bucket (DESIGN).

// DESIGN: intent, not signatures. This is the device-side view of §17.4's `ShapeClass`: the `_cap` fields are its
// capacities, the fields after `sensor_layout` are device-side keys, and `WorldRow` is its `WorldSlots` with
// bucket-local indices. Appendix I holds the one definition.
struct ShapeClass {                 // content-hashed; one kernel set and one launch graph per class per device
    nv_cap: u16,                    // degrees of freedom: one of 12, 18, 24, 36, 48, 64 by default (§17.5)
    link_cap: u16, depth_cap: u8,
    contact_cap: u16, row_cap: u16, sample_cap: u16, actuator_cap: u16,
    sensor_layout: LayoutId, policy_layout: LayoutId,
    contact_law: LawId, precision: PrecisionProfile, iterations: u8,
}
struct Bucket { class: ShapeClass, device: DeviceId, model_variants: u16, scene_variants: u16, worlds: u32 }
struct WorldRow { model_variant: u16, scene_variant: u16, task: u16, job: u16, addr: [u32; 4] }   // 8 + 16 bytes

A bucket is one shape class on one device: one arena, one compiled kernel set, one step plan, one launch graph. A world is a row of indices into the bucket's tables. Three kinds of variation ride on that row:

  1. Parameter variants S. Worlds of one topology carry different masses, lengths, gains, card parameters and contact parameters. This is parity with MuJoCo Warp, which already batches model fields per world (FACT, Appendix H), and is stated here as a requirement: no kernel compile and no graph capture per design.
  2. Topology variants in one bucket F. A variant table holds, per link, the parent index, depth, joint class as a column mask over a 6 × 3 screw basis (so one-, two- and three-axis joints are data), placement, ten-number inertia, limits and an actuator-card slot, plus the start of each depth level. Padding links are identity transforms with zero inertia at trailing indices. Arithmetic: about 192 bytes per link, so a 64-link variant is about 12 kB and 256 variants about 3 MB (DESIGN estimate). Admitting a new variant is a host write of those rows into a free slot between chunks.
  3. Design-family buckets F. A family (§17) is packed so that constants shared by its members are stored once and referenced by index: tables of unchanged rigid sub-assemblies, actuator cards, collision geometry. Continuous parameters occupy columns. Sharing is by content hash of design constants only. State-dependent work is never shared between designs on the throughput tiers: exact sharing would require identical inputs and identical rounding, and reuse of a nearby trajectory would need its own error certificate.

Layout rules. Worlds of one variant are contiguous and allocated in multiples of the device's execution width (32 on both device families today; FACT for CUDA warps, and the gateway's own note for Apple silicon), so that lanes of one wave share control flow. Padding obeys §35.5, which makes a world's result independent of the capacity of the bucket it ran in. Masks apply to every kernel, to rewards and to gradient accumulation: a padded contact that stays active would change the physics.

How many buckets. The cost of padding a batch to its largest member, when kernels do full work on padding, is Ω_p = L_max^p / E[L^p] for work that grows as the p-th power of size L, and 1/r ≤ Ω_p ≤ r⁻ᵖ where r is the mean size over the maximum. Worked values: degrees of freedom 18 and 35 mixed evenly give 1.3, 1.6 and 1.8 for p = 1, 2, 3; mixed 90 to 10 they give 1.8, 3.0 and 4.5. With the default capacities the worst member of a bucket sits just above the previous capacity (25 in the 36 bucket), for Ω₁ ≤ 1.44 and Ω₃ ≤ 3.0. Masked lanes that exit early pay less than this bound: Newton's Kamino solver runs a mixed batch of three body plans at 0.86 of the harmonic mean of its homogeneous rates (FACT, Appendix H). An extra bucket costs one more step plan per step. If a step plan costs L seconds of launch overhead and a world-step costs t, splitting N worlds off pays when N·(1 − 1/Ω)·t > L. HYPOTHESIS: with about 60 launches of 5 to 10 µs each and t = 0.5 µs, the break-even at Ω = 1.3 is 2,600 to 5,200 worlds; the launch cost is measured by the fingerprint and the bucket set is frozen at M2 (OPEN-SPARK-3). The bound on what family batching can deliver end to end is in §36.5.

✖ KILL-BUCKET Trigger: on the reference family corpus more than eight buckets are needed to hold Ω₃ ≤ 2, or the mixed batch of GATE-SPARK-10 runs below 0.5 of the harmonic mean of its homogeneous rates after one optimization pass. Fallback: at most nine padded per-family templates, each a homogeneous environment, scheduled in turn; parameter variants are unaffected. The head-to-head experiment that decides this, with the per-family-template control arm, is specified in §17.14. §17 owns the decision: KILL-PHYLUM (§17.19) governs where the two differ, and the mixed-batch clause of GATE-SPARK-10 is the kernel-level precondition of GATE-PHYLUM-03.

Persistent environments and slot dispatch F. The unit of work is a world, not a job. One environment per device and shape class stays resident in the daemon. A slot whose episode ends pulls its next episode from any admitted job at the masked reset, by writing a new WorldRow. Results are keyed by world address, never by slot or arrival, and dispatch is a pure function of queue state, so a job's digests are the same alone and in any co-tenancy (§35.5). No shape change enters a running environment (FROB-REF-611). Why it matters: a 96-pair comparison fills 192 of 8,192 slots, 2.3% of a batch; without sharing, the rest idles. Ceiling, stated before any test: the gain is at most the reciprocal of the mean fill fraction of per-job batches, and is nothing for jobs that already fill a batch. Defaults (DESIGN): at most 64 jobs per environment and 5% of slots reserved for short jobs. Pilot before the work package is funded: a queue trace replayed on batch-cpu with and without sharing. The dispatch policy belongs to the daemon (§32).

✖ KILL-SLOT Owned by §32.21, which governs. Trigger: any digest difference under co-tenancy, or the throughput or step-time limits of GATE-TETHER-09 fail after one optimization pass. GATE-SPARK-15 tests the same digest equality at kernel level and is a precondition, not a second trigger. Fallback: one environment per job, queued (frobd --dispatch=per-job).

Chunks, cancellation and faults.

  • Chunks. The host synchronizes once per chunk of control intervals. The chunk length is the shorter of one rollout segment and the number of intervals that fit the cancellation latency target (250 ms at the 99th percentile, TARGET).
  • Cancellation is at the kernel-batch boundary: request, drain, finalize, with an executor-minted report (AE-10). A dropped handle is not a cancellation receipt.
  • Faults. Device loss, out-of-memory, compile failure and a per-world non-finite sentinel are typed refusals (FROB-REF-6xx, §21.21). A world that produces a non-finite value is frozen and marked Faulted for teleport. A world that overflows a capacity is marked Degraded with the reason, and the caller's policy chooses refuse, reset or continue-flagged.
  • Stalls. The barrier that stalls the suite's piano renderer is excluded by dialect rule 6. What remains is a command buffer that fails (rule R2) or never returns; for the latter a watchdog on a second thread marks the device Stalled (FROB-REF-615) when a chunk exceeds the upper end of its predicted time by a set factor, and the supervisor restarts the device worker from the last committed epoch. ft-kernel-metal has a fault-injection seam for the first case (FACT: inject_command_buffer_faults, present in builds with debug assertions).

21.12 The placement planner and its receipts

spark.place turns a request into a plan before anything runs. For each stage of the batched step (§16), for the policy, the learner, the statistics and the snapshot path, it chooses a backend and writes down why.

DESIGN: frob.placement/1 (one row per stage)
  stage, kernel ids, requested backend, actual backend, fallback reason, fallback count,
  transfer bytes host→device and device→host, synchronization count, resident duration,
  device seconds, command buffers or graph replays, retries
header: request mode (strict | prefer | mixed-allowed), fingerprint digest, shape class, bucket, capacities,
        determinism class attained, check currency per kernel

Rules (DESIGN):

  1. Strict means strict. Under strict:gpu-cuda or strict:gpu-metal, a stage with no kernel on that backend, a stale check under --require-current-check, or a missing arithmetic class is an error (FROB-REF-604, -605, -616) raised by the planner. No world is stepped and nothing is scored.
  2. Mixed placement is opt-in. Only mixed-allowed lets a stage fall back to batch-cpu, and the receipt then shows actual differing from requested, the reason, and the bytes and synchronizations it cost. prefer never mixes: if the named backend cannot run every stage, the whole plan moves to one backend that can, and the plan says so before the run starts.
  3. The denominator is whole. Transfers, synchronizations, compilation and retries are in the receipt and in every throughput figure derived from it (§36.8).
  4. A cost model decides small jobs. A job goes to a device only if its predicted device time including dispatch, transfer and synchronization is below its host time. The crossover sizes are calibrated per device family from the fingerprint. The pilot of §21.8 is the cautionary case: under 13 µs of device time inside about 290 µs end to end.
  5. Each device closes its own loop. A hybrid plan may claim acceleration only for stages that ran on the device and returned a device-timed receipt (rule R4).

21.13 The CPU-versus-device check

frob spark check runs each kernel, and the full step, on batch-cpu in f32 and on the device from identical inputs, compares them under the kernel's determinism class, and writes the result to the divergence ledger. A device result is not citable until its kernels have a current check on that device family, driver and compiler (AE-08). The check runs again whenever any of those change, which the fingerprint digest detects.

Additions (DESIGN):

  • The ledger stores, per kernel and per pair of placements, the distribution of disagreement in units in the last place as an integer histogram (§35.7), the worst case with its input, and the class attained. This is the stored disagreement record of §21.2.
  • For a class D3 pair the check also runs the decision-level equivalence test of §35.4 on the full step.
  • The check varies what must not matter: batch size, slot, bucket capacity, co-tenant variants and shard count (§35.5).
  • A first divergence is located by bisection over the digest chain and recorded as step, kernel and quantity.
  • Exact and guarded kernels are compared for byte equality on every backend, never under a tolerance.

21.14 Precision

Single precision on devices: Metal has no double type, and an RTX 4090 runs double precision at about 1/64 of its single-precision rate (FACT: render-gpu/README.md). Four measures keep single precision adequate (BET-08): anchored positions (§35.8), group-preserving updates with per-step normalization (C-05), fixed-order pairwise or compensated sums, and a contact solve carried out in power-of-two-scaled variables with a reported conditioning figure (§35.10).

Accumulators that need more (time offsets, odometry, the energy ledger) use a double-float pair type in the dialect, as render-gpu-piano does (FACT). Unlike that renderer, the product does not use a fused multiply-add (ADR-SPARK-10): it uses the split product, with the splitter 4097 = 2¹² + 1 for f32, at 17 operations instead of 3. Two start-up canaries (K-canary) prove on each device that the compiler kept the transformations:

  • Sum: the error-free sum of 1 and 2⁻³⁰ must return the pair (1, 2⁻³⁰). A compiler that simplifies the error term algebraically returns (1, 0).
  • Product: the error-free product of 1 + 2⁻¹² with itself must return (1 + 2⁻¹¹, 2⁻²⁴), since the exact product 1 + 2⁻¹¹ + 2⁻²⁴ is a tie that rounds to even.

A failed canary refuses the device for double-float kernels (FROB-REF-609).

Half precision is not used anywhere in the step; the calculation that rules it out is in §21.20.

21.15 Determinism on devices

Run-to-run determinism per device family is a design requirement, achieved by the dialect rules, sort-based compaction and fixed-order reductions. Newton 1.8 and Isaac Lab 3.0 on PhysX document opt-in bitwise run-to-run determinism for the same batch on the same GPU (FACT, Appendix H), so that property is parity. Two stronger properties are documented in no incumbent read, and each is claimed here only after its gate:

  • A world's trajectory does not depend on the batch it ran in (§35.5, GATE-SPARK-15). Newton's own solver tests hold a world equal to its isolated run only to 10⁻⁵ to 10⁻⁴ (FACT, Appendix H).
  • Bit equality across CPU, CUDA and Metal for floating kernels. This is a HYPOTHESIS. IEEE 754 requires correct rounding of +, −, ×, ÷ and √, CUDA documents compliance, but Apple confirmed in September 2026 that Metal offers no control over rounding mode or subnormal handling and the language specification permits flushing subnormals (FACT, Appendix H). GATE-SPARK-01 measures it instead of assuming it.

Exact and guarded kernels do not share that uncertainty: their decisions are integers and are compared for byte equality on every backend (GATE-SPARK-08).

✖ KILL-D1 Trigger: GATE-SPARK-01 or GATE-SPARK-02 shows that a device family cannot hold bit equality with batch-cpu for the floating kernels of the step. Fallback: that family is class D2 for itself and class D3 against others, with the decision-level test of §35.4; a teleported world is labeled a tolerance replay; a comparison whose arms ran on different families is unpaired (§35.3). The wording of the replay claim in §6 and §29 changes with it.

21.16 The hardware fingerprint

Every figure in this chapter that depends on a machine is measured once per machine state and bound to a digest. spark.fingerprint writes the record; spark.devices lists devices with their current digests.

DESIGN: frob.fingerprint/1
  machine   host, operating-system build, CPU model, performance and efficiency core counts, memory
  device[]  family (cuda | metal), name, stable identifier, architecture, driver, runtime, compiler
            memory: total, free at probe, working-set limit, largest single buffer
            execution width; multiprocessor or GPU-core count
            arithmetic: IEEE probe digest and attainable class; subnormals (ieee | flush | mixed);
                        64-bit integers (yes | no); 64-bit atomics (yes | no); canaries (pass | fail)
            bandwidth: measured GB/s of a stream copy at three sizes around the cache boundary
            launch: empty-kernel launch, graph replay against node count, command-buffer commit and wait,
                    per-dispatch encode cost; each as median and 99th percentile
            compile: cold seconds for K-cay-step; cache-hit milliseconds
            peer: device-to-device access (yes | no | untested); measured rate through the host
            thermal: state at probe and after a 20-minute soak
  digest    BLAKE3 over the canonical record

FACT, as starting values the fingerprint must reproduce: this M5 Ultra has 36 CPU cores, an 80-core GPU and 256 GB of unified memory; its Metal device reports a working-set limit of about 222.7 GiB and a largest single buffer of about 167.0 GiB; trj has two RTX 4090 of 24 GB with no link between them. 64-bit atomics are reported absent on Metal, to be re-run by the fingerprint (§21.26, §33.17). Everything else about the machines (bandwidth, launch costs, peer access) is unmeasured and is not assumed anywhere in this plan.

The digest keys the kernel cache, the currency of spark check, the calibration of sim.estimate and the disclosure block of §36.7. A receipt without a fingerprint digest, or with one that no longer matches the machine, is refused as evidence (FROB-REF-606).

21.17 Division of labour between the machines, and the fleet scheduler

Policy S. The two machines are not a fast GPU and a slow GPU. They have different jobs.

Work This M5 Ultra trj
World bank: many scene variants resident; 2 cm warehouses and buildings; 1 cm interiors (§36.10) Yes Only the variants of the running job
Training specialists, family policies and long runs Small and medium runs on gpu-metal; development The training tier, one shard per card
Evaluation on large scenes; camera-heavy worlds (§36.12) Yes Low resolution only
Referee, batch-cpu shadowing, replay and teleport targets Yes, 36 cores Its CPU is often heavily loaded (FACT)
Derivation, certificates, proof and symbolic work Yes No
Viewer, stills, the dashboard Yes No
Large-memory jobs of any kind Yes No

What crosses. Seeds and scene recipes (bytes to kilobytes), design and module differences (kilobytes), model-variant tables (kilobytes to megabytes), policy weights (megabytes), receipts, digests and telemetry summaries. What never crosses: compiled scene fields, world state, rollout buffers. A scene is regenerated on the receiving machine from its seed with the pinned generator, and must arrive at the same SceneId (GATE-HAB-02 already requires equality across Linux and macOS). A world that failed on trj is replayed on this Mac from its address, its design, its policy weights and its scene recipe; nothing else is needed, and the replay is exact or a tolerance replay according to the class attained (§35.3). TARGET: at most 1 GB crosses per campaign-hour, metered per transfer class; an attempt to ship a field or a state slab is FROB-REF-618. A comparison whose arms ran on different device families is unpaired unless class D1 holds (§35.3).

Fleet scheduler F, outline. Owner: the daemon (§32); Spark supplies the resource model.

  • A job carries a resource vector (device family or any, memory, shape class, determinism class required, envelope class of §36.14), a role tag, and the cost interval of sim.estimate.
  • One queue per device. fleet.plan assigns by role affinity first, then shortest estimated work first with aging so that long jobs cannot starve. There is no adaptive controller beyond that.
  • Job specifications and results are content-addressed records; an idempotency key makes a re-sent job a no-op.
  • If a machine drops out, its jobs re-queue on the other only where role and class allow; otherwise they wait.
  • Balance uses measured queue length and estimate error, nothing more elaborate for two nodes.

Ceiling, stated before any test. Adding this Mac's GPU to training adds at most its share of the pair of cards: 0.125 at BET-09's threshold of 25% of one RTX 4090, 0.30 at the stretch of 60%. So the gain of a fleet cannot come from training throughput. It can come only from moving non-training work off the training cards, and removing a fraction g of the work on the critical machine speeds a campaign by at most 1/(1 − g): a factor of 1.5 needs g ≥ 1/3. The pilot is therefore a phase ledger of one campaign on trj alone. If non-training work is under a third of its wall-clock, the scheduler is not funded and the policy table above is applied by hand.

✖ KILL-FLEET Trigger: the pilot shows g < 1/3, or after one repair cycle the paired lower bound of campaign speedup at GATE-SPARK-13 is still below 1.3 against the better single machine. Fallback: the machines run standalone; jobs are assigned by the role table through ordinary job.* verbs; the seeds-only rule stays.

21.18 Specialization for survivors only; the fused step kernel

Specialized kernels F. A specialized kernel is the generic kernel with one design's constants folded in and its loops unrolled. Ceiling, stated first. FACT (MuJoCo Warp's own event trace of a humanoid at 8,192 worlds, Appendix H): of 262 ns per world-step, the stages a design-specialized recursion could touch (kinematics, centre-of-mass terms, composite inertia, bias forces, forward acceleration) sum to 53 ns, 20% of the step; the solve is 124 ns and constraint construction 38 ns. An infinitely fast recursion would give 262/209 = 1.25; a recursion twice as fast gives 1.11; a manipulation scene gives a share of 16% and a ceiling of 1.19. Generated code also does not reliably beat a good generic library (FACT, Appendix H).

DESIGN:

  • Specialization is never the default and never part of the edit loop.
  • It is built only for the survivors of a tournament that go on to finalist training, and only if our own M2 profile shows a specializable share of at least one third of the step, which is the share at which the ceiling reaches 1.5.
  • A specialized kernel must be bit-identical to its generic kernel: unrolling keeps the operation order and only exact constant operations are folded (§21.7). spark check enforces it; a mismatch is FROB-REF-613.
  • Survivor cohorts move to specialized kernels only at an acknowledged chunk boundary, with a versioned state map, and the specialized kernel is another cache entry.
  • Pilot before the work package is funded: three arms on one 12-joint quadruped at 8,192 worlds, measuring whole-step time and compile time: (a) the padded generic kernel, (b) the unpadded data-driven kernel, (c) a fully unrolled, constant-folded kernel written by script.
✖ KILL-SPECIALIZE Trigger: in the pilot (c) over (b) is below 1.2 on the whole step, or (c) takes more than 60 s to compile; or the M2 profile shows a specializable share under one third. Fallback: data-driven kernels only. Derivation stays for correctness, certificates and the derivative kernels of CPU solvers, where rigid-body derivatives dominate.

Fused per-world step kernel M. One persistent kernel per world group would keep a world's working set in thread-group memory for a whole control interval and touch device memory only at its ends, in the manner of the piano renderer's persistent launch (FACT). The lever is memory traffic (§36.11). Ceiling, stated first: the renderer itself refuses a block that needs more than 48 KiB of shared memory (FACT: cuda.rs), and one quadruped world's working set is about 35 kB (§36.10), so a block holds one world, and occupancy is expected to fall to a few percent of the device's lanes (HYPOTHESIS). The benefit is therefore unproven in sign. It is behind a flag, piloted on CUDA first (on Metal the thread-group memory limit is read from the device), and off every critical path.

✖ KILL-MEGA Pilot: within one week, fuse kinematics, dynamics terms and one Newton iteration for the reference quadruped and compare with the staged kernels at 8,192 worlds, digests unchanged. Trigger: paired lower bound below 2.0 on the whole step. Fallback: the staged step plan, which is the design of record.

21.19 Routes considered and not taken

FACT from the research pass, with the decision:

Route State in late 2026 Decision
wgpu as the only layer 30.x; portable; hides launch graphs, allocators and barriers; no doubles on Metal Not the primary; possible third backend; not admitted
Slang as the kernel language CUDA target supported; Metal and WGSL targets experimental Revisit when Metal is no longer experimental
NVIDIA's Rust-to-PTX compiler Early alpha, Linux and NVIDIA only Watch
rust-gpu, CubeCL and Burn Alpha or a large dependency tree Conflicts with the Franken-only core
Python kernel compilers of incumbent simulators (one source lowered to CPU and CUDA, or to CUDA, Metal and others) Mature; the reason single-source kernels are prior art Design references; not dependencies
Wrapping MuJoCo Warp, Newton or Genesis Python ecosystems; Newton is CPU-only on macOS; Genesis has a Metal path through its own compiler Anti-goal; used as oracles, as parity baselines (§36) and as design references
An IR with built-in sine and cosine, fusion of multiply-adds and an approximate mode Convenient; incompatible with a three-way parity gate Not taken (ADR-SPARK-02)
Our own Metal binding beside the gateway Possible (the suite's piano renderer binds metal directly, FACT) Not taken by default; OPEN-SPARK-5

21.20 Rejected mechanisms, with the calculation that refutes each

Idea Refuting calculation Used instead
Factor the contact Hessian in half precision on tensor cores and refine in single precision Half precision has unit round-off 2⁻¹¹ ≈ 4.9·10⁻⁴; refinement needs κ·u < 1. On contact states of three robots, after unit-diagonal scaling, κ·u ≥ 1 in 37 to 64% of states and refinement fails outright in 9 to 18% (FACT, recorded measurement, Appendix H). A rank-one worst case with the proposed shift has κ = 1 + 48/0.01 = 4,801, so κ·u ≈ 2.3. Each refinement step costs about 3n² operations against n³/3 for the factorization it replaces, a ratio of 9k/n for k steps: 2.3 at n = 27 and k = 7, 1.5 at n = 35 and k = 6. The published four- to five-fold gain is against double precision at n = 22,000; Apple GPUs have no such units Single precision in scaled variables (§35.10). The factorization is 1,944 operations at n = 18 and 14,292 at n = 35, under a tenth of the step
Copy-on-write forks of device world state State is write-dense: every step rewrites every world's configuration, velocity, contacts and warm starts, so a fork has copied everything after one step and has paid for indirection in every read. The whole copy is small: 8,192 worlds at 30 kB are 0.25 GB, about half a millisecond of read and write at an RTX 4090's memory bandwidth (§36.11), an eighth of one batch step at 2.0 M world-steps/s Fork by indexed copy of the chosen rows into free slots (K-snapshot); warm template environments stay resident (§27, §29)
Minimal perfect hashing or succinct structures for brick directories A dense directory of 8³-voxel bricks with 4-byte slots is 63 × 63 × 19 = 75,411 slots, 0.3 MB, for a 10 × 10 × 3 m interior at 2 cm; 587,500 slots, 2.3 MB, for a 60 × 40 × 8 m warehouse at 4 cm; 18.8 MB for that warehouse at 2 cm. That is under 1.5% of the scene it indexes. A hash adds a second, verifying read to every lookup A dense directory per scene variant over a shared brick pool (§19)
Mergeable quantile sketches whose result depends on insertion order A two-centroid sketch that merges its closest pair, fed {0, 3, 5, 9}: in the order 0, 3, 5, 9 it ends as (8/3 with weight 3; 9 with weight 1); in the order 0, 9, 5, 3 it ends as (1.5 with weight 2; 7 with weight 2). Same data, different sketch, so the statistic depends on shard count and arrival Integer log-bin histograms (K-x-hist): bins from the bit pattern, integer counts, merge by addition (§35.7)
Live zero-copy CPU reads of GPU state on unified memory Shared storage must be quiescent before the other processor reads it (FACT: Apple's documentation); and the data are tiny: a humanoid's integration state is about 540 bytes, so shadowing 256 worlds at 50 Hz moves 6.9 MB/s Fenced snapshots (rule R1) on any backend
Per-design kernels as the default path Ceiling 1.19 to 1.25 on MuJoCo Warp's trace (§21.18); its cold kernel compilation takes about 96 s (FACT, Appendix H) Model variants as data (§21.11)
Unordered floating-point scatter, accepted if "99.9% of windows" replay bitwise 0.999¹⁰⁰⁰ = 0.37: a campaign of a thousand windows would replay exactly about one time in three Stable-key sort and segmented reduction (§35.7)
Tensor-core products for a 256-wide policy The learner and inference together are about 13% of training wall-clock on one incumbent stack, so their removal gives at most 1.15 (§36.4); reduced precision leaves class D2 Plain f32 products in the dialect; device matrix paths only inside the labeled D4 profile (§35.2)
Per-world adaptive time tiles between interactions Separate environments are already independent; policy and sensor frames at 50 to 100 Hz bound a tile to 5 to 20 steps; shapes become dynamic Fixed decimation in a static graph
Full-physics sampling control as the default evaluator At the smallest published setting (30 samples, 3 iterations, 0.9 s horizon, 50 Hz) a simulated second costs 2.0 M world-steps against 500 for a policy, a factor of 4·10³: a battery of 256 twenty-second episodes takes about 86 minutes of a device at 2.0 M world-steps/s against 1.3 s A sampling controller for single worlds and as a fallback (§22); never a stand-in evaluator for a trained policy (§22.8)

21.21 Refusals

Backend and device refusals take the 6xx family (§35.11). Each carries the fingerprint digest and ranked repairs.

Code Condition First repair offered
FROB-REF-601 Device unavailable or lost Rerun on another device, or on batch-cpu under mixed-allowed
FROB-REF-602 The memory estimate exceeds the device budget at admission Fewer worlds, a smaller capacity, or this Mac
FROB-REF-603 Kernel compile failure; compiler diagnostics verbatim Report; the generic kernel if a specialized one failed
FROB-REF-604 Strict backend: a stage has no kernel on the requested backend mixed-allowed, or the other backend
FROB-REF-605 The CPU-versus-device check is not current for a kernel in the plan spark check
FROB-REF-606 Fingerprint missing, or changed since admission spark fingerprint
FROB-REF-607 A launch or command buffer failed after bounded retries Resume from the last committed epoch
FROB-REF-608 Worlds marked Faulted exceed the caller's threshold Teleport a faulted world to batch-cpu
FROB-REF-609 A double-float canary failed Run without double-float kernels, labeled; file the compiler evidence
FROB-REF-610 Capacity overflow under the refuse policy A larger capacity; continue-flagged
FROB-REF-611 A shape change was requested inside a running environment A new bucket
FROB-REF-612 Kernel cache entry corrupt and recompilation unavailable spark precompile on a build host
FROB-REF-613 A specialized kernel is not bit-identical to its generic kernel The generic kernel
FROB-REF-614 Host access to a slab outside its lease Read the committed or snapshot slab
FROB-REF-615 Device stalled (watchdog) Restart the device worker
FROB-REF-616 The device lacks an arithmetic feature the kernel declares (for example 64-bit integers) The 32-bit variant, or another backend
FROB-REF-617 A view cannot cross a stride convention without a copy, under a zero-copy contract Allow the copy
FROB-REF-618 A transfer class that may not cross machines Send the seed or recipe

21.22 Invariants and proof obligations

ID Statement Verification
INV-SPARK-01 The core workspace never depends on a GPU crate Dependency lint in xtask
INV-SPARK-02 Every device result carries AE-01 to AE-12, a placement receipt and a fingerprint digest Schema check at the agent boundary
INV-SPARK-03 No device allocation after environment creation Allocated-bytes counter flat over 10⁶ steps (GATE-SPARK-11)
INV-SPARK-04 No kernel reads or writes another world's rows, except stat kernels, whose outputs never feed state Footprints in manifests; GATE-SPARK-15
INV-SPARK-05 No shape change inside a running environment; one graph capture per bucket Capture counter (GATE-SPARK-12)
INV-SPARK-06 The host reads only committed epochs Compile-fail tests on Metal; epoch check on CUDA
INV-SPARK-07 A strict request never scores on a fallback GATE-SPARK-09
INV-SPARK-08 Generated code is never edited by hand; CI regenerates and compares digests check-generated
INV-SPARK-09 A specialized kernel is bit-identical to its generic kernel spark check
INV-SPARK-10 Only seeds, recipes, differences, variant tables, weights, receipts and summaries cross machines Transfer-class meter (GATE-SPARK-13)
PO-SPARK-01 Every kernel is dimensionally homogeneous (T-09) Decided exactly by the IR type checker
PO-SPARK-02 For each guarded predicate, a certified bound ε such that a computed residual r̂ of magnitude above ε has the sign of the exact residual r Rounding analysis of the operation sequence by frob-derive, classed Certified; exhaustive check on the boundary corpus
PO-SPARK-03 Resource bounds in the manifest (live values, stack bytes, loop bounds, argument count) hold and fit each backend's limits Computed by the IR; checked by the emitter
PO-SPARK-04 Declared read and write footprints contain the IR's load and store sets Derived for generated bodies; a lint for skeletons. The gateway checks declarations against each other, not against the source (FACT), so this obligation is ours

21.23 Known answers and negative controls

Known answer Expected Negative control that must be caught
One dialect source for K-cay-step on three backends Matches the Rust reference inside its class A prelude with contraction switched on
Contraction detector: (1 + 2⁻¹²)·(1 + 2⁻¹²) − (1 + 2⁻¹¹) 0 without contraction; 2⁻²⁴ with The fused form itself
Double-float canaries (1, 2⁻³⁰) and (1 + 2⁻¹¹, 2⁻²⁴) A build with fast math on
Strict predicate 2²⁴ < 2²⁴ + 1 True in the exact and guarded kernels The bare f32 kernel answers false
Boundary corpus of 2 × 65,536 rows with exact ties Exact kernel: zero errors, identical bytes on every backend. Guarded: zero errors after fallback The bare f32 kernel must mislabel at least one row
Stride conventions on a 2 × 2 view Element 3 at [1, 1] A verbatim stride copy reads element 12
Strict placement with one device kernel withheld Refusal before any world is stepped; zero episodes The same request under mixed-allowed must show actual ≠ requested
Run digest under slot permutation Unchanged A digest that chains in slot order
Kernel cache A one-byte change of source, options, compiler or driver identity is a miss A corrupted entry must be rejected by digest
Commit fence with injected command-buffer faults Final digests equal those of the unfaulted run With the fence disabled the mismatch must be detected
Specialized against generic kernel on 10⁶ states Bit-identical A reassociated variant must be refused
Scene regenerated on the other machine from its seed Same SceneId A generator with one changed constant

21.24 Gates and kill gates

Gate Claim Threshold (TARGET) Earliest
GATE-SPARK-00 Known answer One dialect source for K-cay-step compiles and runs on CUDA (trj) and on Metal (this Mac) and matches the Rust reference inside its class M0
GATE-SPARK-01 Known answer IEEE probe per device family: exhaustive bit comparison of f32 square root and reciprocal over all 2³² inputs; 10⁹ sampled operand pairs for +, −, ×, ÷ including subnormal and overflow edges; the contraction detector; subnormal and NaN behaviour recorded. The outcome sets each device's best attainable determinism class M0
GATE-SPARK-02 Fidelity Every M2 kernel and the full step agree between batch-cpu (f32) and each device inside the declared class: bit-equal for class D1; for class D3 at most 4 ulp per kernel output, statistical equivalence of 1,000-step rollouts, and the decision-level test of §35.4 M2
GATE-SPARK-03 Performance CUDA rows of §36.3 inside their tolerances, with the disclosure block and the paired lower bound of §36.6 M2
GATE-SPARK-04 Performance Metal throughput at least 25% of one RTX 4090 at matched settings on this M5 Ultra (BET-09), reported beside an incumbent with a Metal path measured on the same machine M2
GATE-SPARK-05 Known answer Precision battery: over 10⁶ steps, group-membership error within 4 ulp, closure and contact violations inside their envelopes, energy-ledger residual under 10⁻⁴ relative, at distances up to 1 km from the origin M2
GATE-SPARK-06 Known answer Cancellation and fault injection: every injected fault yields its typed refusal, no leaked device memory after 10³ cancel cycles, and the drain report is executor-minted M2
GATE-SPARK-07 Known answer Fingerprints of both machines are complete (every field measured, or unavailable with a reason), repeatable (two runs agree on every discrete field and within 5% on every rate), and reproduce the starting values of §21.16 M0
GATE-SPARK-08 Known answer Exact and guarded kernels pass the boundary corpus and the strict-predicate case on batch-cpu, both RTX 4090 and Metal, with identical bytes; the guarded kernel's fallback fraction is reported M0
GATE-SPARK-09 Known answer For every stage, withholding its device kernel under a strict request yields the refusal with zero worlds stepped; receipt counters for transfers and synchronizations match an instrumented shim within one count M2
GATE-SPARK-10 Performance Per-world parameter variants cost at most 5% against identical worlds at 8,192 worlds; 256 topology variants of one bucket run on one captured graph with zero compiles after warm-up; a mixed batch of three body plans reaches at least 0.70 of the harmonic mean of its homogeneous rates; admitting a variant into a free slot changes no other world's digest M2
GATE-SPARK-11 Known answer Allocated device bytes are flat over 10⁶ steps; a warm start loads every kernel with zero compiler invocations; each cache control of §21.23 is caught M2
GATE-SPARK-12 Known answer Zero graph captures after warm-up and zero synchronizations inside a chunk; no pool memory leaked after 10³ cancel cycles; 10⁵ snapshot reads each match the digest emitted for their epoch; the commit-fence known answer holds under 10³ injected faults M2
GATE-SPARK-13 Performance F On the standard campaign, the paired lower bound of the speedup of two machines over the better single machine is at least 1.3; at most 1 GB crosses per campaign-hour; no forbidden transfer class crosses M4
GATE-SPARK-14 Performance F A specialized kernel is bit-identical to its generic kernel and at least 1.2 times as fast on the whole step, with compile time at most 60 s M4
✖ KILL-METAL — Trigger: GATE-SPARK-04 fails after one optimization pass, or the Metal backend cannot hold run-to-run determinism. Fallback: Metal becomes a development and replay tier (single-device debugging, small batches, the viewer); CUDA is the training tier; the Mac trains on batch-cpu using its 36 CPU cores M2

KILL-KIR, KILL-BUCKET, KILL-SLOT, KILL-D1, KILL-FLEET, KILL-SPECIALIZE and KILL-MEGA are stated where their mechanisms are described. Determinism gates continue in §35.15 and performance gates in §36.16.

Mechanism register. Owner, verb, test and gate for each mechanism of this chapter.

Mechanism Owner Verb Test Gate Kill
Shared-dialect skeletons frob-kernels, annexes spark.check One source, three backends GATE-SPARK-00, -02 —
Derived arithmetic and the IR frob-kir, derive/frob-derive prove.kernel Emitted Rust against FrankenSymPy and dual numbers GATE-SPARK-02 KILL-KIR
Exact and guarded kernels frob-kernels spark.probe Boundary corpus GATE-SPARK-08 —
CUDA backend, graphs, pools spark/frob-spark-cuda spark.precompile, sim.run Capture and leak counters GATE-SPARK-03, -12 —
Metal backend, readback rules spark/frob-spark-metal sim.run Commit fence under faults GATE-SPARK-04, -12 KILL-METAL
Arena, leases, kernel cache frob-kernels, annexes spark.cache, spark.place Flat allocation; cache controls GATE-SPARK-11 —
Variants and buckets frob-kernels with frob-phylum sim.run, spark.place Slot and capacity invariance GATE-SPARK-10, -15 KILL-BUCKET
Slot dispatch daemon (§32) over frob-kernels job.* Co-tenancy digests GATE-SPARK-15 KILL-SLOT
Placement planner and receipts frob-kernels spark.place Withheld kernel GATE-SPARK-09 —
CPU-versus-device check frob-kernels, frob-conformance spark.check Planted divergence GATE-SPARK-02 KILL-D1
Hardware fingerprint frob-kernels, annexes spark.fingerprint, spark.devices Repeatability GATE-SPARK-07 —
Roles and seeds-only transfer daemon (§32) fleet.status Scene identity across machines GATE-SPARK-13 —
Fleet scheduler daemon (§32) fleet.plan Phase-ledger pilot GATE-SPARK-13 KILL-FLEET
Specialized kernels frob-kir, annexes spark.precompile Bit identity GATE-SPARK-14 KILL-SPECIALIZE
Fused step kernel spark/frob-spark-cuda flag on sim.run Digests unchanged pilot KILL-MEGA

21.25 Open questions

  • OPEN-SPARK-1. Owner ratification of ADR-SPARK-01. Default: exploratory label on GPU results. Blocks M2 exit.
  • OPEN-SPARK-2. A compute envelope for gate measurements and training campaigns (§36.14). Default: FrankenRobotics inherits FrankenSim's machine rule for CPU work and limits a GPU job to one device and one hour until the owner approves an envelope (envelope E0 of §36.14). Blocks GATE-SPARK-03, -04, -20 and -21 as specified; §36.16 restates the same question.
  • OPEN-SPARK-3. The bucket set and the capacities of each reference workload. Default: degrees of freedom {12, 18, 24, 36, 48, 64}, as §17.5 registers them, and the capacities of §36.3. Decided by the M2 profile; blocks GATE-SPARK-10.
  • OPEN-SPARK-4. Whether the pinned cudarc exposes graph capture and the asynchronous pool calls, and if not, whether the capsule may declare them. Default: preallocated transient buffers and per-launch submission, with the throughput cost disclosed. Decided at INT-CUDA-01.
  • OPEN-SPARK-5. What Spark does if INT-FT-01 fails, or until the upstream work item FD-FT-01 lands. Default: stay inside the gateway's limits and accept KILL-METAL's fallback; a Metal binding of our own is not built without an owner decision.
  • OPEN-SPARK-6. Whether the two RTX 4090 of trj allow device-to-device access. Default: host-staged transfers of small statistics only. Decided by the fingerprint.

21.26 Counts beyond 2³² without 64-bit atomics

What it is. S One design for device-side counting that is correct on every backend, given that exact kernels are 32 bits wide (§21.6, rule 7) and that 64-bit atomics cannot be assumed.

Facts. FACT to recheck (a reported compile probe on this M5 Ultra; re-run by spark.fingerprint): Metal Shading Language 3.2 and 4.0 offer no 64-bit integer atomic add and no 64-bit compare-and-swap. By rule 7 a kernel may use 64-bit atomics only if every device family in use records them, so no kernel of a mixed run uses them. A 32-bit count wraps at 2³² = 4,294,967,296 increments. A run at 4 M world-steps/s reaches that in 18 minutes if one histogram bin receives every sample.

Mechanism. DESIGN:

  • Bins. K-x-hist increments u32 bins with an integer atomic add, which is order-independent (§35.7).
  • Drain. At each chunk commit the host reads the bins of the committed slab and adds them into u64 totals in the ledger. The first kernel of the next chunk zeroes the device bins. Totals never live on the device.
  • A no-wrap bound, checked at plan time. The step plan records, per histogram, the largest number of increments one bin can receive in a chunk: worlds × control intervals per chunk × samples per interval. The plan is admitted only if that product is at most 2³² − 1. Otherwise the chunk is shortened, or the plan is refused (refusal: a device count could wrap within a chunk).
  • Sums wider than 32 bits, such as integer sums of quantized values, are never accumulated by atomics. Each world writes its own u32 partial sum, and K-seg-reduce adds two-limb values in its fixed tree: (lo, hi) + (lo′, hi′) = (lo + lo′ mod 2³², hi + hi′ + carry), with the carry computed by an unsigned comparison.
  • Digests are wrapping sums by definition (§21.8) and need neither.

Owner: frob-kernels (K-x-hist, the two-limb reduce); frob-chronicle (drain and totals); the Spark annexes.

Known answers. Two-limb add: (0xFFFFFFFF, 0) + (1, 0) = (0, 1). Boundary of the bound: 65,536 worlds × 4,096 intervals × 16 samples is exactly 2³² and is refused; one sample fewer per chunk is admitted. Planted overflow: a mutant plan that lets one bin receive 2³² + 5 increments in a chunk reads 5 on the device.

GATE-SPARK-23 Known answer. On CUDA and on Metal a histogram run of 10¹⁰ increments, concentrated in one bin, matches a CPU u64 reference in every bin; the boundary plan is refused and its neighbour admitted; the dialect lint finds no 64-bit atomic in any kernel source; the two-limb reduce matches u64 addition on 10⁶ random pairs. M2.