FrankenRobotics

Bets, each with its falsifier

4,860 words1 tablePlan version 1.1

A bet is a claim the project depends on and could lose. Each names what would falsify it, the competing arm it must beat or the potency control that shows the test is able to fail, the fallback named in advance, and where it is measured. Thresholds are TARGETs. Gates carried from version 0.2 are cited by identifier; for new bets the "tested at" column names the owning section, and the gate identifier and threshold are in §40.

Two terms are used throughout. A competing arm is a rival method given the same information, the same budget and a competent implementation; a bet that never meets one has not been tested. A potency control is a planted effect that the test must detect, or a null on which it must stay silent; a test that passes without one may simply have no power (§0.7). A falsified bet is reported as falsified, and its fallback ships under the fallback's own name.

BET-01 to BET-12 are carried from version 0.2 with their wording corrected by the incumbent audit. BET-13 onward are new. BET-13, BET-14 and BET-17 are the three on which the thesis of §2 turns.

ID Bet Falsifier Competing arm or potency control Fallback Tested at
BET-01 The algebra pays for itself. A Lie-typed, derived formulation gives competitive speed and removes whole bug classes. It is not a throughput lever: the recursions it could specialize are about 16–20% of an incumbent's step (H-04). Derived kernels run more than 1.3× slower than hand-tuned equivalents on the same backend after one optimization pass, or frame and unit bugs still appear in Gauntlet triage at the rate of an untyped baseline. Arm: a hand-tuned kernel for the same stage on the same backend. Control: planted frame, sign and unit defects that the types and the dimension checker must reject. KILL-KIR: hand-written arithmetic in Rust and the dialect under differential tests. GATE-SPARK-03, M2
BET-02 One source, three backends, inside declared determinism classes. Derived arithmetic plus a shared kernel dialect keeps CPU, CUDA and Metal inside their classes, the measured disagreement is stored per kernel, and a world's trajectory does not depend on the batch it ran in. One source for several backends is prior art; the classes and the record are the bet. Unexplained cross-backend disagreement above the class envelope survives two repair cycles, or the batch-composition invariance test fails. Controls: a planted contraction (fused multiply-add) in one backend and a planted read of another world's rows must both be detected. The affected device family is declared at a weaker class (tolerance replay), and the bit-for-bit wording of N6 is withdrawn for it. GATE-SPARK-01, GATE-SPARK-02 and the invariance test of §35; M2
BET-03 Convex compliant contact, audited by a referee, is good enough for locomotion decisions. On the standard battery the fast tier disagrees with the referee beyond the fidelity thresholds on slip distance, stance impulse or cost of transport and no parameter mapping fixes it; or the ranking of finalists changes under the model-form rank check (hard-contact law, cone shape, stiffness halved and doubled, half step). Arm: the optional hard-contact complementarity batch law F. Control: a planted energy-creating contact exploit, which the work–energy audit must catch. KILL-CONTACT: the Hunt–Crossley law CL-2 becomes the default at a published lower throughput; then locomotion claims are restricted to the referee tier. GATE-FID-01 at M2; the model-form rank check of §18 and §28 at M3
BET-04 FrankenSim can be consumed by path from outside its tree, stably, and the nightly coupling through fs-qty does not block the core. INT-FSIM-01 or INT-TOOLCHAIN-01 fails, or upstream breakage costs more than 20% of working days in any four-week window. Control: the admission tests for the five reported defects or limits of §1.7 must fail on the pinned commit if the defects are real, and pass after the upstream fix. In order: a pinned FrankenSim worker process per device that exchanges whole jobs; then KILL-FSIM, lifting a minimal referee set with provenance. frob-units if only the units crate blocks. INT-FSIM-01, INT-TOOLCHAIN-01; M0, then rolling
BET-05 Static analysis prunes bad designs before simulation and does not throw away a winner. Its measured ceiling as a pruning lever is modest (§2.2); its main value is feedback in seconds. On the design corpus, design check verdicts predict simulated failure no better than a mass-and-torque heuristic (area under the ROC curve below 0.8, or less than 0.05 above the heuristic); or the blinded audit of discarded strata finds a discarded design in the true top-k; or the planted winner is discarded. Arm: a mass-and-torque heuristic. Controls: a planted winner that the screen represents poorly; sampled discarded strata. KILL-STATIC: verdicts are downgraded to advisory lints and the pruning claim is withdrawn. GATE-ATEL-01, M3
BET-06 A baseline controller synthesized from the model makes most well-formed designs move, in seconds per morphology class and with no per-robot cost tuning. Fewer than 80% of corpus designs that pass design check traverse 5 m of flat ground under the baseline controller. Arm: a sampling controller with class-default costs on the same designs (BET-27). KILL-BASELINE: synthesis is limited to the morphology classes that pass; others report not-synthesizable and route to the sampling fallback or to training. GATE-REFLEX-02, M3
BET-07 Symmetry buys sample efficiency on symmetric robots and on tasks that preserve the symmetry. Equivariant policies need more than 0.7× the environment steps of unconstrained policies to reach the same return, across ten paired seeds; or they do no better than symmetry augmentation at equal compute. Arm: an unconstrained policy with symmetry augmentation. Control: a task that breaks the symmetry, on which the equivariant constraint must be refused. KILL-EQUIV: symmetry is used for augmentation and metamorphic testing only. GATE-CORTEX-03, M4
BET-08 Single precision is adequate on the GPU, given anchored coordinates, group-preserving updates, diagonal scaling of the contact system, and guarded comparisons with an exact fallback for strict predicates. Half precision is not used (H-15). The precision battery fails at long horizons (constraint violation, group-membership error or energy error exceed their envelopes); or a strict predicate decided in guarded single precision disagrees with exact integer evaluation. Arm: batch-cpu in double precision on the same worlds. Control: a boundary screen on which unguarded single-precision comparison is known to err (one pilot: 452 false admissions and 23,438 false exclusions in 65,536 cases; FACT, Appendix K) must end with zero errors under the guard. Double-float accumulators on the failing quantities (§35); quantities that still fail are evaluated on batch-cpu and labelled. GATE-SPARK-05, M2
BET-09 Metal on Apple silicon is a real rollout tier, not a demo. On this M5 Ultra, quadruped throughput is below 25% of one RTX 4090 at matched settings. Arm: Genesis on Metal on the same machine, reported beside our figure. KILL-METAL: Metal becomes the development and replay tier and the Mac trains on batch-cpu. The Mac's other roles (§1.8) do not depend on this bet. GATE-SPARK-04, M2
BET-10 Agents design better with structure than with reward alone. On the design benchmark of §24, agents given the full studio surface do not reach an accepted design in at most half the restricted mean time of agents given only sim run and a scalar score, at equal compute and tokens (one-sided 95% lower bound on the ratio above 1.5). Arms: the same agent with sim run and a score; the same agent on the composed incumbent tool layer. Control: a planted-defect corpus on which turns to a fix are counted. The surface is reduced to the verbs that showed an effect. The rest are kept as audit tools and carry no speed claim. GATE-ATEL-03, M4; the partial ten-fold gate, M3
BET-11 Proof-carrying kernels are affordable. Named kernels carry offline-checkable certificates of algebraic identities. The certificates say nothing about floating-point results, and their value to a decision is unproven. Instance certificates cover fewer than 80% of derived kernels within FrankenSymPy's budgets. Control: mutated kernels (a flipped sign, a dropped term) whose certificates must fail to check. Record: whether any certificate changed a decision (§0.8). KILL-LEMMA: certificates become exact-rational randomized tests recorded as Tested. GATE-LEMMA-01, M5
BET-12 Narrow purity is affordable. A Franken-only core with GPU annexes reaches M2 without a foreign physics engine. M0 plus M1 slip by more than 50% because a sibling lacked a primitive, its upstream work item had not landed and no interim posture in Appendix F applied. — A written re-plan before M2 starts. Program gate, end of M1
BET-13 Ten-fold on the sealed suite. On at least one named stratum, restricted mean time to an independently accepted answer is ten times shorter than the strongest composed rival's. The decision rule of §39 does not confirm a ratio of 10 on any named stratum at M7. Earlier: a measured ratio under 3 on a mission class at the M3 partial gate withdraws the speed claim for that class. Arm: the strongest composed rival, improved by a competent baseline engineer on development fixtures and then frozen. Controls: the ceiling computed before the test; routine easy wins for conventional methods locked into the suite; consequential edits hidden among inconsequential ones. The measured ratio is reported as measured, per stratum. Between 3 and 10 the word "ten-fold" is not used and the product is described by its ratio. Under 3 the product is positioned on auditability: replay, qualification, certificates and the evidence ledger. §39: partial gate at M3 (legged and wheeled briefs); tournament at M7
BET-14 The mechanisms need this engine (the composed-incumbent tie rule). A short robust program over an incumbent engine cannot reach the same accepted answers, with the same checking scope, in comparable time (H-19, ACTIVE). On a stratum, the composed-incumbent arm reaches the same accepted scope with a restricted mean time within a factor of 1.5 of ours. Or the composed incumbent's recovery ratio (§39) is at least 0.8 on elapsed time and on compute. The arm is the composed incumbent. A second arm runs our own layer over the incumbent engine through the same verbs, where the verbs allow it. Say so in the report. Ship the layer over that engine for that stratum. Keep engine work only for the gaps internal to an engine. Two are documented: a mixed-topology batch with run-to-run replay, and the Metal tier. Two are unverified and are checked before they are relied on: replay of one world outside its batch, and solver-internal certificates. Spend the research budget on the first mission the rival cannot resolve. §39, M3 and M7
BET-15 Family policies. One policy per morphology class, conditioned on the design, decides most designs in the class without per-design training. F Zero-shot return on held-out in-class designs is below 70% of the per-design specialist's; or rank agreement with specialists is below the BET-17 threshold; or pretraining, counted in compute, is not repaid within the break-even count N = P/(s·(1 − f)) of §23.11 at the measured P, s and f (illustrative: 60 GPU-hours at f = 0.1 repay after about 590 quadruped designs at s = 410 s, or 70 to 130 humanoid designs). Arm: per-design PPO from the baseline warm start. Control: designs outside the class must be flagged out-of-distribution, not walked badly in silence. The per-design behaviour-cloning warm start of version 0.2; the sampling fallback for out-of-distribution designs. No universal policy is attempted. §23; pilot before funding; M4. Needs an owner-approved compute envelope
BET-16 Inheritance. A child design's policy, fine-tuned from its parent along a lineage edge, reaches its return target in a small fraction of from-scratch steps. F A parameter edit needs more than 20% of the steps of a from-scratch control stopped at 95% of its final return; or inheritance does not beat a raw-checkpoint warm start for the edit class; or after a chain of ten inherited generations the final return is more than 5% below a from-scratch control (loss of plasticity). Arm: from-scratch training with the baseline warm start. Control: an arm that resets on a measured regression of its audit twin, and an arm with a depth schedule. Inheritance for parameter edits only, with resets on a measured regression and a depth schedule where no audit twin exists; topology edits start from the family policy or from scratch. The gain is bounded by the share of designs that are trained at all. §23, M4
BET-17 Rank agreement between rungs. Cheap rungs (static check, baseline controller, family policy, short horizon) order designs as the trained, referee-promoted rung does, well enough to eliminate. F Kendall τ between the baseline-controller rung and the trained-policy rung on the held-out corpus is below 0.7 for a morphology class and task family, with thresholds fixed before data are collected. Below 0.5 the cheap rung's verdicts are advisory for that class and the class leaves the speed claim. Or top-k recall, measured by the audit lane, is below its declared floor; or a cut sized from τ alone discards the higher rung's best design in the calibration set (the winner check of §23). Arm: successive halving on a fixed schedule. Controls: a planted winner; sampled discarded strata; classes expected to be hard (stairs, rubble, bipeds). The rung is demoted to advisory for that class, the elimination margin widens, and more designs are trained. §23, §28; M3 (legged, wheeled), M4 (tracked, hybrid). The decisive early experiment (§2.7)
BET-18 Mixed-topology batches. Many different robots in one batch beat per-family templates and are competitive with the published incumbent mixed batch on the two numbers that matter. F Against Newton with Kamino or Steel on trj we lose on both numbers (distinct designs times steps per second; time from an edit to the first GPU step); or nine per-family templates cover at least 95% of the mutation corpus at higher throughput; or the mixed batch runs below 0.5 of the harmonic mean of its homogeneous rates (the tolerance is 0.70, §36.3 row 10); or any of the 512 designs of a mixed cohort fails to reproduce the digests of its single-design run. Arms: per-family templates; padding every world to the largest robot; the incumbent heterogeneous solver. Per-family templates with per-world parameter variants, which are S; then sequential environments. §17, §21; M2
BET-19 Mission queries. A query over a design family, answered with inner and outer sets, witnesses and a next experiment, reaches an accepted answer sooner than point-by-point evaluation and survives amendment. F On the family strata of the sealed suite, a matched reducer (a lifted linear program, an SMT or interval tool, an expert's program) with the same grammar and information reaches the same witnessed answer at comparable cost; or an answer accepted by frob-verify is wrong on any locked case. Arm: an expert with standard solvers. Controls: the quantifier-order fixture, in which every design has a command that works and no single command works for all, must give different answers to the two queries; registered certificate mutations must be rejected. A tie retires any claim of exclusive expressivity on that fixture, and the query interface ships as a front end that compiles to standard solvers. A false answer withdraws the proof rule that produced it. §25: a CPU slice at M1, scale at M4; §39
BET-20 Region certificates. Per-constraint certificates with a split set of about six parameters, composed by monotonicity, symmetry and witnesses, resolve a useful share of a family. F On the reference families the unresolved band O ∖ I exceeds the mission's declared maximum after the budget; or a certificate check exceeds its budget; or a registered mutation (swapped contact ordering, an unreported force floor, a passive-authority tag) is accepted. Arm: dense sampling with statistical labels. Control: the midpoint counterexample (two feasible designs whose midpoint is infeasible) must be excluded from every inner set. Family answers are reported as tested sample sets with witnesses and make no inner-set claim. Isotropic paving of a high-dimensional box is not attempted in any case (H-06). §12, §25; M4, M5
BET-21 Blinded qualification. Error control holds across an agent's adaptive retries. S The false-promotion rate exceeds α under 500 adaptive retries on no-signal worlds; or power on planted gains falls below 0.8. Control: the unblinded procedure, repeated comparison on the same held-out worlds, must show an inflated error rate on the same no-signal worlds. That shows the test can fail. Fixed-n exact confirmation with a hard cap on submissions per campaign. §24, M3
BET-22 Minimal counterexample worlds. Adversarial search inside legality bounds finds failures faster than randomization, and each failure shrinks to a world of at most three edits that reproduces bit for bit and that the design's own body could have coped with. F Planted traps of probability at most 3 × 10⁻⁵ are found less than 3× faster than by uniform randomization; or fewer than 70% of reported counterexamples carry a solvability witness; or a shrunk world fails to reproduce bit for bit. Arms: uniform randomization at equal compute; a scenario-language generator given the same monitors. Controls: planted traps; an unsolvable world, which must be rejected. Importance-weighted tail sampling, with failing worlds reported unshrunk. §26, M3
BET-23 Interventional autopsy is stable enough to act on. Attribution by intervention on forked replays names the cause and says how stable the attribution is. S Top-1 accuracy on planted causes is below 90%; or the placebo flip rate exceeds 0.2 on more than half of the reference failures, so that most are unattributable; or an agent with autopsy needs more than one third of the repair turns of an agent with logs. Arms: first-anomaly ordering; an agent with logs only. Controls: the placebo fork, which must replay bit for bit; planted causes. Single-factor paired rewind; then first-anomaly ordering, labelled Estimated. §27, M3
BET-24 A requirement language. Bounded-time requirements serve as the acceptance metric and as a falsification score. They do not replace shaped rewards (H-13). S Requirement robustness disagrees with the reference monitor on any trace of the conformance corpus; or designs accepted by requirement verdicts fail the sealed variants at a rate no lower than designs accepted by a reward threshold. Arm: acceptance by reward threshold. Control: a policy known to exploit a shaped reward must pass the reward threshold and fail the requirement. Metric thresholds with declared comparators. Training keeps shaped reward terms in either case. §24, M3
BET-25 An energy-budget safety tier. A safety claim stated as an energy budget needs no contact model and survives contact uncertainty; reduced-order shields add a statistically measured margin. Classed Tested, never Proved (H-20). F Any false admission on the .fenergy mutation corpus; or a shielded policy keeps less than 85% of unshielded return; or the shield costs more than 5% of step time; or the measured margin is violated on shifted worlds beyond its stated rate. Arm: an unshielded policy trained with a penalty. Control: two friction regimes with identical histories before first slip (the pre-slip aliasing fixture of §22.21), where a monitor alone cannot see the breach in time. Monitors and reports only, with no shield claim. No full-order certificate is attempted in any case. §22, §12; M4
BET-26 Gait orbits as screening. Periodic orbits of a design under its baseline controller, continued in design parameters, give a screening signal for periodic gaits on regular ground. Never a referee-ready controller (H-05). F The median orbit solve takes more than 10 minutes of CPU (published solves span 4 to 260 s, which is about 1% to 63% of a 410 s training run); or continuation fails on half the catalogue; or a grazing or unstable control case is not refused; or the orbit rung fails BET-17 for the class. Arm: baseline-controller rollouts without orbit analysis. Control: grazing and unstable orbits planted in the catalogue. The gait table and max-plus timing analysis of version 0.2. §22, M4
BET-27 A sampling controller as fallback. With class-default costs it controls designs the synthesized baseline cannot, and supplies demonstrations for distillation (H-01). F On designs marked not-synthesizable, fewer than half traverse the reference task under it; or a 20 s episode costs more than 1 GPU-minute for a quadruped or 6 for a humanoid at the median (the thresholds of GATE-REFLEX-06); or distillation from its demonstrations does not reduce training steps. Arm: training from scratch. Those designs route to training. A sampling controller never stands in for a trained policy as acceptance evidence; its own episodes are accepted only under the delivered-controller lane of §22.8. §22; M3, M4
BET-28 Calibration bands. Within one calibrated robot and task family, conformal bands on the gap between simulation and measurement reach their stated coverage (H-14). F On held-out real episodes in the family, empirical coverage misses nominal by more than the finite-sample allowance (about six points at n = 20); or the exchangeability alarm fires and a band is issued anyway; or a band is issued for a design that was never built. Arms: uncalibrated simulation with randomization ranges; measuring the target directly. Control: a fixture with an omitted law that passes every in-family residual check and reverses the decision (one pilot: 675 of 675 correct in class, 0 of 225 on the omitted law; FACT, Appendix K) must trigger the alarm or a refusal. reality: no-claim. §30, M5. An import lane: without data the gate records no-claim
BET-29 Generative structure. Links optimized under simulated load histories are lighter than engineered parts at equal margin. It improves the output and is not counted in the thesis (H-12). F On the reference links the optimized part is less than 7% lighter than the engineered part at equal stiffness and fatigue margin; or the build packet fails its geometry round trip or its tolerance closure. Arm: the engineered reference part. The structural margin analysis alone (A-15). §31, M5
BET-30 Equivariance under the observation-preserving subgroup. The right group is Γ_obs ≤ Γ_mech, the subgroup that also preserves sensors, payload and task. F On a robot with asymmetric sensing, a policy constrained to Γ_mech does not lose return against one constrained to Γ_obs (then the distinction is unnecessary); or the Γ_obs policy fails the BET-07 threshold. The geometric-algebra tree policy M is falsified if it does not reach threshold in 1/1.3 of the steps of a plain morphology transformer (H-24). Arms: Γ_mech-equivariant; augmentation only; a plain morphology transformer. Augmentation and metamorphic tests under Γ_obs. §23, M4
BET-31 Campaign quotienting. Exact deduplication of designs under symmetry and mission-observable equivalence removes a measurable share of campaign work without removing a winner. F One false equivalence (two designs in one class whose mission outcomes differ beyond the determinism class); or the planted winner loses rank 1; or fewer than six of the top eight agree with the unquotiented arm; or digest classes save less than 5% of dispatches on the campaign corpus. Arm: the same campaign with quotienting off. Controls: a planted winner; the coupled-bus fixture, in which two modules with identical local summaries share one current-limited bus and must not be merged. Deduplication by canonical form and content hash only. §14, §28; M3, M4
BET-32 Rare-failure bounds. Bounded-weight importance sampling gives valid bounds on failure rates. Once failures are found it refutes a failure-rate target, and tightens an estimate, with fewer episodes than crude sampling; it does not certify absence more cheaply (H-07). S On fixtures with a planted failure rate, the bound's coverage is below nominal; or the effective sample size falls below 30% on the reference family; or, on a fixture whose planted rate of 10⁻⁴ is above the target, the episodes needed for a lower confidence bound to exceed the target are not at least 3× fewer than crude sampling needs (published range 2–20×); or a planted second failure mode outside the proposal is reported as absent. Arm: crude Monte Carlo with an exact binomial bound. Control: a planted rate. Crude sampling alone. Splitting remains exploration and yields Estimated rates only. §26, M3
BET-33 Revalidation. "Which verdicts survive this edit" is answered without ever returning a stale verdict (H-09). S One false reused verdict on the sealed amendment suite, which kills the reuse rule that produced it; or unnecessary invalidations above the declared share. Arm: a hermetic dependency graph with full recomputation of dependents, which tied a more elaborate rule exactly in one pilot. Control: omitting one declared dependency must produce a stale verdict that the shadow recompute catches. Full recomputation per verb, with compile-only memoization. §28, M3
BET-34 Referee coverage. The referee, with its transactional event-located step, completes the scenes where decisions are made, and the sampled referee covers enough of the fast tiers. S At M2 exit the referee completes fewer than 90% of standard-battery scenarios (the upstream contact step refuses more than 21 contacts; FACT, §1.7); or a promotion exceeds its time budget; or sampled coverage is below 0.1% of GPU world-seconds, with coverage computed from the referee's measured world-seconds per second and no band assumed. Control: a thin-barrier fixture and a witness-migration fixture that a fixed-step tier gets wrong. Upstream work items FD-FSIM-12 and FD-FSIM-13. Until they land, the strict batch-cpu profile stands as the highest tier for scenes the referee cannot complete, and results there stay provisional and say so. §16, M2
BET-35 Two machines. The division of labour of §1.8 pays. F A campaign on both machines is less than 1.3× faster than on the two RTX 4090 alone; or a larger world bank does not reduce the held-out gap; or more than 1 GB per campaign-hour crosses between the machines. Arm: the two cards alone. The machines run as independent shards, and the Mac keeps proof, replay and viewer work. §21, §36; M4
BET-36 Living worlds change decisions. A contact-level film-friction law and rigid proxies stand in for coupled matter S; whether they reproduce the failures of coupled models is the pilot of §19.10.7; films advected by the robot change design rankings F (H-16). The film law misses its known answers, changes fewer than 10% of episode outcomes on the wet-floor battery or adds more than 5% to step time (KILL-FILM, §18.8); or, in the pilot of §19.10.7, the proxies reproduce fewer than 90% of the coupled model's failures in the same class (the proxy claim is then withdrawn for that class and the island stays as an escalation); or advected films leave design rankings on the spill battery at a rank correlation of 0.95 or above with static patches, or cost more than 15% of the step (KILL-ADVECT, §19.10.7). Arm: static friction maps. Static friction maps with wet patches, as in version 0.2. Coupled multiphysics islands stay M. §19; M3, M6
BET-37 The building is a design variable. A priced retrofit catalogue finds a cheaper fix in the world than in the robot for a meaningful share of failures, with the policy frozen. F Kits of at most three items fix fewer than 30% of minimized failures on the retrofit corpus; or an accepted kit violates a dimensional norm from its cited source. Arm: robot-only repair. Control: a failure that no listed retrofit can fix must return the bottleneck and no kit. The clearance complex reports the bottleneck only. §19, M4

Moonshot items (continuation-based synthesis, coupled multiphysics islands, rods, cloth and slosh, the rotor port, tactile hands, firmware export, lifecycle compression, facility co-design and multi-robot handoff, skill transport with learned residuals) are not bets. The plan does not depend on any of them. Each has a pilot, known answers and a kill gate in its own chapter.

Bets are tested in an order that puts the cheapest decisive evidence first: BET-04 and BET-12 at M0 and M1; BET-17 as soon as two rungs exist; BET-02, BET-08, BET-09, BET-18 and BET-34 at M2; BET-13 and BET-14 in part at M3 and in full at M7.