FrankenRobotics

The ten-fold test

15,590 words22 subsections22 tables6 listingsPlan version 1.1

On this page22
39.1 Position39.2 The estimand39.3 What counts as an accepted answer, and who adjudicates39.4 The sealed mission suite39.5 The arms39.6 The phase ledger39.7 Statistics39.8 The ceilings39.9 Disqualification39.10 The outcome rules, written in advance39.11 The incumbent parity baseline at M0 and M139.12 The partial gate at M339.13 The tournament at M739.14 The replication kit39.15 What the test does not measure39.16 Worked card: from a phase ledger to a ratio39.17 Invariants and known answers of the harness39.18 Gates and kill gates39.19 Open questions39.20 Estimating the ratio, and the smallest suite that can certify39.21 Rival arms, protocol controls and the sealed suite39.22 Three arms, the recovery ratio and the phase ablations

This chapter defines what "ten-fold" means as a measurement and how the claim is confirmed, reported as something smaller, or withdrawn. §2 states the thesis and its arithmetic; this chapter is the protocol that tests it. It fixes the estimand, what an accepted answer is and who says so, the sealed mission suite, the arms, the phase ledger that every run records, the statistics, the ceilings that are computed before any speed test is run, the grounds for disqualification, and the outcome rules, which are written here before any measurement exists. Nothing in this chapter is a result. Every threshold is a DESIGN choice or a TARGET, and every figure about an incumbent is a FACT from Appendix H, to be re-read from its primary source before a gate cites it.

39.1 Position

  • The claim under test is BET-13, and the tie rule is BET-14 (§4). The test is the only evidence the plan accepts for either. Engine throughput is a parity requirement with its own gates (§36); it is an input to this test and never a substitute for it.
  • The harness lives in frob-conformance (§34, §38). It has no product verbs of its own. It measures FrankenRobotics through the public verbs of §32, the same surface a user has, and it drives every rival tool out of process, as §38 requires of foreign oracles. Its records use the schema names frob.tenfold.mission/1, frob.tenfold.freeze/1, frob.tenfold.ledger/1, frob.tenfold.receipt/1 and frob.tenfold.report/1. They are harness records, not product formats, and they travel in the replication kit (§39.14).
  • Three parties. The builders wrote FrankenRobotics. The rival advocates build and tune the rival arms and have written no FrankenRobotics code. The adjudicator writes the sealed missions and the referee protocol, runs acceptance and releases receipts, and belongs to neither group. A person or an agent session holds exactly one of the three roles for the life of a gate (INV-TENFOLD-01).
  • Two program gates. A partial gate at M3 on the classes that exist by then, and the tournament at M7 on the whole suite. Before both, the incumbent baseline is measured at M0 and M1 (§39.11), because no end-to-end phase ledger of an agent-driven design loop on the current incumbent stacks has been published (FACT, Appendix H), and a ratio without a measured denominator is not a ratio.

39.2 The estimand

The clock. T is the wall-clock time from the moment the agent receives the mission card and its permitted inputs to the moment the adjudicator accepts the first correct answer with all requested scope and precision. Unsuccessful runs, training, tuning, compilation, device transfer, queueing, investigation of unknowns, certificate generation and checking, repair, review and every rejected submission are inside it. Time the harness itself spends between an arm's request and its service is measured and charged to the phase wait (§39.6). It is not subtracted.

The restricted mean time. Each stratum s has a common deadline H_s, fixed before the suite is sealed. A run that is not accepted by H_s is censored and contributes H_s. The restricted mean time of an arm on a stratum is RMT = E[min(T, H_s)] over the sealed missions and campaign seeds of that stratum. No run is dropped: a crash of the arm's own software is time passing, and a run that never finishes costs H_s.

The ratio. For arms A (FrankenRobotics) and B (the strongest composed rival, §39.5),

R_s = RMT_B ÷ RMT_A = E[min(T_B, H_s)] ÷ E[min(T_A, H_s)].

It is estimated by the pooled ratio of §39.7, which is a ratio of sums over paired runs and not an average of per-mission ratios. An average of ratios is dominated by the missions where arm A is fastest and hides the mission it never finishes. A ratio of sums charges that mission its full deadline. This restricted-time ratio is the only quantity the word "ten-fold" attaches to; the ratio of accepted-output rates is a different estimand and is cited only for statements about decisions per hour (§39.20). Where the composed incumbent of §39.22 (arm C here, arm W there) has a smaller restricted mean time than arm B, the word is governed by the ratio against it: the word is never awarded on arm B alone.

Regimes. Cold and warm runs are separate results and are never pooled (§39.4). A ratio is always reported with its regime: "cold", or "warm at N amendments".

Why not steps per second. DESIGN: the unit of success is an accepted answer. Three FACTs settle the choice (Appendix H). An infinitely fast environment step shortens training by at most about 7.7 times (1/(1 − 0.87)) on one incumbent stack and about 1.4 times on another, because the step is 87% and 29% of the respective loops. An incumbent already measures about 2.55 M humanoid world-steps/s. And in the one published agent-driven design loop with a time breakdown, a round takes about an hour and is dominated by policy training, which no step rate describes. A step rate also says nothing about whether the answer it produced is right. Steps per second are therefore measured and gated in §36 as parity, and they enter this test only through the phases they shorten.

Columns reported beside the clock. Under identical resource caps, each run also records GPU-hours, CPU-hours, energy where a credible meter exists (§36), agent tokens and tool calls, human minutes (the default is zero, and any intervention is logged), and money at declared prices. Two derived units are reported for design strata: designs decided per hour, and compute per accepted decision at a matched error rate. The primary claim is wall time. A cost claim would need its own prices and its own registered test, and none is made here.

39.3 What counts as an accepted answer, and who adjudicates

Answer kinds. A mission card names exactly one.

Kind What the arm submits Accepted when
Qualified design A design in the mission's family, a controller behind the standard observation and action interface, and the declared clocks The acceptance battery passes in the adjudicating simulator: over n sealed world clusters, the exact one-sided lower confidence bound on the requirement pass rate is at least the mission's threshold
Family answer Inner and outer design sets with witnesses, exclusions with impossibility witnesses, and the unresolved band Every mandatory interior, exterior and boundary obligation of the card is discharged; the unresolved band or optimality gap is within the card's maximum; every certificate checks; direct evaluation at the adjudicator's sealed probe points contradicts nothing. Strict and closed endpoints are part of acceptance where an exact boundary is asked for. A vacuous enclosure discharges nothing
Amendment verdict For a changed mission: which earlier verdicts survive, which are withdrawn, and the repaired answer The repaired answer passes as its own kind; no kept verdict is one the adjudicator's ground truth withdraws. A missed withdrawal fails the mission; unnecessary invalidations are counted and reported
Truthful unknown A statement that the supplied evidence does not identify the choice, with the required witness: two admissible worlds or models under which the best choices differ, or the distinguishing experiment the card asks for Only on missions whose question is whether the evidence identifies the choice. Everywhere else unknown is not an answer and the clock keeps running
Counterexample and repair A replayable failing world inside the legality bounds, and a repaired design The failure reproduces in the adjudicating simulator, and the repaired design passes the battery of the first row

The neutral adjudicating simulator. DESIGN: dynamic acceptance batteries run in a simulator that neither arm used for training, tuning or screening on that stratum, whose contact model was derived independently of both arms, that runs deterministically on a CPU, and that is pinned by revision. The reason is that an arm scored in its own engine is scored on its own modelling errors, and two programs that share a solver have correlated error (§8). The default candidate is Drake at a pinned release (FACT, Appendix H: a CPU multibody engine with its own convex contact solver). On a stratum where an arm uses that engine as a tool, or where it cannot represent the mission, the adjudicator substitutes another engine that meets the four conditions. If none exists the stratum falls back to cross-home adjudication: an answer must pass in the highest-trust tier of both arms (the referee of §16 and the rival's CPU reference), run by the adjudicator, and the report labels the stratum adjudication: cross-home (KILL-TENFOLD-ADJ). Models reach the adjudicating simulator through a neutral exchange description that the adjudicator's own importer reads. A controller runs as a separate process behind the observation and action interface with the mission's clocks. The gap between an arm's home simulator and the adjudicator is that arm's risk, equally for both.

Certificates and family answers. Every submitted mathematical assertion is replayed by frob-verify (§12), whose schema is public and which any arm may target. Because the builders wrote that checker, it is not the last word in this test. The adjudicator also evaluates each family answer directly against the mission, with tools of the adjudicator's choosing: exact rational evaluation at sealed probe points, an SMT or interval tool on the raw mission statement, dense sampling in the adjudicating simulator. A disagreement between an accepted certificate and the direct evaluation is a false certificate (§39.9).

The referee protocol is not written by the builders. The adjudicator writes it from the public class descriptions, the mission grammar and the schemas. Builders and rival advocates may comment on drafts through one logged channel until the suite hash is committed. After that the protocol is frozen and its hash is in the ledger (GATE-TENFOLD-04). The builders supply schemas, the Gymnasium-compatible adapter of §23 and the development missions. They supply no acceptance code.

Submissions and retries. Acceptance uses the protocol of the blinded qualification lane (§24), run by the adjudicator on the adjudicator's own sealed key. A submission is committed before any world is drawn. Each submission draws fresh world clusters. The i-th submission on a mission is tested at level α_i = α / [i·(i + 1)] with α = 0.05. The arm receives a receipt holding only the decision, the index, the level spent, the units used and a hash. A rejected submission returns no per-world outcome. Retrying is allowed, costs clock time, and cannot buy acceptance by repetition. Every arm is under the same schedule. Where the submission format allows it the adjudicator scores an answer without being told which arm produced it, and the paired results (times, per-world outcomes and transcripts) are released only after every arm of the gate has closed.

Who the adjudicator is. DESIGN: an external party, a person or a team with no authorship in any arm. OPEN-TENFOLD-2 names it. The fail-closed default, if none can be engaged, is a fresh-context agent session with no write access to any arm, operating from the protocol document alone under a named human, with every action logged. Every ratio measured that way is labelled self-adjudicated. Under that label a ratio may be reported, and the words "independently accepted" and "ten-fold" wait until an external party has re-run the replication kit (§39.14).

Three rules that protect the bar. The time score is never improved by lowering what counts as accepted. An answer that was accepted and is later shown wrong within its scope scores that mission as unresolved at H_s for the arm that submitted it. A correct refusal that arrives after the deadline is a missed deadline.

39.4 The sealed mission suite

Coverage grid (DESIGN). The design strata cover four robot classes on eight world types: open terrain and the seven interior archetypes of §19. The task families are those of §24.3, and the robots are drawn from the eighteen archetypes of §14. At M7 every cell holds at least two sealed mission families, or is declared not applicable with a written reason before the suite is sealed.

Robot class Terrain Home Apartment Office Warehouse Manufacturing plant Retail store Commercial kitchen
Legged (quadruped, hexapod, biped, humanoid) Stairs, gaps, stepping stones, rubble Thresholds, rugs, a stair flight A lift-lobby sill and tight turns Castor chairs and glass doors A ramp with payload The split-grip landing; an oil patch Aisles with movable carts A wet galley crossing
Wheeled (differential, car-like, mecanum, omni, swerve, balancer, mobile manipulator base) Slopes and curbs Threshold delivery with a loaded cart Timed docking under a hidden plant gain Carpet rolling resistance; a door closer Payload transport over a dock plate Boarding and leaving a conveyor Parallel shift and docking in an aisle A greasy patch and a swing door
Tracked (tracks with flippers) Stairs, rubble, soft ground A stair flight with a carried load Turning in place on a sill Scrub energy on carpet Ramp traction with payload Floor grates and cable covers Stopping distance in an aisle A low-traction turn
Hybrid (wheel-leg, tri-wheel cluster, rocker-bogie) Curbs and stairs on wheels Stairs and thresholds, switching between rolling and stepping A lift sill Thresholds between carpet and tile A ramp and a dock plate Cable covers and grates A curb at an entrance Drains and mats

Strata. A stratum is a set of mission families that share a robot class or an answer kind, and a deadline. Each named stratum gets its own ratio, and a result on one stratum earns that stratum alone.

Stratum Contents Answer kind Strongest control inside the rival arm First run
legged-quad Quadruped and hexapod design briefs with parametric and single-limb edits, across the first row of the grid Qualified design Per-design reinforcement learning with an outer-loop optimizer; a shared design-conditioned policy where one is published M3
legged-biped Biped and humanoid briefs Qualified design The same, with the incumbent's humanoid training recipe M7
wheeled The second row of the grid Qualified design The same, plus a model-based planner where the task has one M3
tracked The third row Qualified design The same M7
hybrid The fourth row Qualified design The same M7
envelope Body-envelope family queries with amendments: adjustable stance and static load; timed docking; the quantifier-order fixture (§25) Family answer; amendment verdict A lifted robust linear program, a conventional projection, an SMT or interval tool, written by an expert M3
sensing Sensing and material missions: the split-grip landing with a last usable observation; observation aliases; an unfamiliar contact law with a probe to choose Qualified design or truthful unknown Belief-state reachability, equal-budget reinforcement learning, an expert probe, direct measurement, active system identification M3
rebody Change an accepted body under a frozen task interface (a gearbox, a wheel, a lower limb), then perturb latency, friction or energy reserve Amendment verdict Retraining, a shared conditioned actor, warm-started control, ordinary graph caching M7
failure Find a planted failure inside the legality bounds and deliver an accepted repair Counterexample and repair Uniform randomization, a scenario-language generator with the same monitors, logs M7
throughput A heterogeneous family at scale: fixed and changed topology, grazing contacts, dense depth sensors, memory pressure. The complete loop is timed, including certificates, divergence handling, cold compilation and the learner Qualified design over a family An optimized batched incumbent; and FrankenRobotics with family sharing switched off M7
facility M Recommission a facility: move a door or a counter, perturb scan uncertainty, reduce a shared resource, alter a shift Amendment verdict Recomputation, an incremental mixed-integer program, standard motion planning, selective replay with cached bounds M7, only if GATE-AUGUR-24 has passed; otherwise not run

Sizes (DESIGN). Each stratum has 12 public development missions and an initial 24 sealed mission families. The final sealed count is fixed by the pre-analysis of §39.7 before the suite is opened, and is never below 12. A stratum that declares a population result needs at least 32 independent families at α = 0.05, and 46 where four strata share the level; a stratum with fewer declares a finite-suite result (§39.20). A mission family is the independent unit: missions that share a generator draw, a site, a material batch or a parent design belong to one family and stay together in every split. Twenty-four missions on one site are one replicate.

How missions are written. The adjudicator writes each mission as a card: the robot family and its bounds (grammar, masses, actuator catalogue, and the symmetry knowledge every arm is given equally); the world family and its generator parameters; the task, stated in prose and as a requirement in the language of §24 with the adjudicator's own monitor; the answer kind and its obligations; the acceptance battery (n, threshold, tier); the deadline; the resource caps; the permitted inputs (calibration data, prior answers, caches); for warm missions the amendment sequence; and the control flags below. Cards are written from the public class descriptions. Some worlds are built with an evaluator independent of every proposal engine, because a sampling run and a solver that share one mistranslated world model agree on the wrong answer.

Anti-shortcut controls, locked into every stratum.

  1. Easy wins for conventional methods. At least one family in six is a mission on which a conventional method is expected to win or tie. If the frozen rival wins none of them on the development suite, the rival is presumed under-built and the freeze is refused (GATE-TENFOLD-04).
  2. Genuinely unresolved answers. At least one family in twelve has unknown with a witness as its correct answer.
  3. Independent mutations. Unit, contact-order, causal-clock and passive-load mutations are derived by the adjudicator, not by an arm.
  4. Hidden consequence. In an amendment sequence the card does not say which edit matters. Missed withdrawals and unnecessary invalidations are both measured.
  5. The quantifier-order fixture is in envelope permanently: every design has a command that works and no single command works for all, so a per-build query and a shared query must return different answers (§25).
  6. Tier traps. At least one family per design stratum has a candidate that passes on a fast tier and fails in the adjudicating simulator.

Sealing. The order is fixed, and each step is a ledger entry with a hash and a time (INV-TENFOLD-02): (1) class descriptions and the development suite are published; (2) the adjudicator commits the hash of the sealed suite and of the referee protocol; (3) arms are tuned on the development suite only; (4) each arm's freeze manifest is published (§39.5); (5) the pre-analysis is recorded; (6) the suite is opened to the harness, and a mission card is shown to an arm only when that arm's clock starts. Step 2 precedes step 4 so that no mission can be written against a frozen arm. Step 4 precedes step 6 so that no arm can be tuned against a mission.

Rotation. A sealed family serves one gate. When the gate closes, its missions, worlds and receipts are revealed and the family is retired into the development pool. It never serves as a sealed family again. The tournament at M7 uses no family that the partial gate at M3 used. A family touched by a leak is retired at once (§39.9).

Cold and warm. A cold run starts from the mission card and nothing else that is specific to the mission: authoring the environment, the design domain, the reward or requirement and the evaluation set is inside the clock for every arm. A warm campaign runs a declared sequence of amendments to an accepted cold answer, with identical reusable inputs and equal cache budgets for every arm. A fair rival also caches graphs, factors, compiled kernels and policies. A failed amendment stays in the sequence and costs its deadline. Accepted count, total time and time per acceptance are reported separately.

Standing assets and the crossover reuse count. A standing asset is anything an arm built for a mission class before a clock started: a pretrained family policy, a kernel or shader cache, a compiled scene bank, a tuned reward library. Each arm declares its standing assets in its freeze manifest with their measured build cost S. Let ā and b̄ be the pooled restricted mean times per amendment of arms A and B. The warm ratio at reuse count N is

R_warm(N) = (S_B + N·b̄) ÷ (S_A + N·ā),

reported at N = 1, 10 and 100 and in the limit b̄ ÷ ā. The crossover reuse count N* is the smallest N at which S_A + N·ā ≤ S_B + N·b̄. The warm verdict of §39.7 is evaluated at N = 10 (DESIGN: a class that needs a hundred reuses to pay for itself is reported as such and is not summarized by its limit). A warm result is never quoted without S and N*.

39.5 The arms

Arm What it is Built and tuned by What it decides
A FrankenRobotics at the gate's commit, driven by the common agent through robot-docs and the verbs of §32 The builders The denominator of R
B The strongest composed rival: a named incumbent engine, per-design reinforcement learning, a standard outer-loop optimizer, analytic bounds and a verifier where the stratum needs them, and the same agent The rival advocates The numerator of R
C The ablation arm: a short robust program over an incumbent engine The rival advocates BET-14: whether the mechanisms of §2.4 need this engine

Arm B, named (DESIGN). The advocates choose, per stratum, the configuration that finishes the development suite soonest; it is one frozen configuration per stratum, never a per-mission choice. The candidates are FACTs of Appendix H at the releases read for this plan (MuJoCo and MuJoCo Warp 4.0.1, Newton 1.8.0, Isaac Lab 3.0.0, Genesis 1.4.7, Drake 1.58.0, ManiSkill 3.0.1), re-pinned by revision at GATE-TENFOLD-01.

Strata Engine and learner Outer loop and tools
legged-quad, legged-biped, wheeled, tracked, hybrid MuJoCo Warp under MuJoCo Playground or mjlab, or Isaac Lab on Newton or PhysX, each with its own PPO recipe, one policy per design; a published shared design-conditioned policy where one fits CMA-ES or Bayesian optimization over the design parameters with successive halving; the engine's per-world model fields for parametric variants; warm compilation caches; sampling-based control with no training where it helps
throughput Newton with a solver that batches different topologies, or ManiSkill3 The same, with mixed-topology batches
envelope, sensing An expert's program over Drake, a linear or mixed-integer solver, an SMT or interval tool A scenario language for world generation; system identification from the same calibration data
rebody, failure, facility The locomotion configuration above Retraining with a warm start; uniform randomization with the same monitors; an incremental mixed-integer program and a standard motion planner

Every capability an incumbent documents is available to arm B and is expected to be used where it helps (FACT, Appendix H): per-world model parameters in one batch, state-preserving model edits, opt-in run-to-run determinism, mixed topologies in one batch, electrical and thermal actuator models, sampling control without training, interior generators, and agent access over MCP. A rival that leaves a documented capability unused on the development suite is asked why before the freeze.

Arm C, defined (DESIGN). At most 2,000 lines of program text, counted by the adjudicator, over the engine of arm B. Its authors have read this plan. It may implement any mechanism of §2.4 that needs no engine internals: batched parametric variants through the incumbent's model fields, a ladder of rungs with a published anytime-valid test, a sampling controller or a published shared policy as the cheap rung, a standard solver for family queries, restorable state for forks, the incumbent's own caches. Arm C is the plan's test of itself. If it ties, §39.10 says what happens. Where the verbs allow it, a fourth arm D runs FrankenRobotics' own layer over the incumbent engine through the same verbs; it is optional and is reported when run.

The same three arms under the names of §39.22. §39.22 writes F for arm A, keeps B for the declared incumbent, and writes W for the composed incumbent. Arm C is the first form of W: W is the same engine with every portable method layered on by a competent baseline engineer, built as one pipeline before arm A exists, and it is not held to arm C's 2,000 lines where a portable method needs more. The tie test of §39.10 and the recovery ratio of §39.22 are both computed on that arm.

Equal terms (binding). The same agent model at one pinned revision for the whole gate, the same token and tool-call caps per campaign, a fresh context per campaign, and no human in the loop. The same hardware envelope: the primary result is on matched devices, one RTX 4090 of trj and the same number of CPU cores per campaign, with card assignment and run order randomized and counterbalanced and the arms interleaved in time so that load and temperature are shared. The same calibration data, reusable prior answers and proof problems. The same documentation budget: arm A's agent reads robot-docs; the agents of arms B and C read the incumbents' own documentation and an advocate-written guide no longer than robot-docs, frozen with the arm. A second result on each arm's best-supported platform may be run (arm A may add the M5 Ultra); a platform an arm cannot use is printed as unavailable, and a gap that exists on one platform only is a portability observation, not the ratio.

Tuning by advocates. The advocates receive the development suite, the common agent, this plan, and a tuning budget at least equal to the compute the builders logged on the same development suite and never below 40 device-hours per stratum (DESIGN). They tune on development missions only. They may improve the rival in any way a competent engineer would: hyperparameters, reward shaping, curricula, warm caches, policy reuse, batching, analytic pruning. Before the freeze the builders may object in writing that the rival is under-built, and the adjudicator rules.

Freeze and publication. Incumbents are frozen first: no paired run of any kind happens before every arm in it has a published freeze manifest (INV-TENFOLD-03). A manifest is content-addressed and is published, with the advocate's signed statement that this is the strongest configuration the budget allowed and a list of what more budget would have added. After the freeze nothing changes in any arm, arm A included. A defect found in a frozen arm after the suite opens voids the affected cells, which are re-run at the next gate; it is never patched mid-gate.

{ "schema": "frob.tenfold.freeze/1", "gate": "M3-partial", "arm": "B", "stratum": "legged-quad",
  "engine": {"name": "…", "revision": "…", "licence": "…"}, "learner": {"name": "…", "revision": "…"},
  "outer_loop": {"name": "…", "script": "b3:…"}, "tool_layer": "b3:…", "agent_guide": "b3:…",
  "agent": {"model": "…", "revision": "…", "token_cap": "…", "tool_call_cap": "…"},
  "caches": {"compile": "warm", "policies": ["b3:…"]},
  "standing_assets": [{"name": "…", "build_wall_s": "…", "build_device_s": "…", "hash": "b3:…"}],
  "phase_tags": "b3:…", "tuning_log": "b3:…", "development_scores": "b3:…",
  "advocate": {"statement": "b3:…", "would_add_with_more_budget": ["…"]},
  "hardware": {"fingerprint": "b3:…"}, "frozen_at": "…", "manifest_hash": "b3:…" }

The learner-swap diagnostic. The Gymnasium-compatible adapter of §23 lets a rival learning stack train in this environment. Of the four combinations of engine and learner, three can be run: ours with ours, theirs with theirs, and our engine with their learner. The three split a difference in the train phase into an engine part and a learner part. The diagnostic never enters R. If the adapter fails its gate (KILL-GYMADAPT), the rival learner drives whole rollouts through the job interface and the added latency is recorded in the ledger.

39.6 The phase ledger

Every run of every arm records where its clock went. The ledger is what turns a ratio into an explanation, and it is the only input the ceilings of §39.8 accept.

Phase What is charged to it Chapter phases that map here
author Writing the environment, design domain, reward or requirement, and evaluation set from the mission card §28 authoring
propose An agent turn that reads results and emits the next request: model latency and tool-call overhead §28 review
check Design realization and static analysis: edit to a valid model, preflight, pre-simulation analyses §28 check
synthesize Controller synthesis without training: baseline controllers, gait orbits, trajectory optimization §28 synthesis
compile Compilation or cache lookup, model upload, variant admission, graph capture, scene compile, up to the first device step §36 compilation, admission
screen Rollouts on cheap rungs that train nothing §28 simulation; §36 screening
train Policy training, fine-tuning and distillation, learner time included §28 training; §36 training
evaluate Evaluation batteries, comparisons, promotion to a higher tier §28 promotion; §36 evaluation, promotion
qualify The arm's own blinded confirmation before it submits §36 qualification
certify Proof search, certificate writing and the arm's own checking —
diagnose Failure analysis: replays, forks, attribution, counterexample search, reading logs §28 review
wait Queueing, device transfer, idle time, harness service time §36 waiting
adjudicate From a committed submission to its receipt —

Attribution (DESIGN). The raw ledger is a list of intervals, each with a start, an end, a phase, a resource and, for agent turns, token counts. From it the harness derives two tables. The exclusive table assigns every instant of the clock to one phase: an agent turn in progress owns its instants; while the agent is blocked on jobs, the job that finishes last among those it waits for owns them; while it is blocked on the harness or the adjudicator, wait or adjudicate owns them. The resource table gives device-seconds and core-seconds per phase, including work that ran in the background of an agent turn. The exclusive table sums to the clock.

Closure. The exclusive phases sum to T within 2%. Any remainder is recorded as unattributed. A run with more than 2% unattributed keeps its place in the ratio, is flagged ledger-open, and is excluded from ceiling arithmetic (INV-TENFOLD-04). For arms B and C the advocate's wrapper tags every process it launches, and the tag map is part of the freeze manifest.

{ "schema": "frob.tenfold.ledger/1", "gate": "M3-partial", "arm": "A", "stratum": "legged-quad",
  "mission": "b3:…", "family": "f:…", "campaign": 7, "regime": "cold", "deadline_s": "14400",
  "clock_s": "1500", "accepted": true, "censored": false, "submissions": 1,
  "exclusive_s": {"author": "600", "propose": "360", "check": "20", "synthesize": "0", "compile": "12",
                  "screen": "96", "train": "150", "evaluate": "60", "qualify": "40", "certify": "0",
                  "diagnose": "30", "wait": "42", "adjudicate": "90", "unattributed": "0"},
  "resource": {"device_s": {"screen": "96", "train": "150", "evaluate": "60"}, "core_s": {"check": "160"}},
  "agent": {"turns": 4, "tokens_in": "…", "tokens_out": "…"}, "designs_decided": 128, "trainings": 1,
  "disclosure_block": "b3:…", "freeze": "b3:…", "receipts": ["b3:…"] }

What reads it. The ceilings of §39.8. The attribution of a measured ratio to the levers of §2.4: lever 1 acts on propose through the number of turns and on screen; lever 2 on train; lever 3 on screen and evaluate; lever 4 and lever 7 on diagnose; lever 5 on certify and propose; lever 6 on check, compile and evaluate; lever 8 on compile and author. And the kill gates that promise to record a lost factor here: KILL-RANK, KILL-PARITY and KILL-GYMADAPT.

The end-to-end rows of §36. Row 20, edit to first step, is the compile phase of one edit, measured for the incumbent at GATE-TENFOLD-01 and for both arms in every run. Row 21, designs decided per hour, is designs_decided over the clock. Row 22 is the restricted mean time itself. The learner's share of train is recorded in the resource table, so that an engine difference and a learner difference can be told apart (§39.5).

39.7 Statistics

Two studies, never mixed (§24). The verdict of a gate is a fixed-n study: its sample is fixed before the suite opens, its rule is fixed, it has a pre-analysis of which verdicts are reachable, and it is computed once, at n. A separate anytime-valid study monitors the incoming runs for operational decisions. At the partial gate the monitor can stop a cell and withdraw a claim. It can never award one.

Design. A cell is a stratum and a regime. A campaign is one arm's attempt at one mission with one campaign seed, which fixes the agent's sampling seed where the provider allows it, the optimizer seeds and the arm's own world seeds. A pair is the two campaigns of arms A and B on the same mission with the same campaign index, scored on the same sealed acceptance worlds. A cell has m mission families and c campaigns per family, n = m·c pairs. DESIGN: m = 24 and c = 1 by default, c = 2 where fewer than 24 families exist. No verdict of any kind is issued on fewer than ten paired campaigns per arm (n ≥ 10), and the family is the unit of inference. Ten is a floor for reporting anything, not a licence for a population claim. A cell of 24 families gives a finite-suite result: its verdict is a statement about the sealed suite. A population result needs at least 32 independent families at α = 0.05, and 46 where four confirmatory cells share the level, and the cell declares which of the two it is before sealing (§39.20).

The pooled ratio. For a cold cell, with the sum over all pairs j of the cell,

R̂ = Σ_j min(T_B,j, H) ÷ Σ_j min(T_A,j, H).

For a warm cell R̂ is R_warm(10) of §39.4, with ā and b̄ pooled the same way. The per-family ratio ρ_i is the same expression restricted to the pairs of family i.

The censoring rule. (a) A campaign not accepted by H contributes H. (b) A pair in which both arms are censored stays in both sums. (c) A harness fault that is not the arm's own is repaired by re-running the whole pair once under the next campaign index; a second harness fault excludes the pair for every arm, and it is counted. A cell with more than 5% of pairs excluded is descriptive only. (d) When arm B is censored, ρ_i is a lower bound on that family's ratio, which is conservative for arm A; when arm A is censored, ρ_i is at most 1. (e) A run is never dropped because its outcome is inconvenient, and an exclusion is decided before the pair's times are compared.

The interval. One-sided 95% bounds L and U on R come from a bias-corrected bootstrap that resamples mission families with replacement and keeps each family's pairs together, with 10,000 replicates, on ln R̂. The interval is reported with every ratio. It is approximate and descriptive, and it does not decide the word: resampling cannot show a slow tail that the sample does not contain, and on a 24-family suite drawn from a population whose true ratio is 1.82 a percentile bootstrap reports 20 in 79% of suites (§39.20). A population claim rests on the bounded paired contrast of §39.20 and never on this interval.

The exact guard. The count X of families with ρ_i > 5 is compared with a binomial(m, ½) tail: the null hypothesis "the median per-family ratio is at most 5" is rejected at level α when P(X ≥ x) ≤ α. The test is exact and distribution-free at fixed m. Its critical counts at α = 0.05 are 5 of 5, 6 of 6, 7 of 8, 10 of 12, 12 of 16, 17 of 24 and 22 of 32 (FACT: binomial arithmetic; for example P(X ≥ 17) = 0.032 and P(X ≥ 16) = 0.076 at m = 24). At m = 4 even 4 of 4 has tail 1/16 = 0.0625, so the guard cannot be passed: that verdict is unreachable, and a cell of four families is refused as a confirmatory cell.

The verdict rule for a cell (DESIGN). The rows are tested in order, and the first that applies is the verdict.

Verdict Condition
disqualified A ground of §39.9 applies to arm A in this cell
descriptive n < 10, or more than 5% of pairs excluded, or the cell was not named confirmatory, or the pre-analysis found the cell underpowered (KILL-TENFOLD-REACH)
withdrawn R̂ < 3; or, at the partial gate, the futility monitor stopped the cell
ten-fold R̂ ≥ 10 against the stronger of arm B and the composed incumbent (arm C, the W of §39.22); and the exact guard rejects at the cell's adjusted level; and arm A's completion and coverage are non-inferior to arm B's; and the ablation arm has not tied (§39.10); and the adjudication condition of §39.3 holds
measured Every other case: the report states R̂ with [L, U] and does not use the word

On a cell of fewer than 32 independent families the verdict ten-fold is a finite-suite result: the report says "ten-fold on the sealed suite" and uses no population wording. Population wording needs, in addition, a cell of at least the family count its registered bound requires and a nonnegative lower bound on the paired contrast of §39.20.

The floor of 5 in the guard comes from the first draft of this rule, whose lower bound had to exceed 5 and which never confirmed a true ratio of 5. The pooled rule keeps that property: a true ratio of 5 is called ten-fold with probability near zero in every design below. The threshold of 3 is a convention. It sits at the bottom of the range §2.6 expects for quadruped briefs, so withdrawal of that class at the partial gate is a live outcome and not a remote one.

Why the rule was re-specified. The first draft required, on at least four of six briefs including both held-out ones, a ratio of medians of at least 10 with a lower bound above 5, over five seeds. A Monte Carlo of that rule confirmed a true ratio of 10 in at most 18% of campaigns, 0 to 11% with five seeds, and reached 80% only near true ratios of 15 to 20 (FACT, Appendix K). The cause was compounding: every brief needed its own point estimate above 10, which a true 10 delivers half the time, and the requirement was multiplied across briefs. The pooled rule has one estimate per cell. No honest rule can confirm a true ratio of exactly 10 more than about half the time, because a fair measurement of 10 falls below 10 half the time; the pooled rule reaches that half, and reaches 80% at a true ratio near 11 to 12.

Pre-analysis (illustrative). The table is a planning computation under a stated model, not a result: lognormal times, a per-family difficulty spread of 0.5 in the logarithm, per-family log-ratio spread s, no censoring, one confirmatory cell, 20,000 simulated gates per entry. HYPOTHESIS: s lies between 0.5 and 0.8 on the design strata. The binding pre-analysis is recomputed from the development suite's measured times, with censoring at H and the multiplicity adjustment, and is recorded before the suite opens (GATE-TENFOLD-04).

Families m Spread s Guard needs P(ten-fold) at a true ratio of 5 / 7 / 8.5 / 10 / 12 / 15 / 20 P(withdrawn) at 2 / 3 / 5 True ratio at which P(ten-fold) reaches 0.8
24 0.5 17 of 24 0.000 / 0.002 / 0.08 / 0.50 / 0.94 / 1.00 / 1.00 1.00 / 0.50 / 0.000 11.25
24 0.8 17 of 24 0.000 / 0.03 / 0.20 / 0.49 / 0.83 / 0.98 / 1.00 0.98 / 0.50 / 0.005 12.0
12 0.8 10 of 12 0.002 / 0.05 / 0.19 / 0.39 / 0.65 / 0.89 / 0.98 0.93 / 0.50 / 0.03 13.75
6 0.8 6 of 6 0.006 / 0.05 / 0.12 / 0.23 / 0.38 / 0.57 / 0.77 0.86 / 0.49 / 0.08 above 20
4 any unreachable 0 at every ratio 0.81 / 0.50 / 0.13 never

Two readings. Between true ratios of 7 and 10 the word can be awarded by chance, in up to a fifth of gates at 8.5; that is the price of a threshold on a measured quantity, and it is why the interval is printed beside the word. And six families give a rule no better than the one that was rejected.

Reachability conditions for opening a cell. A cell is confirmatory only if its recorded pre-analysis shows: P(ten-fold) at least 0.8 at a true ratio of 15; P(withdrawn) at most 0.05 at a true ratio of 5; the guard reachable at its adjusted level; and the horizon cap H ÷ RMT_A on the development suite at least 10, since R ≤ H ÷ RMT_A whatever the rival does. A cell that fails any of these is opened as descriptive, and the report says so before any sealed run (KILL-TENFOLD-REACH).

Multiplicity. At most four confirmatory cells are named per gate (DESIGN). The exact guard is applied with Holm's step-down procedure at family level 0.05 across them, so the chance that any cell whose median per-family ratio is at most 5 is awarded the word is at most 5% for the gate as a whole. With four cells the first step needs 18 of 24. All other cells are reported with unadjusted intervals and cannot carry the word.

Completion and coverage. For the word, arm A's completion rate by H, and on family strata its share of mandatory obligations discharged, must be non-inferior to arm B's: the exact one-sided 95% lower bound on the difference A minus B is above −0.05 (DESIGN margin, frozen before sealing). Rare failures get an interval that is valid at zero counts: zero misses in 24 independent families still leaves a one-sided 95% upper bound of 1 − 0.05^(1/24) = 11.7% on the miss rate, where a bootstrap of 24 zeros would report zero width.

The monitor. One anytime-valid study per cell, declared separately, on the per-family ratios in their randomized run order. Its null hypothesis is "the median per-family ratio is at least 3". With Y_i = 1 when ρ_i ≥ 3 and 0 otherwise, the process E_n = Π (1 + λ·(1 − 2·Y_i)) with λ = ½ is a supermartingale under the null, and the cell is stopped for futility when E_n ≥ 20. Each family below 3 multiplies E by 1.5 and each family at or above 3 multiplies it by 0.5, so eight consecutive families below 3 stop the cell (1.5⁸ = 25.6) and seven do not (1.5⁷ = 17.1). At the partial gate a futility stop yields withdrawn on the monitor's own authority at level 0.05, and saves the compute of a cell that cannot succeed. At the tournament the stop is disabled: the monitor reports, and every cell runs to n, because the published record is worth its compute and a tournament is not stopped by an interim number. The monitor also carries the operational alarms: ledger closure, excluded pairs, a leak signal. No monitor output enters the fixed-n computation, and a request to compute the fixed-n verdict at a monitor's stopping time is refused, as §24 refuses it for any study.

Descriptive reporting, always. Time-to-acceptance curves per arm; completion probability by H; conditional times; the per-phase ratio of the two ledgers; every cost column; every false positive, false negative and abstention; the fastest qualifying baseline and all arms with raw pairings. A per-mission best-of-rivals ensemble may be shown as an extra lower bound and is labelled as one. Nothing is inferred about systems that were not run.

39.8 The ceilings

A speed test is run only after its ceiling has been computed from a measured ledger of the rival. Each ceiling is arithmetic, and each exists because a tempting comparison ignores it.

# Ceiling Statement
1 Removing a phase If a phase holds the share p of the rival's restricted time and a mechanism removes it entirely, the speed-up is at most 1/(1 − p). Stated on what remains, q = 1 − p, it is 1/q. Removing half the loop gives 2; removing 85% gives 6.7; ten needs p ≥ 0.9
2 Improving a part If the share f is improved by the factor r and the mechanism adds overhead h, the speed-up is at most 1/(1 − f + f/r + h). The three terms are measured on the rival's critical path
3 Unremovable cost If arm A's time has a floor u that no mechanism removes (authoring, the common agent's turns, adjudication, checking, serialization), then R ≤ RMT_B ÷ u. When u exceeds one tenth of the rival's time, a ratio of ten needs another stage changed or the claim changed
4 Completion If arm A misses the share q of deadlines, RMT_A ≥ q·H and RMT_B ≤ H, so R ≤ 1/q. With 12% unfinished the ratio cannot exceed 25/3 ≈ 8.33
5 Common latency With the same agent latency t per turn in both arms and a rival cost C per decision, deciding one design per turn gives at most (C + t)/t
6 The horizon R ≤ H ÷ RMT_A, because the rival's restricted time cannot exceed H

The funding rule. A mechanism whose removable cost is under 10% of the measured loop cannot be funded as a speed lever: by ceiling 1 its whole effect is at most 1/0.9 = 1.11. It may still be funded for what it does to trust, coverage or correctness, with that reason written in its work package and no speed figure attached (KILL-TENFOLD-LEVER). The rule is applied to the rival's ledger at M1 and again with our own profile at M2.

The order rule. A pruning ratio is never multiplied by a speed-up measured on work the pruning already removed. Each factor in a product of levers is measured on the time that the earlier factors left.

The refusal. A registered speed comparison records p or (f, r, h) from a ledger-closed run of the strong baseline, and is refused when its ceiling is below the ratio it sets out to show (GATE-TENFOLD-09). FACT, Appendix K: in one planning pilot the faster arm's batch held a residual of 575.7 µs that the solver does not touch (parsing, assembly, checking, serialization and harness, not separately timed), so a solver of zero cost could have reached about 1.46 at most (842.8 ÷ 575.7); the measured ratios were 1.345 warm and 1.088 cold, and the speed gate of that pilot failed. A ceiling computed first would have refused the test.

39.9 Disqualification

Correctness is a gate and not a term in a score. Two events disqualify, and both apply to every arm alike.

One false certificate. A certificate that an arm submitted, that its declared checker accepted, and that the adjudicator's direct evaluation or any later audit shows false within its stated scope. The arm is disqualified from the scoped claim: every cell of the gate in which it used that proof kind has no ratio for that arm, and the verdict of the cell is disqualified. For arm A the procedure of §25 follows as well: the proof kind is suspended, its cells fall to Tested, and KILL-AUGUR-FALSECERT decides whether the rule or its implementation was at fault. A truthful unknown, a refused certificate and a certificate with a narrower scope than was asked are not false certificates; they cost coverage and time.

Any leak from sealed confirmation worlds. Any read, by an arm's agent, tools or developers, of sealed content before the gate closes: a mission card before its clock starts, a sealed world address, a per-world outcome, a probe point, another arm's transcript. Detection uses the firewall audit of §24 and §37, access logs, and canary values that the adjudicator plants in sealed worlds and that must never appear in an arm's transcript. The cells touched by the leak are void for that arm, the families are retired at once, and if the extent cannot be bounded the whole gate is void and is re-run on a rotated suite. A leak is never repaired by excluding the affected runs and keeping the rest of the cell.

Also disqualifying. A change to a frozen arm after its freeze; use of the adjudicating simulator in development or tuning; an edited ledger or receipt. A breach of a resource cap is not a disqualification: the run is censored at the instant of the breach and costs H.

What a disqualification does not do. It does not award anything to the other arm. A cell whose rival is disqualified has no denominator, is reported as such, and is re-run at the next gate.

39.10 The outcome rules, written in advance

The thresholds 3 and 10, the guard floor 5, the tie factor 1.5 and the margin 0.05 are fixed when the first suite hash is committed, and none is changed after any sealed run. Each outcome is recorded as an ADR in the milestone in which it occurs. A kill outcome is not re-tested on the same suite.

Outcome on a confirmatory cell What the report says What the program does
withdrawn: R̂ below 3, or a futility stop "Measured R̂ with [L, U] on ⟨stratum, regime⟩. The speed claim for this class is withdrawn." KILL-TENFOLD-SPEED. The class leaves the speed claim in §2, the README and every report. Work packages for the class whose only justification is speed are re-planned before the next milestone starts. The class may be measured again at M7 on fresh sealed families
measured: R̂ from 3 up to 10, or 10 and above without the guard "⟨R̂⟩ times shorter, and at least ⟨L⟩ with 95% confidence, on ⟨stratum, regime⟩." The word "ten-fold" is not used for the class anywhere The ledger names the binding ceiling. Lever work for the class continues only on phases whose removable share is at least 10%
ten-fold "Ten-fold on ⟨stratum, regime⟩: measured R̂ with [L, U]", with S and N* for a warm cell BET-13 is met for that stratum and for no other
Tie with the ablation arm: arm C reaches the same accepted scope and RMT_C ÷ RMT_A is at most 1.5; or the recovery ratio of §39.22 is at least 0.8 on elapsed time and on compute "A short program over ⟨engine⟩ reached the same accepted answers within a factor of 1.5", or "a layer over ⟨engine⟩ recovers ⟨r_T⟩ of the saving in time and ⟨r_C⟩ in compute." KILL-TENFOLD-TIE. Say so. Ship the layer over that engine for that stratum, behind the same verbs, schemas and checker. Keep engine work only for the gaps that are internal to an engine. Two are documented: a mixed-topology batch with run-to-run replay, and the Metal tier. Two are unverified and are checked before they are relied on: replay of one world outside its batch, and solver-internal certificates. Spend the research budget on the first mission the program cannot resolve. The ratio against arm B is then the layer's ratio, and no claim is made that the mechanisms need this engine
Engine advantage not established: RMT_C ÷ RMT_A above 1.5 with its lower bound at or below 1.5 "The advantage of the engine over a layer was not established." The layer is offered beside the engine for that stratum. BET-14 is neither met nor failed there, and is re-tested at the next gate
The rival is faster: R̂ below 1 The measured ratio As withdrawn. Where licences allow, the rival's method is adopted behind the verbs
descriptive The numbers, with the reason no verdict was possible Nothing is cited as passing or failing BET-13. The cell is enlarged for the next gate
disqualified The event, its scope and its evidence §39.9

Across strata. Each stratum is its own result. A portfolio ratio with equal weights over the strata run is reported as descriptive. BET-13 is met when at least one named stratum has the verdict ten-fold. Whatever the verdicts, the table of expected ranges in §2.6 is replaced by the measured table in the next version of this plan.

What remains valuable if the speed claim is withdrawn. The claims that never depended on speed: any world can be re-run alone on the CPU from its address (§29); comparisons carry error control that survives an agent's retries (§24); family answers are checked by an auditor that shares no code with their producer (§12); failures arrive as minimal replayable worlds with an interventional explanation (§26, §27); every result names its claim, its class and its evidence (§8); and the energy ledger closes (§29). Under 3 the product is described by those and by its measured ratio, and the comparison with incumbents is stated as parity of speed with a different kind of evidence.

39.11 The incumbent parity baseline at M0 and M1

The denominator is measured before any mechanism is funded against it. Incumbents are frozen first.

At M0 (GATE-TENFOLD-01). The rival audit pins every incumbent of §39.5 by revision, records what each one documents that it can do, and freezes a warm configuration for each after tuning on reference tasks only. Then the phases that need no agent are timed on trj: cold start; a parameter edit and a topology edit to the first device step; and training to the return threshold on one card and on two. These are the trj rows of GATE-SPARK-21 that need no agent, and this gate reads them into the first phase ledger. FACT (Appendix H) for the last: about 410 s on one RTX 4090 and about 325 s on two for a flat-ground quadruped policy, with a compile region of about 45 s inside those times. Each published figure that §2.2 uses is reproduced within 20%, or the difference is explained in writing and the measured figure replaces the published one. The audit's pins also serve the instruments that other chapters borrow from incumbents: the incumbent arms of the mixed-batch head-to-head of §17 and the proxy-adequacy pilot of §19.

At M1 (GATE-TENFOLD-02, with GATE-SPARK-21). GATE-SPARK-21 completes the table of incumbent rates that §36 needs. GATE-TENFOLD-02 runs the composed rival with the common agent, end to end, on at least 12 development missions of legged-quad and wheeled, and records closed ledgers. The table of §2.2, which composes published figures from different stacks and is not a measurement, is replaced by measured rows. Every ceiling of §2.3 is recomputed. Every lever of §2.4 receives its removable share p on the measured ledger, and a lever with p under 0.10 is struck from the speed case before M2 begins.

What the baseline is allowed to change. The ranking of the levers, the funding of work packages, the deadlines H, and the expected ranges of §2.6. It is not allowed to change the estimand or the outcome rules. If a physics row of §36 later falls below 0.33 of the incumbent (KILL-PARITY), the arithmetic here is redone with measured rates before the partial gate, and whether to ship the layer is decided at that gate under §39.10.

39.12 The partial gate at M3

Scope. The cells that the M3 product can run: (legged-quad, cold) and (wheeled, cold) as confirmatory cells; (envelope, warm) and (sensing, cold) as confirmatory cells if their pre-analyses allow it, and as descriptive cells otherwise. Tracked and hybrid briefs wait for M4 and are first measured at M7. Arms A, B and C all run.

What arm A has at M3. The daemon with slot dispatch, mixed batches, the baseline controllers and the sampling fallback, the ladder with fixed allocation, the qualification lane, Nemesis and Autopsy, the memo graph, and the mission queries on the CPU. It does not yet have family policies, inheritance or rank-agreement allocation, which land at M4. The partial gate therefore measures the product without lever 2. HYPOTHESIS: on legged-quad the ratio at M3 lies near the bottom of the range of §2.6. The rules of §39.10 apply all the same, and a class withdrawn at M3 stays withdrawn until M7 measures it on fresh families.

What else is decided on its record. KILL-AUGUR-TIE and KILL-FAMILYCERT on the envelope and sensing strata; KILL-GYMADAPT; the incumbent arm that BET-10 names; and whether the train phase still holds a removable share of at least 10% for each class, which decides how much of the M4 work on lever 2 is funded as a speed lever.

Cost. A cell costs m·c·Σ over arms of E[min(T, H)] device-hours. With 24 families, three arms and a deadline of 4 h the worst case is 288 device-hours per cell, reached only if every run is censored. HYPOTHESIS: about 70 device-hours per design cell at restricted mean times near 1.5 h, 1.0 h and 0.4 h for arms B, C and A. That is above the default envelope class of §36.14, so the gate needs the owner's approval of a campaign class for each run, and of the pretraining class where an arm builds a standing asset (OPEN-TENFOLD-4).

Passing. GATE-TENFOLD-06 passes when the record exists and the outcome rules have been applied, whatever the ratio. M3 does not exit without it.

39.13 The tournament at M7

Every stratum of §39.4 that has a product to run, on sealed families that no earlier gate used, with every pin of every arm and of every sibling frozen for its length (§33), because a ratio of times measured across a pin move is not one measurement. The order of work is: refresh the rival audit; publish the enlarged development pool, which now holds the families the partial gate retired; commit the suite and protocol hashes; tune; freeze and publish every arm; record the pre-analysis and name at most four confirmatory cells; run with the arms interleaved; close and reveal; publish the report and the kit; and have an external party re-run the kit. DESIGN: adjudication at M7 is external, or the word is not used until the external replication has passed. No cell of the tournament is stopped on an interim number. A tournament is voided and re-run only for a breach of protocol: a pin moved during the freeze, sealed material read by an arm, or an arm changed after its freeze.

The report is one table per cell (R̂, [L, U], the verdict, completion and coverage for each arm, the tie test against arm C, N* and S for warm cells, the adjudication label) and one phase ledger per arm per cell, with the ceilings of §39.8 evaluated on the rival's ledger. GATE-TENFOLD-07 passes when that report is published with its ledgers and its kit, whatever it says.

39.14 The replication kit

A gate's result is published with everything needed to recompute it and to re-run it.

  • Contents. The pre-registration and the pre-analysis. The mission cards, the world generators and the revealed keys. The referee protocol and the adjudicator's acceptance code. Every freeze manifest, with the scripts, wrappers, tag maps and agent guides it names. Every ledger, receipt and agent transcript. The scorer, which, like frob-verify, depends on nothing outside the standard library. Hardware fingerprints and disclosure blocks (§21, §36). A manifest of hashes over all of it.
  • Level 1, recompute. From ledgers and receipts alone, the scorer reproduces every published ratio, interval, count and verdict, bit for bit, on a clean machine.
  • Level 2, re-run. From the frozen manifests, on hardware with a comparable fingerprint, one sampled pair per stratum is run again for every arm. TARGET: the acceptance outcome is the same and the time is within a factor of 1.5 in at least 90% of the sampled pairs.
  • Limits. Incumbent code is not redistributed: the kit pins revisions and gives fetch instructions. A private scene or dataset cannot enter a kit, and a kit built from one is refused (§37). Every exported artifact carries the no-authorization flags, all false.

39.15 What the test does not measure

  • Reality. Acceptance happens in simulation. The adjudicating simulator is a third model, not the world, and every report of this test prints reality: no-claim unless a band of §30 covers the mission.
  • How good the best design is. The clock stops at the first accepted answer. Quality beyond the acceptance threshold is not scored, except where a mission states an optimality gap.
  • People. Both arms are driven by one agent. Human engineering time, learning curves and usability for a person are outside the test, and the ratio is conditional on the pinned agent.
  • Money. Cost columns are reported and no cost claim is made.
  • Systems that were not run. Nothing is inferred about an incumbent, a version or a configuration outside the frozen arms, or about missions outside the sealed population. Manipulation with hands, flight and soft bodies are outside the suite.
  • The cost of building either system. Standing assets for a mission class are charged; the engineering that produced FrankenRobotics or an incumbent is not.
  • Step rate, rendering and safety. Throughput is §36's. Rendering is outside every claim here. Nothing in this test is a safety statement about a machine.

39.16 Worked card: from a phase ledger to a ratio

Every number on this card is illustrative. It shows the arithmetic and is not a forecast.

One pair, cold. A legged-quad mission with H = 14,400 s. Exclusive ledgers in seconds:

Phase Arm B Arm A Note
author 900 600 Cold: both write the environment from the card
propose 1,080 360 Twelve agent turns against four, 90 s each
check 240 20
compile 540 12 Twelve edits at 45 s against cached variants
screen 0 96 Arm A screens a family of designs per turn
train 4,920 150 Twelve trainings at 410 s against one fine-tune of a finalist
evaluate 360 60
qualify 0 40
diagnose 600 30
wait 180 42
adjudicate 180 90 Two submissions against one
Clock T 9,000 1,500 Ratio for this pair: 9,000 ÷ 1,500 = 6.0

Three ceilings read off the rival's column. Training is 4,920 ÷ 9,000 = 54.7% of its loop, so removing training entirely gives at most 9,000 ÷ 4,080 = 2.21. Arm A's floor on this mission is the time no mechanism removes: author, propose and adjudicate, 600 + 360 + 90 = 1,050 s, which is 11.7% of the rival's clock, so the cold ratio on this mission cannot exceed 9,000 ÷ 1,050 = 8.57 whatever the engine does. And check is 2.7% of the rival's loop: a mechanism that only removes it is worth at most 1/0.973 = 1.03 and cannot be funded as a speed lever.

The pooled ratio over six families. Times in seconds; "c" marks a censored run, charged H.

Family T_B T_A Per-family ratio ρ
1 9,000 1,500 6.00
2 6,300 900 7.00
3 14,400 c 2,100 6.86, a lower bound
4 5,400 700 7.71
5 11,700 14,400 c 0.81
6 7,200 800 9.00
Sum 54,000 20,400 R̂ = 54,000 ÷ 20,400 = 2.65

Five of the six per-family ratios lie between 6 and 9, and the pooled ratio is 2.65, because the one mission arm A never finished costs it 14,400 s, more than the other five together. The completion ceiling says the same: with one miss in six, R̂ ≤ 54,000 ÷ 14,400 = 3.75 on these rival times, and at most 1/q = 6 on any. A scorer that dropped family 5 would report 42,300 ÷ 6,000 = 7.05, and one that averaged the per-family ratios would report 6.23; both are planted wrong scorers in the harness tests. With six families there is no verdict (n is below 10, and the guard would need 6 of 6 above 5 where only 5 are). Had these proportions held in a registered cell, the verdict would be withdrawn.

Warm, with a standing asset. One amendment of the same mission takes arm B 4,050 s (six turns, six trainings) and arm A 470 s (two turns, one inherited fine-tune): a marginal ratio of 8.62. Arm A's family policy for the class cost S_A = 14,400 s to pretrain, and arm B has no standing asset. Then R_warm(1) = 4,050 ÷ 14,870 = 0.27, R_warm(10) = 40,500 ÷ 19,100 = 2.12, R_warm(100) = 405,000 ÷ 61,400 = 6.60, and the limit is 8.62. The crossover reuse count is N* = 5, since 14,400 + 470·N ≤ 4,050·N first holds at N = 5. The warm verdict is taken at N = 10, where this class measures 2.12: below 3 after ten reuses, above 6 only after a hundred.

39.17 Invariants and known answers of the harness

ID Invariant
INV-TENFOLD-01 The builder, rival-advocate and adjudicator roles are disjoint for the life of a gate, for people and for agent sessions
INV-TENFOLD-02 The six sealing steps of §39.4 appear in the ledger in order, each with a hash and a time
INV-TENFOLD-03 No paired run exists whose arms were not all frozen and published before it started; no arm changes after its freeze
INV-TENFOLD-04 Exclusive phases sum to the clock within 2%, or the run is flagged ledger-open and leaves ceiling arithmetic
INV-TENFOLD-05 No run is dropped; a censored run contributes H; an exclusion is a harness fault recorded before times are compared
INV-TENFOLD-06 A sealed family serves one gate and is then retired
INV-TENFOLD-07 No output of a monitor enters the fixed-n verdict, and the verdict is computed once, at n
INV-TENFOLD-08 Every reported ratio carries its regime, m, n, H, [L, U], the adjudication label and the freeze hashes; a warm ratio also carries S and N*

Known answers owned by this chapter (numbered in §38):

Known answer Expected Guards against
The six-family card R̂ = 2.647; the completion ceiling 3.75; per-family ratios 6.00, 7.00, 6.857, 7.714, 0.8125, 9.00 Averaging ratios (6.23) and dropping a censored run (7.05)
The single-pair ledger 6.0; training removed 2.206; unremovable floor 8.571 A ceiling computed on the wrong arm's ledger
The warm card R_warm at 1, 10, 100: 0.272, 2.120, 6.596; limit 8.617; N* = 5 Quoting the limit as the warm ratio; omitting a standing asset
Exact guard Critical counts 5 of 5, 6 of 6, 7 of 8, 10 of 12, 12 of 16, 17 of 24, 22 of 32 at 0.05; 18 of 24 at 0.0125; no count at m = 4 An approximate test; a verdict that cannot be reached
Zero-count bound 1 − 0.05^(1/24) = 0.1174 A zero-width interval from 24 zeros
Completion ceiling 12% unfinished caps R at 25/3 A ratio claimed above its ceiling
Monitor Stops at eight consecutive families below 3 (25.6) and not at seven (17.1) An early stop without anytime validity

Negative controls, each of which the harness must catch: a planted false certificate (the arm is disqualified); a canary value from a sealed world in a transcript (the cell is void); a ledger with 3% unattributed (flagged, and excluded from ceilings); a manifest whose freeze time follows the suite opening (the run is refused); a request for a confirmatory cell with four families (refused as unreachable); a request for the fixed-n verdict at a monitor's stopping time (refused); an arm that touched the adjudicating simulator during tuning (disqualified).

39.18 Gates and kill gates

Gate Claim Threshold (TARGET) Earliest
GATE-TENFOLD-01 Parity The rival audit and the no-agent baseline of §39.11: every incumbent pinned with its capability table and licence; warm configurations frozen; cold start, edit to first step and training to threshold on one and two cards timed on trj with closed ledgers (the trj rows of GATE-SPARK-21 that need no agent); each published figure used in §2.2 reproduced within 20% or the difference explained in writing M0
GATE-TENFOLD-02 Performance The rival loop with the common agent on at least 12 development missions of legged-quad and wheeled: closed ledgers; the table of §2.2 replaced by measured rows; a removable share and a ceiling recorded for every lever of §2.4 M1
GATE-TENFOLD-03 Known answer The harness reproduces every known answer of §39.17 exactly and catches every negative control M3, before the suite opens; repeated at M7
GATE-TENFOLD-04 Known answer Freeze and seal: the six steps in order; a published manifest and advocate statement for every arm; the frozen rival reproduces its home-task training time within 20% on trj and wins at least one easy-win development family per stratum; the pre-analysis recorded, with every confirmatory cell meeting the reachability conditions of §39.7 M3; M7
GATE-TENFOLD-05 Known answer Adjudicator acceptance: the adjudicating simulator passes the known answers of §38 that it can express, inside its own stated tolerances; of 20 planted answers per stratum (10 that must pass, 10 that must fail) it classifies 20 correctly; no arm's logs show it in training or tuning; on a 5% sample of accepted answers the discordance with each arm's home referee is reported M3; M7
GATE-TENFOLD-06 Performance The partial gate of §39.12: every cell in scope has its ratio, interval, verdict, completion, coverage, tie test and one phase ledger per arm, and the outcome rules of §39.10 have been applied. It passes when that record exists, whatever the ratio M3
GATE-TENFOLD-07 Performance The tournament of §39.13: the same record for every stratum run, with every pin frozen for its length, published with its ledgers and its kit. It passes when the report is published, whatever it says M7
GATE-TENFOLD-08 Known answer The replication kit: level 1 reproduces every published number bit for bit on a clean machine; level 2 re-runs one sampled pair per stratum with the same acceptance outcome and a time within a factor of 1.5 in at least 90% of pairs; a kit with a private input is refused M7
GATE-TENFOLD-09 Performance Ceiling before speed, program-wide: every work package funded as a speed lever cites a removable share of at least 10% on a ledger-closed run of the strong baseline, and every registered speed comparison records its ceiling and is refused when the ceiling is below the ratio it sets out to show M1, then every exit
Kill gate Trigger Fallback, named in advance Decided
✖ KILL-TENFOLD-SPEED On a confirmatory cell R̂ is below 3, or at the partial gate the futility monitor stops the cell The speed claim for that class is withdrawn from §2, the README and every report; speed-only work packages for the class are re-planned; the product is described for that class by the capabilities of §39.10 that do not depend on speed, and by its measured ratio M3 for the classes run; M7
✖ KILL-TENFOLD-TIE On a stratum, arm C reaches the same accepted scope with a restricted mean time within a factor of 1.5 of arm A's; or the composed incumbent's recovery ratio is at least 0.8 on elapsed time and on compute (BET-14, §39.22) Say so; ship the layer over that engine for that stratum behind the same verbs, schemas and checker; keep engine work for the engine-internal gaps only (two documented, two unverified: §39.10); spend the research budget on the first mission the program cannot resolve M3; M7
✖ KILL-TENFOLD-CERT One false certificate from an arm That arm is disqualified from the scoped claim; for arm A the proof kind is suspended and KILL-AUGUR-FALSECERT applies Standing, from M3
✖ KILL-TENFOLD-LEAK Any leak from sealed confirmation worlds The affected cells are void and their families retired; if the extent cannot be bounded the gate is void and is re-run on a rotated suite Standing, from M3
✖ KILL-TENFOLD-LEVER A mechanism's removable cost is under 10% of the measured loop It is not funded as a speed lever; it may be funded for trust, coverage or correctness with that reason written and no speed figure M1 on the rival's ledger; M2 on ours
✖ KILL-TENFOLD-ADJ No neutral simulator passes GATE-TENFOLD-05 for a stratum Cross-home adjudication, labelled; if neither home referee can run the other arm's answers, the stratum is descriptive M3; M7
✖ KILL-TENFOLD-REACH A cell's pre-analysis fails a reachability condition of §39.7 The cell is opened as descriptive and says so before any sealed run; it is enlarged for the next gate M3; M7

39.19 Open questions

  • OPEN-TENFOLD-1. Which simulator adjudicates each stratum, and whether it can represent terrain, films and tracks. Owner: the adjudicator with the Gauntlet lead. Blocks GATE-TENFOLD-05. Default: a pinned incumbent CPU engine that no arm trains in; where it cannot represent the mission, cross-home adjudication, labelled. §42 carries this question and the next as one, OPEN-PROG-2.
  • OPEN-TENFOLD-2. Who the adjudicator is. Owner: the project owner. Decided before any sealed world family exists. Default: a fresh-context session with no write access to any arm; every ratio is then labelled self-adjudicated, and the word waits for an external re-run of the kit.
  • OPEN-TENFOLD-3. Who the rival advocates are, and their budget. Owner: the project owner. Blocks GATE-TENFOLD-04. Default: agent sessions that have contributed nothing to FrankenRobotics, 40 device-hours per stratum, and the label rival: internally advocated on every result.
  • OPEN-TENFOLD-4. The compute envelope for the partial gate and the tournament under the classes of §36.14: a campaign class for each run, and the pretraining class where an arm builds a standing asset. Owner: the project owner. Blocks GATE-TENFOLD-06 and -07. Default: nothing above the default class runs, the gate is recorded as not run, and M3 does not exit.
  • OPEN-TENFOLD-5. The deadline H of each stratum. Owner: the adjudicator. Decided at GATE-TENFOLD-04 from the development suite. Default: three times the frozen rival's median development time, rounded up to half an hour, and raised until the horizon cap is at least 10.
  • OPEN-TENFOLD-6. The agent model, its revision and its caps. Owner: the project owner. Blocks every freeze. Default: the agent the project itself is developed with at the gate's date, pinned; one agent only, with the result stated as conditional on it.
  • OPEN-TENFOLD-7. Incumbent licences: whether each permits publishing a comparison, and what the kit may carry. Owner: the project lead. Blocks GATE-TENFOLD-01. Default: revisions and fetch instructions only; an incumbent whose licence forbids publication is replaced by the next strongest, and the replacement is printed.
  • OPEN-TENFOLD-8. Ratification of the thresholds 3 and 10, the guard floor 5, the tie factor 1.5, the margin 0.05 and the limit of four confirmatory cells. Owner: the project owner. Blocks the first suite hash. Default: as written.
  • OPEN-TENFOLD-9. Whether arm D, the product's own layer over an incumbent engine, is built. Owner: the Tether lead. Blocks nothing. Default: not built unless arm C ties on a stratum.

39.20 Estimating the ratio, and the smallest suite that can certify

What it is. S The inference rules for the ratio of §2.1, fixed before the sealed suite is opened: what is estimated, which procedure is invalid, and how many independent mission families a population claim needs.

Two estimands, never interchanged. The restricted-time ratio is R_time = Σᵢ min(T_rival,i, H) ÷ Σᵢ min(T_frob,i, H). The accepted-output rate ratio is R_rate = (accepted_frob ÷ makespan_frob) ÷ (accepted_rival ÷ makespan_rival). Known answer: 25 serial missions with H = 100 s; the rival takes 50 s on each (1,250 s, 25 accepted); the candidate finishes 24 in 1 s each and times out once (124 s, 24 accepted). Then R_time = 1,250/124 = 10.081 and R_rate = (24/124)/(25/1,250) = 9.677. The word ten-fold attaches to R_time only; a statement about decisions per hour cites R_rate. With parallel jobs the makespan is the measured campaign makespan under matched resource caps. Each arm's independent acceptance is timestamped before the joint reveal.

The unit of inference is the independent mission family. Missions that share a family, an originating site or a material batch form one cluster; 24 rows from one site are one site replicate. Connected groups stay together across the development and sealed splits. The resampling unit, H and the cluster structure are registered with the suite.

A procedure that is invalid here. Let H = 1,000 s, the rival always take 20 s, and the candidate take 1 s on 99% of families and 999 s on 1%. Both always complete. The population ratio is 20/10.98 = 1.8215. A suite of 24 families contains only fast missions with probability 0.99²⁴ = 0.78568, and then every percentile-bootstrap resample reports 20. Resampling cannot show a tail that the sample does not contain.

The test. For the claim R_time ≥ 10, form the bounded paired contrast Xᵢ = min(T_rival,i, H) − 10·min(T_frob,i, H), which lies in [−10·H, H]; the claim is E[X] ≥ 0. For n independent families a distribution-free one-sided lower bound is X̄ − 11·H·√(ln(1/α)/(2n)). The claim is made only when a registered valid bound is nonnegative.

How many families a population claim needs. DESIGN (derived; hand-checkable). If a fraction p of families has X = −10·H and the rest have X = H, then E[X] = H·(1 − 11p), which is negative for p > 1/11, and a suite of n families shows no slow family with probability (1 − p)ⁿ. A procedure with one-sided error α that reports a nonnegative bound on the best possible sample must therefore have (10/11)ⁿ ≤ α, so n ≥ ln(1/α)/ln(1.1): 32 families at α = 0.05, 46 at α = 0.0125 (four strata under a Bonferroni split), 49 at α = 0.01. The bound of the previous paragraph needs n ≥ 121·ln(1/α)/2, which is 182 at α = 0.05, even on that best sample. With 24 families, zero slow missions is compatible at the 5% level with a slow-family rate of 1 − 0.05^(1/24) = 11.7%, and 11 × 0.117 exceeds 1.

Consequence. Each stratum declares one of two result types before sealing. A population result has at least the family count that its registered bound requires, and never fewer than ln(1/α)/ln(1.1). A finite-suite result states the ratio on the sealed missions as an exact fact about that suite, with its completion table and its phase ledger, and carries no population wording. Completion and useful coverage are tested as paired differences against non-inferiority margins frozen with the suite. A rare binary failure is reported with an interval valid at zero counts (0 of n gives the upper bound 1 − α^(1/n)); a bootstrap of n zeros has zero width and is not used.

eval.preanalyze accepts a ten-fold study plan (n, cluster structure, H, α, result type) and prints the reachable verdicts. It refuses a population result whose n is below the necessary count (refusal: suite too small for a population claim). Owner: frob-arena; the harness is in frob-conformance.

GATE-TENFOLD-10 Known answer. The pre-analysis reproduces 10.081, 9.677, 1.8215, 0.78568 and the counts 32, 46, 49 and 182. In 2,000 simulated campaigns of 24 families with the rare-tail candidate, the contrast test certifies ten-fold in none, and the percentile bootstrap, run as the negative control, certifies it in 78.6% ± 3 points. M3, before the partial gate.

39.21 Rival arms, protocol controls and the sealed suite

What it is. S The parts of the protocol that keep the comparison fair to the rival and hard on the candidate.

Rival classes, and what must be earned against each. FACT (Appendix H; each named system and its revision is re-read and pinned at the rival audit below).

Problem Strongest rival to compose What the candidate must earn
GPU learning with sensing, control and transfer The leading batched engines of §1.1 with their training stacks and system-identification workflows More accepted designs per unit time, counting setup, training and independent evidence
Industrial and rich-world simulation The industrial stacks and the batched manipulation engines of §1.1 A checked family answer across several contact or sensing regimes at competitive total cost, at fixed scene diversity and sensor resolution
Joint synthesis of body and control Grammar-based morphology search, differentiable co-design and learned design generators (RoboGrammar, DiffHand, N-LIMB, DiffuseBot) A cheaper design under the same sealed constraints, or stricter checkable coverage at matched cost; the rival receives the same grammar, masses, symmetry knowledge and controller budget
Certified geometry, adversarial scenes, robust design Certified configuration-space regions; a scenario language (Scenic); linear and mixed-integer programs; interval, reachability and satisfiability tools More of the mission decided with correct quantifiers and interpretable counterexamples
Digital twins and systems engineering A multibody systems-engineering tool with a skilled author; an agent-built twin Less engineering time from a changed mission to a reconciled portable answer

DESIGN: the most serious rival may be a short analytic program wrapped around a mature simulator. It is always an arm.

The rival audit. Before any sealed mission is opened, each rival arm is frozen. Its configuration, exact revisions, licence, wrappers, tolerances and optimizer scripts are committed to the experiment pack, after a competent baseline engineer has improved the fastest rival on the development fixtures. Each arm receives the same calibration data, hardware envelope, reusable prior answers, cache budget and access to the checker.

Controls. Missions are paired across arms with isolated transcripts. Order, operator, site and resource assignment are randomized and counterbalanced. Agent arms start from fresh contexts on the same model revision, with the same caps on model calls and parallel workers; human operators are matched for proficiency. An adjudicator who does not know which system proposed an answer releases paired receipts after both arms close. Where a platform lacks a native path, a best-supported trial and a matched-device trial both run, and a missing cell is published as unavailable. The fastest qualifying rival, all arms and the raw pairings are published; a per-case best-of-rivals ensemble is reported only as a labeled extra lower bound on the ratio.

A second claim with its own acceptance: a new engineering operation. A fresh team receives a typed continuous design family, realistic uncertainty, a causal controller contract and sealed changes. Acceptance requires (i) a machine-checkable inner or outer answer with explicit scope, (ii) a physically interpretable witness or a scope-preserving ambiguity report, and (iii) correct reuse or withdrawal after an amendment. Each mission predeclares its mandatory interior, exterior and boundary obligations, a largest unresolved band or optimality gap, and any mandatory distinguishing experiment. Strict and closed endpoints are part of acceptance, and a vacuous enclosure fails. The rival's answer goes through the same checker. A tie retires the claim of exclusive expressivity on that fixture, and can still justify shipping the simpler implementation behind the same interface.

The sealed suite. Six mission types, each with what the verifier changes privately and the strongest control.

Sealed mission Private changes Strongest control
Adjustable stance under static load (§25.13) Foot authority, a force floor, package dimensions, asymmetric payload, a missing foot, passive-spring compatibility A lifted robust linear program; conventional projection; a force-law switch
Split-grip landing and the last usable observation (§24.11, §25.15) Mirror the strip; hide or delay the surface cue; shift the payload; change the contact order Belief-state reachability; equal-budget reinforcement learning; geometric control
Gripping unfamiliar compliant stock (§25.14) Geometry; an early-law alias; contact coupling An expert probe; direct measurement at mission dwell; active system identification
Changing an accepted carrier or walker (§28.13) Replace a gearbox, a wheel or a lower limb; perturb latency, friction or energy reserve Retraining; a shared conditioned actor; warm-started control; ordinary graph caching
Recommissioning a facility (§25.17) Move a door or a counter; perturb scan uncertainty; reduce a shared resource; alter a shift Recomputation; incremental mixed-integer programming; standard motion planning; selective replay with cached bounds
A heterogeneous high-throughput family (§17) Mix fixed and changed topology; grazing contacts; dense depth sensors; pressure to spill to the host An optimized batched incumbent; this engine with family sharing switched off

Sealed cases include routine easy wins for conventional methods and answers that are genuinely unresolved. Some worlds come from an evaluator independent of the proposing engine, because a rollout and a solver that share one mistranslated model agree on the wrong answer. Independently derived mutations of units, contact, causal clocks and passive load are planted. Warm campaigns use predeclared amendment sequences; one-time authoring is amortized over 1, 10 and 100 amendments; failed amendments stay in the sequence; the consequential edit is concealed; missed withdrawals and unnecessary invalidations are both counted.

GATE-TENFOLD-11 Known answer. The rival audit is complete before sealing: every arm's frozen configuration reproduces its own development-fixture figure within 10% (TARGET) on the tournament hardware; the baseline engineer's sign-off and the digest of the committed pack are recorded; no arm's configuration changes afterwards, and a changed digest voids the arm. M3 for the partial gate; M7.
GATE-TENFOLD-12 Known answer. Suite hygiene: every stratum holds at least one routine easy win and one unresolved answer; the checker rejects 100% of the planted unit, contact, clock and passive-load mutations; in the amendment sequences an arm keeps its scoped claim only with zero missed withdrawals, and unnecessary invalidations are reported per arm. M7.

39.22 Three arms, the recovery ratio and the phase ablations

Three arms S. One ratio cannot answer the two questions the test is asked: how far this system is ahead of the incumbent pipeline as it is commonly run, and how much of that lead an incumbent engine recovers once every portable method is layered over it. Every stratum is therefore run with three arms on the same sealed missions, acceptance protocol, hardware and resource caps.

Arm What it is Frozen by
B, the declared incumbent A leading incumbent simulator with per-design reinforcement learning and a standard outer-loop optimizer Its own advocates, on development fixtures
W, the composed incumbent The same engine or engines with every portable method layered on: native state restore and forks, per-topology buckets or the engine's own heterogeneous solver, a sampling controller or a multi-embodiment prior with checkpoint resume, racing and stopping, importance-weighted failure search, a second engine as referee A competent baseline engineer (§2.1), on development fixtures
F, this system The release under test The release

With T_X the restricted mean time of arm X on a stratum and C_X its GPU-hours, the recovery ratio of the composed incumbent is

r_T = (T_B − T_W) / (T_B − T_F)        r_C = (C_B − C_W) / (C_B − C_F)

defined when F beats B on that measure. A value of 1 means the layer recovers the whole saving and the engine adds nothing on that stratum; 0 means the layer recovers nothing. If T_F = 0.10·T_B, then r_T ≥ 0.8 exactly when T_W ≤ 0.28·T_B. Three cases are fixed before any measurement. If T_F ≥ T_B the stratum is reported as "no saving to recover" and r is undefined. If T_W < T_F then r exceeds 1 and the stratum is a loss to the composed incumbent. A mission unresolved at the deadline H costs H in every arm.

Decision rules (TARGET thresholds, registered before the sealed suite opens):

  • r_T ≥ 0.8 and r_C ≥ 0.8 on a stratum: a thin layer over an incumbent is enough there. The report says so and the fallback of BET-14 applies to that stratum.
  • Both below 0.5: the engine is a source of the gain on that stratum.
  • Otherwise the stratum is reported as mixed, with the phase the layer could not recover named from the ledger.
  • The word "ten-fold" is governed by F against the stronger of B and W (§2.1), never by F against B alone.

One receipt first. FACT (Appendix H): in the incumbent documents read, run-to-run determinism, heterogeneous worlds and the fastest solver sit on different solver paths, so each is available and their conjunction is not shown. Arm W is therefore built as one pipeline, on one fixed-topology brief and one mixed-topology contact-rich brief, before F exists, and the property that binds when all are demanded at once is recorded. One expensive incompatibility can sink the layer; several rarely exercised gaps may leave it competitive. Feature counts decide neither.

Phase ablations S. The phase ledger separates three accounts of where a gain comes from. On two briefs (one narrow wheeled, one contact-rich legged), on arm B's engine, three single-change replays run against the traced baseline.

  1. Information only. Batch proposals and a sound cheap diagnostic filter; the same controller acquisition and the same solver.
  2. Controller acquisition only. The same candidates, with per-design training replaced by a qualified cheaper controller: a prior with fine-tuning, inheritance, or a delivered sampling controller (§22).
  3. Stepping only. The same candidates and controllers on a faster solver at matched fidelity.

Each returns its own ratio and the share of the baseline's critical path that it touched. The ablations must respect two bounds computed from the baseline trace. If training that an intervention does not remove is a share s of elapsed time, that intervention's ratio is at most 1/s (1.43 at s = 0.7). If an intervention removes that share entirely its ratio is at most 1/(1 − s) (3.33). Illustration: 50 trained designs at 35 minutes each with 1.5 minutes of agent time per design make training 1,750 of 1,825 minutes, 95.9%, and cap information alone at 1.04. With controller acquisition at 45 s per design, agent time is 75 of 112.5 minutes, two thirds, and information becomes the larger lever. Which regime a stratum is in is a measurement, not a premise.

Owner and verbs. The protocol is this chapter's; the phase ledger is frob-chronicle's. report.campaign --arms B,W,F prints T, C, r_T and r_C with intervals, and report.campaign --ablate {information, controller, stepping} prints an ablation with its bound.

Gate Claim Threshold (TARGET) First milestone
GATE-TENFOLD-13 Design comparison; Performance r_T and r_C are published with 95% intervals for every stratum that has a ratio, whatever their values. The three ablations are run on both briefs, and a ratio above its 1/s or 1/(1 − s) bound voids the stratum as an accounting error. The one-receipt pipeline for arm W is built and its binding property recorded before any engine-internal work package beyond M2 is funded M3; M7 for the sealed suite