FrankenRobotics

Definition of done and the release rubric

2,338 words4 subsections3 tablesPlan version 1.1

On this page4
47.1 Definition of done47.2 Rules for any release47.3 The rubric by milestone47.4 The order in which evidence is procured, and complete dispositions

This chapter says when the program has delivered, and what must be true before anything is released on the way. Labels: every clause is a DESIGN statement of acceptance; the gates named beside a clause carry its thresholds (TARGETs, §40); no clause requires a particular measured number except where its gate does.

47.1 Definition of done

The program has delivered when each clause is true and its gate has a recorded value. A clause whose mechanism fell back at a kill gate is true when the fallback is in force, recorded as an ADR, and described by its own name.

  • The spine is real. Every construction C-01 to C-40 has its type, its verb, its failing-when-wrong test and its measured payoff, or has been moved to Appendix C with the reason (D-2).
  • The referee is real. Reference robots simulate on the referee tier under the transactional, event-located step, with known answers and parity recorded, and with coverage reported per robot class and an adjudicator named wherever coverage is under 80% (GATE-KIN-01, -02, -04; GATE-FID-03).
  • The batch is real. One kernel source runs on the CPU, CUDA and Metal. The divergence ledger explains every difference. A world's trajectory does not depend on the batch it ran in (GATE-SPARK-15). Throughput and training time are measured against the incumbent on the same cards and disclosed, whatever they are (GATE-SPARK-03, -04, -21; GATE-CORTEX-05).
  • The families are real. Many different robots run in one batch, and each world reproduces the digest of its single-design run (GATE-PHYLUM-02). The head-to-head is published with all four arms and the plan has acted on its outcome (GATE-PHYLUM-03). A family policy decides designs without per-design training wherever its rank agreement was measured, or KILL-FAMILY is on record and specialists with inheritance stand in its place (GATE-CORTEX-11, -13).
  • The ladder is real. A planted exploit is caught by promotion, and no report cites a provisional result as established (GATE-FID-02). Rank agreement between rungs is on record per morphology class and task family, a rung without a record is refused, and a screen has been audited by sampling what it discarded and by planting a known winner (GATE-CORTEX-12, GATE-ATEL-07, -09).
  • The places are real. Seven interior archetypes and the terrain families generate valid, reproducible scenes with feasibility tags (GATE-HAB-02). A clearance complex returns a route witness or an impossibility cut (GATE-HAB-10). One real place has been ingested privately, or the absence of a capture is recorded as no-claim (GATE-HAB-03).
  • The adversary is real. Search inside legality bounds finds planted traps faster than randomization, every reported failure carries a solvability status, a shrunk world replays with identical digests, and a stated failure-rate bound covers (GATE-NEMESIS-02, -03, -04).
  • The autopsy is real. A planted cause ranks first, a planted chaotic failure is given no cause, and a design-parameter intervention is re-run from the episode start (GATE-AUTOPSY-03, -05, -06). The effect on an agent's repair turns is measured and reported, whichever way it falls (GATE-AUTOPSY-08).
  • The mission engine is real. A mission query returns inner and outer design sets with witnesses, exclusions and an unresolved remainder. Every Certified cell is accepted by frob-verify built from an empty environment on a machine with no GPU, and every registered mutation is rejected. An amendment withdraws exactly what it must (GATE-AUGUR-02 to -06, -12; GATE-LEMMA-03, -04).
  • The studio is real. A fresh agent with only robot-docs designs, checks, simulates, explains, edits, compares and qualifies a robot inside its budget, through a CLI and an MCP surface generated from one registry (GATE-TETHER-01, -02, -13). Qualification holds its error rate under adaptive retries (GATE-QUAL-01).
  • The record is real. Any world replays alone from its address. The energy ledger closes to its stated residual. Every claim in the gate registry resolves to evidence whose hashes recompute, and retention removes no pinned object (GATE-CHRON-01, -03, -10, -12).
  • The ground is real. Every report prints its reality line. Predictions are frozen before they are compared. A band is issued only inside a calibrated family and refused everywhere else. Where no data was admitted the line reads no-claim, and that is recorded as the state and not as a pass (GATE-GROUND-01, -05, -06; -10 as an import lane).
  • The foundry is real. A qualified design yields a .fbuild packet whose hashes match, whose geometry round trips, whose tolerance stack closes, and whose four flags are present and false; or the reduced packet of KILL-PACKET is in force (GATE-FOUNDRY-08, -09, -10).
  • The ten-fold result is published, whatever it is. The report of §39 exists: per stratum, the ratio of restricted mean times with its interval, cold and warm separately, the crossover reuse count, a phase ledger for each arm, the certificate audit and the tie statement. Its wording follows BET-13: the word ten-fold appears only where the decision rule confirms it, a ratio between 3 and 10 is described by its value, and under 3 the product is described by what it lets a reader audit.
  • The honesty is real. Every result carries its claim product: the kind of claim, the evidence class set by the weakest input, tier, determinism class, model identity, seed, and the no-authorization flags, all false. Every gate has a caught negative control. reality: no-claim is printed wherever it is true. A mechanism that fell back is described by the fallback's name. The word "verified" describes no result (GATE-SAFE-01, -08; GATE-LEMMA-06; GATE-GAUNT-01).
  • The foundations are respected. No sibling was forked without a recorded kill decision, and none was patched locally: every change to a sibling was made in its own repository, under its own rules, as an upstream work item with a bead here and a mirrored bead in its home tracker. The whole build graph is on one toolchain T. Every admission record cites a commit whose lockfile contains the foundation (§33).

Three things are not required for done. No M item is required: the plan works without every one of them. No particular ratio is required: a published ratio of 2 with an honest ledger satisfies the clause, and an unpublished ratio of 12 does not. No calibration against hardware is required: a recorded no-claim satisfies the ground clause.

47.2 Rules for any release

A release is anything a user outside the implementing swarm is told they can use. Nine rules apply to every one.

  1. Implemented means §0.2. Each capability in the release meets all eight conditions of §0.2, or is listed as partial with the missing conditions named.
  2. No demo, no release. Each release path has a demos/<demo_id>/demo.yaml whose claims exist in registries/gates.toml and whose commands finish in under 60 seconds on the CPU tier (TARGET, §34.12). A demo shows that a path runs; the numbers come from the gate.
  3. The status table is generated. The README's honest-status table is produced from the gate registry. A failing gate appears as failing, and a fallback appears under the fallback's name.
  4. Each cell names its capability and its limit. No capability is described more broadly than the fixture that qualified it (§25.23). "Exact static family with independent active vertical reactions" is a release cell; "mission synthesis" is not.
  5. A losing experiment reproduces as readily as a winning one. The release ships the commands, addresses and seeds for at least one result that went against the plan.
  6. Performance comes with its disclosure block, and any ratio against an incumbent with the protocol of §39.
  7. Every exported artifact carries the flags, all false, and nothing in the release documentation contradicts them.
  8. The exit review was done by a fresh-context session (§43.2), and its record is part of the release.
  9. Reality is stated. Every report in the release prints reality: no-claim or names its calibrated cell.

47.3 The rubric by milestone

M Release A user can Must be true before it is released Must not be said
M0 None. An admission record, internal Read which foundations were admitted and what the rival's loop costs on trj Every admission has its record; the incumbent ledger has its disclosure blocks Anything about FrankenRobotics' own speed or accuracy
M1 Studio preview, CPU Write and import designs, run design check, simulate on the referee, read telemetry and the energy ledger, and ask the first mission queries with certificates checked by frob-verify NS-01, NS-07 and NS-22 demonstrated by a fresh agent; GATE-AUGUR-05 and GATE-LEMMA-03; the schedule gate (BET-12) recorded Any GPU or throughput statement; "qualified"; realism of any scene
M2 Gym release Train and evaluate at scale on CUDA, Metal or batch-cpu, run many designs in one batch, replay any world alone, fork a run Parity figures published with disclosure blocks, inside or outside tolerance; the M2 kill decisions recorded; a determinism class on every result; device results labelled exploratory unless OPEN-SPARK-1 is ratified An advantage over incumbents; bit equality beyond the class each device attained
M3 Qualification release Evaluate in seven interior archetypes, compare under a declared study, qualify a result in the blinded lane, find and shrink failures, attribute a failure GATE-QUAL-01 and the firewall gates; GATE-NEMESIS-03; GATE-AUTOPSY-03; the partial ten-fold gate recorded with its phase ledgers The word ten-fold, unless the rule of §39 permits it; realism while norms are assumed
M4 Design-science release Run co-design campaigns over families, use family policies and inheritance where admitted, ask mission queries at scale, search retrofits, state an energy-budget claim Rank agreement on record for every class a rung may eliminate in; the family-policy bets decided; every rung's label printed on its results A safety claim above Tested; a ranking from a rung without a record
M5 Evidence release Obtain kernel certificates, issue a reality band inside a calibrated cell, ingest a real place privately, emit a build packet The Reality lane has a record or a recorded no-claim; every flag false; material and card provenance printed Sim-to-real certification; fabrication release; "physically validated" beyond a named calibrated cell
M6 Frontier profiles, one per epic, behind feature flags Enable an epic's physics and run its known answers The epic's pilot and kill decision are on record; its results carry its tier That any frontier profile is part of the ten-fold thesis
M7 The tournament report Read the measured ratio per stratum and re-run the published artifacts Publication whatever the ratio; pins recorded; every certificate replayed; the adjudication status stated, self-adjudicated if no neutral party served "Independently accepted" without a neutral party; ten-fold outside the confirmed strata
M8 Frontier delivery, one per epic, M Use an epic that passed its gate Its gate recorded; its stop rule not triggered That the plan depended on it

A release at milestone N includes every earlier release's rules. A capability withdrawn by a later kill gate is removed from the next release's table with the ADR that withdrew it, and stays visible there as withdrawn.

47.4 The order in which evidence is procured, and complete dispositions

What it is. S Two rules for §47.2, and the check that enforces the first. They add to the rubric of §47.3 and change none of its rows.

Evidence is procured in dependency order. DESIGN: every release cell names the kind of evidence it rests on, and a cell is never scheduled ahead of the kind it needs. There are four kinds, in order.

Kind What it is Available
Authorable at once Analytic fixtures, exact checkers, certificates, CPU runs on the referee and on batch-cpu From M0
Pinned external comparison An incumbent re-measured on the same hardware under the disclosure block of §36.7 From the M0 phase ledger
Operator or dataset Bench logs, scans of real places, proficiency-matched operator trials When the import lane has an admitted dataset (§30)
Native device A completed run on the device family that the cell names, with a placement receipt that shows no fallback (§21.12) When that device ran it

A successful CPU run on the M5 Ultra, or in a shell session on trj, establishes the CPU path and nothing about a device path. A cell that names CUDA or Metal therefore needs a receipt whose actual backend equals the requested one for every stage it claims. Counterexample: a step plan in which one stage fell back to the host produces correct results and a receipt; a release cell that cites it for device throughput describes a path that did not run.

Tied, infeasible, incomparable and inconclusive are complete dispositions. A study that ends in one of them has delivered its result. The release prints it under that name, with the evidence that produced it, and no release note rewords it as a win, a loss or work in progress. Reason: a rubric that counts only decisive outcomes pays for overstated ones.

xtask check-contracts reads the release manifest and, for each cell, verifies the kind of evidence it declares, the class of each cited receipt and, for a device cell, the equality of requested and actual backend on every stage (refusal: release cell lacks evidence of the kind it declares). Owner: xtask, with the manifest schema in frob-core.

GATE-WS-08 Known answer. A release manifest with four planted cells is rejected on each: a device cell backed only by a CPU receipt; a device cell whose receipt records a fallback on one stage; a comparison cell whose incumbent figure is published and not re-measured; a cell whose study ended inconclusive and is listed as a capability. M1, and at every release.