MARB studies

Each entry records its task, kit, driver, prompt or note, tool path, run count, scorer, and date. Shared conditions improve comparison, while seed count, model size, modality, tool interface, timeout behavior, and protocol changes remain visible confounders.

A note on terms. Artifact-completion metrics record whether a run produced a loadable STEP and how many part instances it placed. “Loadable” does not mean physically buildable. The separate grade reports position, orientation, and answer-key gap metrics under the named scorer version.

Study: the Claude Fable 5 effort sweep

Driver: Claude Fable 5, reasoning effort low / medium / high, plus an ultra run (multi-agent: build, 4-agent adversarial audit, targeted fixes). CAD tool: CadQuery. Kit and grader identical to the first results. Blind runs, one per setting. Positional grades (MARB v0.9).

Three single-agent cells varied a recorded reasoning-effort setting under the same task and kit. A fourth “ultra” cell also changed the harness by adding multiple audit agents, so it is not an effort-only comparison. The question is what these four recorded workflows produced—not what more spending generally buys.

Token figures were recovered post-run by summing recorded per-message usage blocks. In these n=1 cells, high effort recorded 3.9× the billed tokens of medium and higher GAP/lower ORIENT values. Host accounting and cache treatment limit cross-system cost comparisons.

Within these three single-agent runs, the metrics were not monotonic with the named effort setting: medium recorded the lowest GAP and highest ORIENT, high recorded the lowest relative POS, and low recorded the highest relative POS. One run per setting is insufficient for a causal or purchasing conclusion.

The ultra run used a materially different multi-agent harness that probed kit parts, ran declared audits, and applied targeted fixes. It recorded 3.0 mm GAP and 30.4 mm relative POS in 83 minutes with a recovered 23.0M-token bill. The audit reported specific shaft, belt, and pulley conditions for review. This single cell cannot isolate whether the harness, stochastic variation, or another condition produced the metric differences.

On the dated board, these cells occupy different positions by the primary GAP metric. Because each frontier cell is a single run and the ultra harness changes more than one factor, treat the differences as hypotheses for repeated testing. The recap retains the token-accounting limitations.

Study: the sighted local cells

Drivers: qwen3-vl:32b (vision, 8 turns n=5 + 12 turns n=2 preliminary) and nemotron3:33b Omni (vision, n=5), goal image inlined on turn 1, lean-v5 guidance. Same kit, grader, and harness as every other cell. Token capture native. Graded MARB v0.9.

These cells supplied a goal image in-loop to vision-capable models. The recorded sighted cohorts did not outperform the selected blind 80B text cohort on the reported metrics, but model size, modality, run budget, and other factors differ; this is not an image-on/image-off ablation.

• Observed sighted 32B cohort. All five runs produced loadable exports with 12–17 placed parts and higher reported errors than the selected blind 80B cohort. The study does not identify the cause.
• Preliminary 12-turn variant. Two completed seeds recorded lower GAP than the eight-turn cohort while still placing fewer than 20 parts; three other attempts were lost to a documented host timeout. No general turn-budget conclusion is supported.
• Observed Nemotron cohort. Five recorded sessions produced no loadable export. This is an artifact-completion result under the stated harness, not a claim about physical buildability or the model generally.
• Caveat, stated plainly: this compares a 32B vision model against an 80B text model — different sizes — because those are the strongest local options per modality. The cell answers a shop's question ("does my best local vision model beat my best local text model here?"), not a controlled ablation. A same-model image-on/off ablation needs a frontier vision model and is queued.

Study: the local open-weight anchor

Driver: qwen3-coder-next:q4_K_M (80B total, 3B active), text only, on a single local AI supercomputer. CAD tool: CadQuery 2.7.0. Kit: v1.1. Blind run, no internet, no memory of past work. 40 runs in six cohorts plus extension seeds. Buildability metrics plus positional grades (MARB v0.9; graded cohorts now n = 9 and n = 8 loadable of 10 attempts each).

This study records a self-hosted 80B coding model on one local system under the same mechanical task. “Self-hosted” does not mean free: hardware, electricity, setup, operator time, and opportunity cost remain outside the token table.

The study began with thirty runs across six five-run prompt cohorts and later added extension seeds, for forty recorded attempts in the stated aggregate. Notes clarified the tool or task without intentionally revealing the gated reference design; this is a protocol statement, not proof against all contamination.

Target is about 100 placed part instances. The kit holds authored STEP parts; the model places each one and exports a single STEP file.

What the recursion taught us

• Export-call association. The cohort given the correct export idiom recorded more loadable exports than the initial five-run cohort. Small cohort sizes and stochastic variation prevent a causal estimate.
• Additional guidance cohorts. Cohorts with build-volume and design-goal notes recorded lower initial loadable-export rates. The study does not establish why.
• Longer-budget cohort. The 14-turn cohort recorded a higher initial export rate than the preceding design-goal cohort but did not establish improved physical or engineering quality.
• Lean-note cohort. The lean note matched the initial five-of-five loadable-export count; extension seeds later reduced the combined rates.

The grades: placed, not jointed

Updated 2026-06-12 with five extension seeds per selected cohort: the initial five-of-five loadable-export counts became 9/10 (mechanics v2) and 8/10 (lean v5), and the reported spreads widened. The aggregates below use the loadable seeds.

The two cohorts were graded with the same GAP, POS, and ORIENT definitions used for the mechanical frontier track. Metric definitions are shared, but model, prompt, kit version, interface, seed count, and execution conditions must remain part of any comparison.

• Parts land 100 to 400 mm off on a 2000 mm machine. Joints that should sit flush are 300 to 400 mm apart at the median, and only 11 to 14% of interfaces fall within 1 mm. About 30% of asymmetric parts get the correct rotation.
• The shape is closer than the anchor. Relative position error is far smaller than absolute (104 mm vs roughly 500 mm for mechanics v2): the model captures local clustering better than it anchors the whole frame.
• Loadable export and positional grade are different outcomes. Lean v5 recorded a higher loadable-export rate while mechanics v2 recorded lower positional errors in this sample.
• The structural gates agree. Both graded builds fail inventory on the part mix (not the count) and show 20 to 28 pairs of clipping solids, while nothing floats free. Parts grouped together, not a connected frame; a connectivity gate that scores seated joints is the next grader level.

What this study does not yet answer

• Design judgement at scale. Asking a small model to weigh manufacturability, thermal load, vibration, fatigue, and kinematics is the real goal of the benchmark. It pays off only with a larger model, a longer budget, and a grader that can score those modes. That work belongs in the shared task definition, not in a local note.

Method, harness, grades, and the run catalogue are in the open-source repository. These mechanical-task cohorts use the stated MARB v0.9 metrics; the separate Pascal lane uses its own grader and is not pooled with them.