How well can AI assemble a machine? The first benchmark results.

We gave three AI workflows the same authored STEP parts and goal renders, then applied the same MARB v0.9 scorer. Each recorded run placed the supplied parts without human placement assistance; none met the benchmark's declared geometric threshold. These are publisher-run, single-workflow measurements—not independent testing or physical buildability findings.

Archive note. This page preserves the founding v0.9 study. The repository's current scoring specification is v0.12; historical run labels and values are not silently rewritten.

The same frame joint in the reference and three AI runs at identical magnification; the reference and both Claude runs seat the V-wheel on its rail at 0 mm, while OpenAI Codex left it 138 mm off the rail — The same frame joint, four ways, at identical magnification. The reference and both Claude runs seat the V-wheel on its rail (0 mm). OpenAI Codex left it 138 mm off. Placing a part is easy; locating it is the hard part MARB measures.

Dated GAP-median versus elapsed-time chart for three single workflow runs; the cells used different tool paths and do not isolate a causal speed or quality effect — Recorded GAP median and elapsed time for three single workflow runs. Lower GAP is closer to the answer key's intended interface gaps; the tool paths and interaction conditions differ.

1. The task, the goal, not the method

The target is the gantry frame of M3-CRETE, an open-source large-format 3D printer / CNC frame: a ~2-metre machine of roughly 100 parts: aluminium extrusions, gantry plates, V-wheels, belts, pulleys, and seven NEMA-23 motors.

Each AI received only two things: the parts kit (the authored CAD files) and four reference renders of the finished machine (a 3/4 overview plus front, top, and side views), plus a short list of design constraints. Critically, we did not give the build sequence. The original human-guided build specified an inside-out order, X axis, then Y, then Z-posts, and detailed steps; here we withheld all of that. The AI had to figure out how to reach the pictured result on its own.

One of four reference renders (the 3/4 overview) of the finished M3-CRETE frame given to each AI — One of four goal renders provided (the 3/4 overview); the kit also includes front, top, and side views.

That design choice tests the submitted artifact under a task without a supplied build sequence. It does not directly measure a model's internal understanding. The recorded protocol limited each driver to the brief and kit rather than the reference solution; this is a publisher-controlled condition, not a security guarantee or third-party audit.

2. The grader across the included tool paths

Each included artifact was graded by CADCLAW, our open-source engine, through the exported STEP rather than a vendor feature tree: it loads the file, labels solids by shape signature, and runs three configured artifact-level gates:

Inventory: are all the right parts present, in the right count?
Interference: do any two solids overlap (a collision)?
Floating: is every part connected, or is something loose?

Grading the exported file applies one artifact-level procedure across tool paths, while export behavior, interfaces, prompts, hints, and tool maturity remain possible confounders. MARB v0.9 reports GAP, ORIENT, and POS against the answer key. The maintainer-authored L0–L7 ladder and TRL/MRL/IRL crosswalk are research discussion aids, not official readiness assessments or certifications.

The MARB pipeline from inputs to AI driver to exported STEP to grader to score — One recorded task and scorer across three workflows. Model, authoring interface, attempts, and other workflow conditions differ and must remain visible in interpretation.

3. The three drivers

Each ran in a fresh, prompt-only session (no project memory, no answer key):

Claude-Fusion: Claude Opus 4.7 driving a live Autodesk Fusion via its MCP connection.
Claude-CadQuery: Claude Opus 4.7 writing a CadQuery (Python) build script.
Codex-CadQuery: OpenAI Codex (GPT-5), the coding agent (not the chat app), also writing CadQuery.

4. Results

#	Driver	GAP median ↓	ORIENT aligned ↑	POS rel median ↓	Time	Attempts	Cost (est.)
n/a	CADCLAW reference (answer key)	0.0 mm	100%	0.0 mm	resolver	n/a	n/a
1	Claude Opus 4.7 · CadQuery	0.0 mm	51%	49.9 mm	48.8 min	9	~$68
2	Claude Opus 4.7 · Fusion	2.0 mm	47%	47.7 mm	33.7 min	12	~$174
3	GPT-5 Codex · CadQuery	7.8 mm	69%	47.2 mm	13.0 min	1	not reported

In this dated cohort, Claude‑CadQuery used nine recorded attempts and produced the lowest GAP median, 0.0 mm, in 48.8 minutes. The Codex-CadQuery run used one recorded attempt, completed in 13.0 minutes, and recorded 69% ORIENT alignment with a 7.8 mm GAP median. Claude-Fusion recorded the remaining values shown in the table. With one run per workflow and different tool paths, the results do not establish that iteration, model, or interface caused the differences. Claude token use was recovered from transcripts under the stated accounting method; Codex usage was unavailable, so its cost is not compared.

GAP, ORIENT, and POS are answer-key metrics under MARB v0.9. The thresholds are benchmark definitions, not machining tolerances, physical accuracy measurements, safety margins, or proof that an artifact can be fabricated or assembled.

5. What went wrong, the same failure class

The grader skips accessories (motors, belts, wheels), so every flagged clip is structural metal overlapping structural metal. All three shared the same mistakes: beams placed coincident at the splice joints and post/frame junctions instead of end-to-end, and the centred insert rails overlapping the beams they splice. They got the inventory and the topology right, but not the precise, non-overlapping placement.

There was also an error the original configured gate set did not catch: in every run, the Z-posts were turned the wrong way versus the reference. A part could be present, connected, and non-overlapping while still being mis-oriented. Current MARB scoring reports ORIENT separately; this historical finding remains part of the founding-study record.

6. Watch them build it

Both CadQuery drivers produced their own review renders as they worked. The Fusion driver inspected via live screenshots that weren't saved, so we recovered its build order afterward by driving the live model through its MCP, revealing the placed parts in sequence under a fixed camera (the design carries no parametric timeline).

Claude-Fusion build progression from 10 to 100 parts — Claude-Fusion, 10 → 100 parts, recovered from the live model via the Fusion MCP.

Grid of in-process review renders generated by the Claude-CadQuery driver — Claude-CadQuery's own in-process checks. These orthographic + isometric renders are the human-reviewable output that lets a watcher stop a bad run early to save tokens.

7. Limitations & honesty

A tooling-maturity bias to disclose. CADCLAW and its placement resolver were first built around CadQuery, before the Fusion connection (MCP) existed. The CadQuery runs may therefore carry a home-field edge. We flag it so the comparison stays honest; later scorer versions report orientation evidence, while broader reference-task coverage remains future work.
The reference "100" is our own method. The clean baseline is the spec-driven build (our home workflow) shown as the target, not a neutral third party. We publish the grader and the benchmark scaffold openly so anyone can reproduce, or challenge, these scores.
Verification is not physical certification. MARB grades declared properties of submitted digital artifacts, not physical buildability, manufacturing quality, safety, durability, or real-world performance.
The scorer is early and version-pinned; effort metrics (time, tokens, attempts) are reported separately and never folded into the artifact score.
One task, three models: a starting line, not a verdict. More drivers and tasks are coming.
The prior-art review is a dated snapshot, not a priority or uniqueness claim. Related and newer assembly benchmarks may exist and should be evaluated directly.

8. Why it matters, and what's next

A versioned task, scorer, and run registry make selected artifact differences auditable within the published method. Broader improvement claims require comparable versions, repeated runs, and attention to tool and protocol confounders.

Planned work includes repeated seeds, additional tasks, and expanded declared gates. The ladder, scoring method, and readiness crosswalk are maintainer-authored drafts offered for review; they are not a ratified standard, accredited test, or official readiness framework.

Reproduce or review: the benchmark scaffold (prompt, scoring, run procedures) and the CADCLAW engine are open. Method overview: marb.cadclaw.io. Benchmark repo (kits, graders, grades): MARB. Engine: CADCLAW. Reviewers welcome, [email protected].