MARB — Mechanical Assembly Readiness Benchmark

The gap

AI-CAD benchmarking includes geometry-generation, part-classification, joint-prediction, and other assembly-oriented methods. MARB focuses on one narrower question: how a submitted multi-part artifact compares with a declared answer key and gate configuration.

The method reports inventory, interference, floating-part, GAP, ORIENT, and POS evidence where the applicable scorer version supports them. It does not prove that a machine is functional, safe, manufacturable, physically buildable, or ready for use.

Capability × Readiness

The maintainers provide an indicative comparison between the research ladder and TRL, MRL, and IRL concepts. It is a discussion aid—not an equivalence, official mapping, agency assessment, or shortcut to a readiness review.

CADCLAW capability ladder ↔ TRL / MRL / IRL, indicative evidence bands, not equivalence.

Two different axes. Formal readiness processes evaluate a technology or system under their own authorities and evidence requirements. MARB evaluates a recorded workflow and exported artifact under a benchmark method. CADCLAW findings may be one input to an engineering review, but they are not readiness evidence by themselves.

How the benchmark runs

Within a declared cohort, MARB records the task, kit, prompt, tool path, model, assistance conditions, software versions, attempts, and grader. Shared inputs and a shared scorer improve comparability; differences in tools, hints, interfaces, seeds, and execution conditions remain potential confounders.

The MARB pipeline, inputs → AI driver → exported STEP → CADCLAW gates → MARB score + readiness → human review, with a read-fix-rerun loop.

The metrics

Each compatible record reports two categories separately: declared artifact metrics under the named scorer version and recorded effort fields such as elapsed time, attempts, and available usage. Neither category proves physical quality or total cost.

The board, 2026-06-11

This is the site snapshot dated 2026-06-11: eleven summarized cells for the mechanical task, ranked by the published primary metric. The repository run registry may contain later or additional records, and the Hugging Face Space is a separate publication surface that must be checked independently for freshness.

Published 2026-06-11 snapshot. Ranks and descriptive labels apply only to this image, scoring version, included cells, and recorded conditions.

Dated token-versus-GAP view for cells with available or recovered usage. Token accounting differs by host and is incomplete; see the method and limitations in the recap.

Deep dives: the recap article (findings to date + the token ledger), the studies log (the Fable 5 effort sweep and the graded local open-weight anchor), and the first-results write-up (the original three-way head-to-head, preserved below as the founding study), and the new MARB-A architecture lane (three models build a house in Pascal; two fall into the same units trap).

First head-to-head (the founding study, 2026-05-26)

Three publisher-run workflows—Claude Opus 4.7 with Fusion and CadQuery, and OpenAI Codex / GPT-5 with CadQuery—used fresh prompt-only sessions and the same founding kit. Each placed the supplied parts without human placement assistance; none met the benchmark's declared geometric threshold. These are single workflow runs, not independent product tests.

One of four goal renders provided (the 3/4 overview); the kit also includes front, top, and side views.

Recorded GAP versus wall-clock time for the included frontier cells. The chart is descriptive; it does not establish that model, iteration count, or reasoning effort caused the observed differences.

MARB v0.9. GAP median = error vs the answer key's intended interface gap (≈0 mm bolted, ≈1 mm motion clearance). ORIENT aligned = % of asymmetric parts in the correct rotation. POS rel median = position error relative to each part's neighbours (frame-invariant). The configured bands (≤5 mm = located, ≤5° = aligned) are benchmark definitions, not machining tolerances or physical-buildability criteria. Cost estimates use the price basis recorded for the dated study; Codex CLI did not report comparable usage. Results 2026‑05‑26 · CadQuery 2.7.0, Autodesk Fusion 2702.1.58 · graded per the MARB v0.9 scoring spec.

Every flagged clip is structural, beams overlapping at splice joints and post/frame junctions, and the centered 2040 inserts overlapping their beams. Both also placed the Z-posts in the wrong rotation versus the reference, an orientation error the current gates don't yet catch, and the next gate we're adding.

Claude-Fusion, building the frame (10 → 100 parts). The model emits no parametric timeline, so we recovered the build order afterward by driving the live Fusion model through its MCP, revealing the placed parts in order under a fixed isometric camera.

Claude-CadQuery's own in-process renders, the orthographic + isometric checks it generated as it built. This is the human-reviewable output that lets a watcher catch a bad run early and stop it to save tokens.

Why it matters, in plain English

Repeatable geometric checks can help teams find selected conditions earlier and compare workflow changes under a controlled method. The economic value depends on the assembly, review process, defect class, compute cost, and what the configured checks can actually observe.

CADCLAW can report selected properties of a supplied STEP artifact against declared rules. It cannot prove an assembly is right, prevent an escaped defect, quantify savings, or replace native-CAD review, engineering analysis, inspection, or physical testing.

What makes it different

MARB is not "a CAD checker." Specifics separate it from CAD-vendor tooling and from prior benchmarks:

• Grades the submitted STEP, not a vendor's feature tree. A shared exported-artifact method reduces some vendor-specific differences while retaining export, tool-path, and metadata limitations.
• Scores one submitted multi-part artifact. The configured checks operate across the supplied assembly, while some public joint and mate datasets evaluate part pairs. Coverage still depends on the declared gates and task.
• Separates benchmark capability from readiness. An indicative crosswalk is published for discussion, but MARB scores do not enter or replace an official gate review.
• Uses a versioned reference assembly. The mechanical task is based on the M3-CRETE experimental reference design; the benchmark does not claim physical conformance or production validation.
• Publishes claim checks and limits. CADCLAW includes configurable text-audit tooling, which is a backstop rather than a guarantee of truthful or complete claims.

Papers and public benchmark data

The public result records link to the repository's scoring spec, graders, public kits, grades, and run registry. The registry lists available per-run provenance fields such as model, tool, timing, usage, and attempts; fields can be unavailable, and answer keys remain separately gated.

Request a sponsored benchmark run

Sunnyday can scope a new model/tool cell, repeat-seed cohort, or private evaluation as a manual service. A sponsored public run is identified in the run registry, follows the versioned method, and is published regardless of whether the result is favorable. Sponsorship does not buy a score, rank, conclusion, endorsement, certification, or claim about the sponsor's product.

Planning ranges are non-binding and exclude applicable taxes, third-party licenses, unusual compute, travel, and custom engineering. This site does not accept payment or create an order. Work begins only after entity, tax, rights, privacy, independence, cancellation/refund, payment, and deliverable terms are confirmed in a signed order or invoice. Sponsors should obtain their own legal, accounting, and tax advice.