MARB-A: three AIs build the same house
MARB was built to grow beyond one machine, and this is the first expansion: instead of a 100-part printer frame graded as STEP geometry, an AI builds a fully specified house in Pascal, a free, open-source 3D building editor, driving the tool's own agent interface in a sealed session. It is also the first MARB lane graded entirely outside the CADCLAW engine — the submitted artifact is the tool's scene export, scored against a reference layout.
Two of the three models twisted every piece of furniture out of square. One was perfect — in ten minutes. The renders below are Pascal's own output, not mock-ups.
The task
The blind kit hands the model a fully dimensioned bungalow: six rooms, six doors, nine windows, 22 pieces of furniture from Pascal's built-in catalog — exact coordinates included, on purpose. Placing things is copying numbers; the test is whether those numbers survive the trip through a real tool's contract. Each model ran sealed — no human help, no access to the reference layout or the grader.
The board
| Model · effort | ORIENT-A | Gates | Wall-clock | Tokens billed |
|---|---|---|---|---|
| GPT-5.5 · Codex · xhigh | 100% | all PASS | 9.9 min | 2.11 M (22k out) |
| Claude Opus 4.8 · max | 31.6% | collisions FAIL | 19.9 min | 13.50 M (313k out) |
| Claude Fable 5 · medium | 31.6% | collisions FAIL | 117.8 min¹ | 4.83 M (129k out) |
The trap: one undocumented number
In the tool version tested (@pascal-app/mcp 0.3.1), the furniture rotation input takes a bare number — no units stated anywhere a model can see — and stores radians (the three.js convention). Both Claude models wrote degrees. The software accepted 90, 180, 270 without a word. Rendered by Pascal itself: furniture 10–37° off square, a kitchen unit through a wall, the collision gate failing. The orientation score reads 31.6%, but earned credit is roughly zero — the "correct" items just happened to face the default direction.
Opus 4.8 at max effort ran the most careful check of the three: it placed two disposable test sofas, read the values back, deleted them — and still concluded degrees, because the field echoes whatever you give it. Reading back your own number is the one experiment that cannot reveal units.
Codex wrote radians and scored 100% — verified the hard way: correct values in the exported file, its own construction route, and a transcript scan showing no access to anything it shouldn't touch. One honest unknown: Pascal's source is public, so we cannot rule out that Codex knew this specific tool from training rather than knowing 3D conventions in general. The trap is real either way — two frontier models fell in.
The cleanest cost lesson is within one vendor: Claude at medium effort and Claude at max effort produced the identical failure. Max effort bought a better process — not a better answer.
The deeper gap: no eyes
On the mechanical board, drivers can render and inspect their work in progress. Headless Pascal has no picture: GLB export is stubbed and there is no render tool, so a model that places furniture cannot see it. One look at the first placed sofa would have caught the units error instantly — it's why you walk the site. Kit v0.2 adds a sanctioned self-render tool (top view plus elevations from the scene export); shipping it starts a new cohort. Nothing forbade the models from building their own renderer this round; none tried.
Check our work: every mistake on the record
- Our wall metric mis-scored a perfectly valid wall layout on run one (the model built the same centerlines as different segments). Fixed with a segmentation-invariant coverage metric, both scores published, grader frozen at v0.1.1 before runs two and three.
- One model's run log claimed it was a different model entirely; another logged a nonsense elapsed time. Names, clocks, and token bills come from session metadata, never from the model.
- Our own cleanup accidentally deleted the first run's kit files mid-run (the model had already loaded the brief and finished normally); that run's wall-clock also includes human approval waits. Both caveats live on its registry entry.
Provenance and shelf life
Task, grader, figures, exported scenes, and the full run registry — including the caveats above — are public in the MARB repository (see results/pascal_runs.json and tasks/pascal_house/). The answer key is gated separately to keep future runs clean. Fair warning: Pascal's development branch already adds unit documentation to the rotation field — the right fix, and it means this particular trap will not survive their next release. That is the point of measuring now. As far as we can find, this is the first scored benchmark for AI assembly in Pascal; if you know of one, tell us and we will link it.
MARB-A results are a separate cohort family and are never pooled with the mechanical board. Every cell here is a single run; repeat seeds are the next step, same as they were on the mechanical side.