MARB-A: three AIs build the same house

Lane: MARB-A architecture · Task: PH-1 "Bungalow" · Kit pascal-v0.1 · Grader v0.1.1 (frozen) · 2026-07-09 · n=1 per cell

This pilot extends MARB beyond the mechanical task: a workflow builds a specified house in Pascal, a free, open-source 3D building editor, through the tested agent interface. The submitted scene export is scored against a gated reference layout under a separate MARB-A method. Sessions used documented access controls; “controlled” does not mean a security sandbox.

In these three single runs, two workflows wrote furniture rotations in the wrong unit, while one met the recorded PH-1 gates in 9.9 minutes. That is a result for this task, tool version, grader, and run—not a claim of perfect architecture, construction quality, or general model capability. The renders below are outputs of the tested Pascal workflow.

Two 3D renders of the same house by Pascal: top shows furniture rotated at odd angles (Claude Fable 5); bottom shows furniture correctly squared (GPT-5.5 Codex). — Same task, same tool, rendered by Pascal itself. Top: Claude Fable 5 — every position exact, every rotation written in the wrong units (Claude Opus 4.8 at max effort failed identically). Bottom: GPT-5.5 Codex — correct units, 100% orientation, 9.9 minutes.

The task

The blind kit supplies a dimensioned bungalow: six rooms, six doors, nine windows, and 22 catalog furniture items, including target coordinates. The test records whether the submitted scene preserves the declared contract. The protocol records no human placement assistance and no reference-layout or grader access; this is a publisher-reported condition, not an independent security attestation.

The board

Model · effort	ORIENT-A	Gates	Wall-clock	Tokens billed
GPT-5.5 · Codex · xhigh	100%	all PASS	9.9 min	2.11 M (22k out)
Claude Opus 4.8 · max	31.6%	collisions FAIL	19.9 min	13.50 M (313k out)
Claude Fable 5 · medium	31.6%	collisions FAIL	117.8 min¹	4.83 M (129k out)

All three placed every wall, door, window, and furniture item at 0.0 mm median error, with an exact bill of materials. ¹ Includes human approval latency (this run predates auto-approval); the other two ran fully auto-approved. Token meters count differently across vendors — the bills are listed, not compared.

Three-panel bar chart comparing orientation accuracy, wall-clock minutes, and token bill for the three models. — Positions saturated at 0.0 mm for all three — orientation and cost are what separate the cohort.

The trap: one undocumented number

In the tool version tested (@pascal-app/mcp 0.3.1), the furniture rotation input takes a bare number — no units stated anywhere a model can see — and stores radians (the three.js convention). Both Claude models wrote degrees. The software accepted 90, 180, 270 without a word. Rendered by Pascal itself: furniture 10–37° off square, a kitchen unit through a wall, the collision gate failing. The orientation score reads 31.6%, but earned credit is roughly zero — the "correct" items just happened to face the default direction.

The recorded Opus 4.8 workflow placed two disposable test sofas, read the values back, deleted them, and still concluded degrees because the field echoed the submitted values. In this interface, that readback alone did not distinguish degrees from radians.

The Codex run wrote radians and recorded 100% ORIENT-A under this grader. Review included the exported values, construction route, and an available-transcript access scan. That scan cannot prove training-data provenance or absence of every undisclosed influence. The observed unit ambiguity affected two runs; generalization requires more tasks and seeds.

The two Claude runs recorded the same rotation-unit failure despite different effort settings. With one run per cell and other possible differences, this is a descriptive observation rather than a causal or purchasing conclusion.

The deeper gap: no eyes

In the tested headless Pascal path, GLB export was stubbed and no sanctioned render tool was available to the workflows. A render might have made the rotation error visible, but this pilot did not test that counterfactual. A later kit with a sanctioned self-render tool must be treated as a new cohort.

Side-by-side floor plans: the goal layout and the as-built layout with furniture footprints visibly rotated off square. — Goal vs built (Claude Fable 5), plotted from the graded scene: walls and openings aligned to the millimeter, furniture footprints skewed by the units error.

Animated build timelapse: walls appear, then doors and windows, then furniture, replayed from the session transcript. — One build replayed step-by-step from the session transcript: walls, then openings, then furniture.

Check our work: every mistake on the record

Our wall metric mis-scored a perfectly valid wall layout on run one (the model built the same centerlines as different segments). Fixed with a segmentation-invariant coverage metric, both scores published, grader frozen at v0.1.1 before runs two and three.
One model's run log claimed it was a different model entirely; another logged a nonsense elapsed time. Names, clocks, and token bills come from session metadata, never from the model.
Our own cleanup accidentally deleted the first run's kit files mid-run (the model had already loaded the brief and finished normally); that run's wall-clock also includes human approval waits. Both caveats live on its registry entry.

Provenance and shelf life

Task, grader, figures, exported scenes, and the run registry—including the caveats above—are public in the MARB repository (see results/pascal_runs.json and tasks/pascal_house/). The answer key is gated separately. Pascal changes can shorten this finding's shelf life, so results remain tied to the stated tool version and date. No priority or uniqueness claim is made.

MARB-A results are a separate cohort family and are never pooled with the mechanical board. Every cell here is a single run; repeat seeds are the next step, same as they were on the mechanical side.