Personal project by Ulf Fäldt Reifenberg · unofficial, not affiliated with any company

plan-reconcile-bench

A detailed floor plan is checked against the preliminary plan of the same property. Which parts of that check should a language model do, and which should ordinary code do? Measured on 48 synthetic plan pairs, counting missed discrepancies and false alarms.

Results

The task

Each pair is a preliminary plan P and a detailed plan D of the same synthetic property (rectangular rooms, doors, windows, coordinates in cm). D is redrawn: new room ids in a new order, corners moved by up to 3 cm, polygons that start at another corner, and many rooms named differently (Bath, WC/Bath, Washroom; WIC, Walk-in closet). None of that is a discrepancy. On top, 0–3 material discrepancies are seeded per pair: a wall moved 30–80 cm, a room not drawn, a door not drawn, one window more or fewer, or a room that became a different kind of room (Bedroom → Office).

Approaches compared

ArmWhat decides
code, strictRooms matched by overlap; a wall that moved more than 10 cm, a missing room, a missing door or a different window count is a finding. Names are compared with a small synonym table written up front; an unknown name counts as a change (flag for review).
code, lenientSame, but names the table does not know are not compared at all.
model, raw plansThe model reads both plans as JSON and lists the discrepancies.
model, computed tableThe model reads the code's comparison (matched rooms, wall shifts, areas, window counts, doors only in one plan) instead of the raw plans, and lists the discrepancies.
hybridCode decides geometry, doors and windows; the model only answers “same kind of room?” for the name pairs the table cannot resolve.

Models: Llama 3.1 8B and Llama 3.3 70B on Cloudflare Workers AI, temperature 0, one call per pair (hybrid: at most one short call). Findings are graded by (type, room in P); a door finding for a room that is itself missing is ignored.

Results

ArmFound (recall)False alarms / pairClean pairs passedPairs fully rightModel calls · input tokens

Brackets: 95% bootstrap intervals, resampling the 48 plan pairs.

Found, by discrepancy type

Paired differences

ComparisonΔ false alarms / pairΔ recall

What this shows, and what it does not

How it is built

gen_plans.py generates the pairs and the answer key · reconcile.py is the code arm (overlap matching, wall shifts, door and window diff, synonym table) and also produces the table shown to the “computed table” arm · run_eval.py runs all arms through a small gateway with a content-hash cache and a hard usage cap · grade.py scores findings · analyze.py computes recall, false alarms and paired bootstrap intervals. Built with coding agents in an afternoon.

Personal project by Ulf Fäldt Reifenberg, built for a job application with coding agents. Unofficial and not affiliated with or endorsed by any company; it uses only generated data and makes no claims about any company's systems. Results are precomputed; this page makes no model calls. Contact: LinkedIn · other projects: proof-oracle-bench, euro-asr-bench