Personal project by Ulf Fäldt Reifenberg · unofficial, not affiliated with any company
plan-reconcile-bench
A detailed floor plan is checked against the preliminary plan of the same property. Which parts of that check should a language model do, and which should ordinary code do? Measured on 48 synthetic plan pairs, counting missed discrepancies and false alarms.
- Loading…
The task
Each pair is a preliminary plan P and a detailed plan D of the same synthetic property (rectangular rooms, doors, windows, coordinates in cm). D is redrawn: new room ids in a new order, corners moved by up to 3 cm, polygons that start at another corner, and many rooms named differently (Bath, WC/Bath, Washroom; WIC, Walk-in closet). None of that is a discrepancy. On top, 0–3 material discrepancies are seeded per pair: a wall moved 30–80 cm, a room not drawn, a door not drawn, one window more or fewer, or a room that became a different kind of room (Bedroom → Office).
Approaches compared
| Arm | What decides |
|---|---|
| code, strict | Rooms matched by overlap; a wall that moved more than 10 cm, a missing room, a missing door or a different window count is a finding. Names are compared with a small synonym table written up front; an unknown name counts as a change (flag for review). |
| code, lenient | Same, but names the table does not know are not compared at all. |
| model, raw plans | The model reads both plans as JSON and lists the discrepancies. |
| model, computed table | The model reads the code's comparison (matched rooms, wall shifts, areas, window counts, doors only in one plan) instead of the raw plans, and lists the discrepancies. |
| hybrid | Code decides geometry, doors and windows; the model only answers “same kind of room?” for the name pairs the table cannot resolve. |
Models: Llama 3.1 8B and Llama 3.3 70B on Cloudflare Workers AI, temperature 0, one call per pair (hybrid: at most one short call). Findings are graded by (type, room in P); a door finding for a room that is itself missing is ignored.
Results
| Arm | Found (recall) | False alarms / pair | Clean pairs passed | Pairs fully right | Model calls · input tokens |
|---|
Brackets: 95% bootstrap intervals, resampling the 48 plan pairs.
Found, by discrepancy type
Paired differences
| Comparison | Δ false alarms / pair | Δ recall |
|---|
What this shows, and what it does not
- Synthetic, structured, easy geometry. Rectangles with clean coordinates, defects of 30 cm and more against a 10 cm tolerance. Real plans come from 360° imagery and drafters' tools; real differences are smaller, and many are judgement calls. Here geometry is a solved problem for code by construction; the point is how much a model loses when it is asked to do it anyway.
- The synonym table is the weak spot on purpose. A table that covered every name in the generator would score perfectly, because it would be built from the answer key. In practice the vocabulary is open, which is why the naming question went to the model.
- Not shown: imagery, vision models, multi-step agents with tool calls, fine-tuning, or anything about a real company's data or process. One synthetic generator, one seed, two open models, one prompt per arm without tuning.
How it is built
gen_plans.py generates the pairs and the answer key · reconcile.py is the code arm (overlap matching, wall shifts, door and window diff, synonym table) and also produces the table shown to the “computed table” arm · run_eval.py runs all arms through a small gateway with a content-hash cache and a hard usage cap · grade.py scores findings · analyze.py computes recall, false alarms and paired bootstrap intervals. Built with coding agents in an afternoon.