Clairvyn just launched KOZO, our flagship model built to generate 2D residential floorplans. It can produce structural layouts too, and it works across multi storey setups. It’s still early and far from perfect, but when we ran it against general purpose LLMs on three benchmarks, the results genuinely surprised our tech team.

Here’s what we tested, why we tested it that way, and what we found.

1. Accuracy: measured with GSM8K

Floorplan generation isn’t just drawing boxes. Every wall, room, and structural element has to add up correctly: dimensions, proportions, load bearing calculations, spacing between elements. That’s a multi step numerical reasoning problem before it’s a design problem.

GSM8K is a benchmark built to test exactly that kind of reasoning: grade school level math word problems that require a model to work through several logical steps to land on the right answer. We used it because generating a CAD ready floorplan, output as both .dxf and .png files, depends on the same kind of step by step numerical accuracy. Get one step wrong and the whole layout is off.

On this test, KOZO hit 77.5% accuracy on generating correct, CAD ready outputs. General LLMs came in at 63.8%. The gap makes sense once you consider that KOZO is purpose built around spatial and dimensional logic, while general LLMs are working with reasoning patterns that weren’t designed for this specific task.

2. Speed: measured with LLMPerf

Design tools live or die on iteration speed. If a model takes too long to generate a layout, it breaks the back and forth that architects and designers actually work in.

LLMPerf is a benchmark built to measure real world model performance: things like tokens generated per second and how quickly a model starts responding. We used it to see how usable each model actually is in a live workflow, not just how accurate it is on paper.

KOZO generated at 119 tokens per second. General LLMs averaged 46 tokens per second. That’s a meaningful difference for anyone iterating on a design in real time, and it’s a big part of why KOZO feels fast in practice rather than just fast on a spec sheet.

3. Logical reasoning: measured with HumanEval

A CAD ready output isn’t free text. It’s structured, rule bound, and has to follow a precise format for it to actually open correctly in design software. That’s closer to generating functioning code than generating a paragraph.

HumanEval is a benchmark originally built to test whether a model can generate code that actually works, not just code that looks plausible. We used it to evaluate how well each model handles the structured, procedural logic behind translating a floorplan into a valid, usable file.

KOZO scored 76% here. General LLMs scored 52%. The difference comes down to KOZO being trained specifically to reason through structured design constraints, rather than general text patterns.

Where this leaves us

KOZO is early. It’s not perfect, and we’re not claiming it is. But across accuracy, speed, and logical reasoning, a purpose built model is outperforming general LLMs on the specific job of turning an idea into a usable floorplan. That’s a strong starting point, and we’re just getting going.