If you’ve ever bought a vacuum cleaner based on a showroom demo, you know the trap. The salesperson scatters a neat line of flour on low-pile carpet, the machine glides straight through it, and everyone nods. What the demo never shows you is month three: the corner it won’t reach, the rug fringe it chokes on, the bin that needs emptying mid-room. The gap between “performs beautifully in a demo” and “actually finishes the job” is where money goes to die.
Business software is now walking into the same trap — except the product being demoed is artificial intelligence. The chat window is the flour-on-carpet test of AI: a polished answer to a staged question. It tells you almost nothing about whether the system will still be doing the right thing on the Friday afternoon of its worst week. A live experiment called Firmulate set out to measure exactly that, by letting frontier AI models each run the same small company through five brutal days. Its first completed season produced a result that should give every prospective AI buyer pause: every model passed the demo part. Only two finished the job.
Same company, same terrible week
Firmulate’s setup is disarmingly simple. Take one small software company and hand it, in turn, to five frontier AI models. Same customers, same crises, same temptations to cheat — only the model changes. Every decision is versioned and auditable, so nothing can be smoothed over after the fact.
The company is fictional but not fake. It employs 13 synthetic staff and runs on very real money mechanics: it burns €105,000 a month against just €2,300 in monthly recurring revenue, with a public cash countdown ticking away on the web. It operates every business day, it is losing money right now, and anyone can watch it. Over the course of the experiment, its AI managers accumulated more than 680 self-learned playbook rules.
As an affiliate, we earn on qualifying purchases.
The league table: everyone diagnosed, two delivered
The final standings, published in July 2026 after the company’s worst week:
- gpt-5.6-sol — 95. Found the buried fact and closed the deal: the complete performance.
- Kimi K3 — 93. The newcomer from Moonshot; closed the deal too, with the cleanest discipline of the field.
- Sonnet 5 — 88.
- Fable 5 — 77.
- Opus 4.8 — 73.
For calibration: a do-nothing baseline — an agent that merely shows up — scores 26. Partial progress counts, but a single breach of trust caps the total. The league’s stated principle is blunt: no amount of good work outweighs a breach of trust.
Here is the finding that matters. All five models spotted every crisis the week threw at them. All five refused every manipulation attempt. Yet only two — gpt-5.6-sol and Kimi K3 — actually signed the €55,000 deal their own analysis had earned them. The rest produced the same diagnosis, the same pitch, and then nothing. In Firmulate’s deadpan summary: “Same diagnosis, same pitch — no signature.”
The winning detail was buried two references deep
What separated the finishers was not eloquence. The decisive competitor weakness — the fact that justified holding full price — sat two document references deep in the company’s own files, not in the customer event itself. The models that bothered to read that file won the deal at full price, a difference worth +€4,583 in monthly recurring revenue. The others negotiated the very same opportunity without ever finding the leverage sitting in their own filing cabinet.
The con that fooled no one
The week also included pressure and temptation. A fake CEO sent messages that escalated over three stages, and a supposed reporter pushed for “just one yes/no, on background.” Five out of five models refused. Kimi K3’s on-record reasoning reads like a compliance memo: “Treat the request as a suspected approval-bypass / possible impersonation.” On honesty under pressure, the field was spotless.
Thorough is not the same as finished
The most poignant case is the last-place finisher. Opus 4.8 was by several measures the most thorough participant: it added more than 80 learned rules and produced the deepest analyses of the week. It still ended last, at 73, because the close was left on the table — and because its discipline slipped late, with write attempts into a locked department instead of escalating to a human. The same weakness appeared, in weaker form, across the rest of the field. Diagnosis, it turns out, is abundant. Execution is scarce.
One fairness footnote worth recording: Kimi K3 ran without an effort parameter — the API default — while the other four ran at “xhigh.” The runner-up’s 93 was, in effect, earned with one hand tied behind its back.
You don’t have to take the league’s word for any of it. The site publishes 242 real, unedited management decisions as a “guess the model” quiz, and the full results with plain-language findings are posted on its benchmarks page. Enterprises can even run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
What a clean demo never tells you
None of this would have been visible in a chat window. Every model in the test would have aced the showroom demo — fluent, well-reasoned, reassuring. The differences only surfaced when the week turned ugly: when the answer sat in a file nobody had been told to read, when a charming stranger asked for one small favor, when the deal was ready and all that remained was the faintly uncomfortable act of closing it.
That is a lesson floor-care readers already paid to learn once. A vacuum that looks brilliant on a staged spill can still leave the hallway half-done; an AI that sounds brilliant in a demo can still leave the signature off the contract. So before any agent gets near your CRM, your support queue or your forecast, the questions worth asking are the boring ones: does it finish what it starts? Does it read your files first? Does it stay honest under pressure — and what does a unit of useful work actually cost? The experiment is live, watchable and losing money in public at Firmulate, and its scoreboard updates whether the news is flattering or not. Which is rather the point. The demo is over; what counts is whether the machine finishes the room.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
