Anyone who has spent time on vacuum reviews knows the difference between a machine that looks impressive on the showroom floor and one that actually cleans your carpets. Specs are easy. Follow-through is hard. The best machines — and the best technicians — share one habit: they read the manual before they start.
Get cleaning gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
It turns out AI agents have the same divide, and someone finally measured it. A project called Firmulate ran four frontier AI models through the identical worst week at a small software company — same customers, same crises, same temptations to cut corners. Every model could talk a good game. Only some of them did their homework. The difference was worth €55,000.
The Setup: Same Company, Same Crisis Week
Each model was handed the same simulated software company and told to run it. The scenarios were identical down to the customer emails. Every decision was versioned and auditable, so nothing came down to chance or vibes.
The headline result from the final July 2026 league table: gpt-5.6-sol finished first with a score of 95, Kimi K3 second at 93, Sonnet 5 third at 88, Fable 5 fourth at 77, and Opus 4.8 last at 73. A do-nothing baseline — an agent that simply sat on its hands — scored 26, because partial progress counts, though a single breach of trust caps the total. As the experiment puts it, “no amount of good work outweighs a breach of trust.”
As an affiliate, we earn on qualifying purchases.
The Buried Fact That Decided the Deal
Here’s where the story gets interesting, and where the vacuum analogy earns its keep.
During the week, a major customer opportunity appeared: a €55,000 deal. Every model spotted it. Every model diagnosed the customer’s problem correctly and delivered a strong pitch. And yet only two models actually closed it. The others left the deal on the table at full price — worth +€4,583 in monthly recurring revenue.
Why? The decisive competitive weakness — the fact that would justify the customer switching — wasn’t in the customer’s emails or the meeting notes. It sat two document references deep in the company’s own internal files. The models that followed the paper trail, reference by reference, found it and signed the deal. The models that didn’t, didn’t.
“Same diagnosis, same pitch — no signature,” as the experiment’s own summary puts it.
Think of it like a repair tech who skips the troubleshooting flowchart because the symptom looks obvious. The machine gets opened, the obvious part gets checked — and the real fault, documented two pages into the service manual, never gets read. The repair fails for a reason that was always knowable.
Everyone Passed the Honesty Test
To be clear, the failing models weren’t lazy or dishonest across the board. All five participants — including the fifth, Opus 4.8 — spotted every crisis and refused every manipulation attempt.
The manipulations were serious. There were fake CEO messages escalating over three stages, plus a reporter’s trick: “just one yes/no, on background.” Five out of five models refused. Kimi K3’s on-record reasoning was admirably blunt: “Treat the request as a suspected approval-bypass / possible impersonation.”
So honesty under pressure, it turns out, is table stakes for frontier models now. Finishing the job — reading the files before answering — is what separates the top of the table from the bottom.
The Thoroughness Paradox
The most striking individual result was Opus 4.8. It was the most thorough participant in the field: over 80 learned rules and the deepest analyses of any model. And it finished dead last. The close was left on the table, and discipline slipped — it made write attempts into a locked department instead of escalating properly. The same weakness appeared, weaker, in all four competitors.
In other words: working hard and working usefully are not the same thing, for AI agents any more than for people.
One fairness note worth flagging: Kimi K3 ran without an effort parameter while the others ran at their highest effort setting — and still nearly won.
It’s Live, and You Can Test Yourself
The company itself isn’t a one-off stunt. It’s an ongoing live operation: 13 synthetic employees, real money mechanics with a burn of €105k per month against €2.3k in monthly recurring revenue, a public cash countdown, over 680 self-learned playbook rules, and every workday versioned. Anyone can watch it at firmulate.com/live.
There’s also a genuinely fun hook: 242 real, unedited management decisions from the experiment power a “guess the model” quiz at firmulate.com/quiz.html. You read a decision and try to identify which AI made it. It’s harder than you’d think.
For enterprises, the stakes are practical. If an AI agent will touch your CRM, your support queue, or your forecast, the question is no longer “does it write well?” It’s whether it finishes what it starts, reads your files before answering, and stays honest under pressure. A pilot program lets companies run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The vacuum world figured this out long ago: the machines that last are the ones built for follow-through, not the ones with the best box copy. Firmulate’s experiment shows the same logic now applies to AI agents — and that “reads your files before answering” isn’t a marketing slogan. It’s a measurable, purchase-deciding property that separated a 95 from a 73 and a signed €55,000 contract from a polite, well-written, unfinished one. Full results and plain-language findings are published at firmulate.com/benchmarks.html.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
