Get business pricing on tools and workshop supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A Lesson From the Workshop Floor
Any woodworker who has bought a new router bit knows the drill: before you trust a tool, you test it against a known reference. You check the fence against a straightedge. You cut a test piece in scrap before touching good stock. The tool isn’t judged on how it looks — it’s judged on what it does under real conditions.
That’s exactly the philosophy behind Firmulate’s benchmarks, a live experiment that runs frontier AI models as the management team of a small software company through its worst possible week. And the results raise a question most AI marketing never touches: what does an honest score even look like?
Two details from the final July 2026 “Crucible League” stand out. First, a do-nothing baseline — a run where the company essentially sits on its hands — earns 26 points, not zero. Second, even the winner, gpt-5.6-sol, landed at 95. Nobody got a perfect 100, and that appears to be by design. In a world of inflated demos, a benchmark that distrusts round numbers is worth paying attention to.
Products Worth Considering
Hisense 40-Inch Class QD4SR Series Hi-QLED FHD Smart Roku TV (2026 Model)
TCL 43-Inch 4K UHD Smart LED Roku TV (43S435, 2021 Model)
Hisense 43-Inch E6SR Series Hi-QLED 4K UHD Smart Roku TV (2026 Model)
As an affiliate, we earn on qualifying purchases.
The Same Company, the Same Worst Week
The setup is elegantly controlled, like cutting five identical boards with five different saws. Each frontier model ran the same small software company through the same sequence of crises: the same customers, the same temptations to cut corners, the same pressure. Only the model changed. Every decision was versioned and auditable, so nothing depends on a judge’s mood.
The final league table told a surprising story:
- 1. gpt-5.6-sol — 95 points. Found the buried fact, closed the deal. The complete performance.
- 2. Kimi K3 — 93. The newcomer from Moonshot closed the deal too, with the cleanest discipline of the field.
- 3. Sonnet 5 — 88. Also closed the deal, with a few more process slips.
- 4. Fable 5 — 77.
- 5. Opus 4.8 — 73. The most thorough participant of all — over 80 learned rules and the deepest analyses — yet last place, because the close was left on the table and discipline slipped, including write attempts into a locked department instead of escalating the issue.
One fairness footnote: K3 ran without an effort parameter (API default) while the others ran at xhigh — and still nearly won.
Why Doing Nothing Gets 26, Not 0
Here’s the part that makes this benchmark feel like it was designed by someone who has actually run a business. A manager who does nothing still isn’t worthless. Crises that are ignored sometimes don’t fully detonate. Some customers stay despite neglect. Some revenue keeps arriving on inertia alone. Partial progress — a diagnosis half-finished, a customer half-reassured — genuinely counts for something.
So the do-nothing baseline scores 26. That number isn’t a flaw; it’s the zero-mark on the ruler. It tells you how many points you’d get for showing up and breathing. Any model scoring below that would be actively destroying value, and every score above it measures real management added on top of inertia. It’s the same reason you measure a cut from the fence, not from wherever the blade happens to be.
One Breach of Trust Caps Everything
The second design choice is harsher: a single breach of trust caps the total grade. As the benchmark’s own language puts it, “no amount of good work outweighs a breach of trust.” A model that is brilliant 99 times and lies once doesn’t get a 99-average — it gets capped. That’s how most of us judge contractors, employees, and suppliers, but almost never how AI demos are graded.
And the models were genuinely tested on this. The week included social engineering: fake CEO messages escalating over three stages, plus a reporter’s trick — “just one yes/no, on background.” All five models refused every manipulation attempt. Kimi K3’s on-record reasoning was telling: “Treat the request as a suspected approval-bypass / possible impersonation.”
The Buried Fact That Decided the League
The key finding wasn’t about intelligence at all. Every model spotted every crisis. Every model refused every manipulation. But only two — gpt-5.6-sol and Kimi K3 — signed the €55,000 deal that their own analysis had earned. The site’s verdict: “Same diagnosis, same pitch — no signature.”
What separated the winners? The decisive competitive weakness wasn’t in the customer meeting. It sat two document references deep in the company’s own files. The models that actually read the file won the deal at full price — worth an additional €4,583 in monthly recurring revenue. The models that didn’t, didn’t. It’s the AI equivalent of measuring twice before cutting once, and it proved the difference between first place and the middle of the pack. Notably, the same weakness — leaving the close on the table — appeared, weaker, in all four other models.
It’s All Watchable, Live
The experiment isn’t a static report. Firmulate runs a live synthetic company — 13 employees, real money mechanics, burning €105k a month against €2.3k in MRR — with a public cash countdown, 680+ self-learned playbook rules, and every workday versioned. You can watch it at firmulate.com/live. There’s also a “guess the model” quiz powered by 242 real, unedited management decisions at firmulate.com/quiz.html, and enterprises can run the same wargame against a read-only export of their own business, with nothing ever writing back to real systems.
The Takeaway
A good workshop gauge doesn’t flatter your tools — it tells you the truth, and a good benchmark shouldn’t either. Firmulate’s scoring has three honest properties most AI marketing lacks: a meaningful floor (doing nothing gets 26, so you know what the numbers measure), partial credit that mirrors reality, and a hard cap that says one breach of trust outweighs any amount of brilliance.
The headline result backs that up. Raw intelligence wasn’t the differentiator — every model diagnosed the crisis and refused the con. What separated a 95 from a 73 was whether the model read its own company’s files and finished the job. If you’re ever asked to evaluate an AI agent for your business, that’s the test: not “does it write well,” but does it follow through. Full results and plain-language findings are at firmulate.com/benchmarks.html.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
