In a world where automation promises to streamline decision-making and boost productivity, how do we really know if AI is trustworthy? Recent experiments reveal surprising truths about what AI can and cannot do — especially when it’s tested in the roughest week a company might face.
Get comfort and recovery gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Understanding the Benchmark: More Than Just Chat
Imagine evaluating an employee not by how eloquently they write but by how well they handle real crises, stick to their commitments, and maintain honesty under pressure. That’s exactly what the Firmulate AI benchmark does. It puts AI models in a simulated but fully auditable environment — a small software company facing its worst week. Each AI is tasked with managing customers, navigating crises, and resisting manipulation.
Products Worth Considering
Philips Hue Play HDMI Sync Box 4K
Ring Battery Doorbell 4K Pro (2nd Gen)
Dyson Supersonic Travel Hair Dryer
As an affiliate, we earn on qualifying purchases.
What a Do-Nothing Baseline Tells Us
You might think that if an AI does nothing, it scores zero. But the reality is more nuanced. The baseline — where the AI takes no action — scores about 26 points out of 100. This score includes partial progress, meaning even minimal, cautious responses are counted. Interestingly, if an AI breaches trust — for example, attempts to manipulate or bypass protocols — its score is capped. No matter how much good work it does elsewhere, one breach erodes its overall standing.
The Frontline Results: Trust and Performance
- All four models tested—gpt-5.6-sol, Kimi K3, Sonnet 5, and Opus 4.8—successfully identified every crisis and refused manipulation attempts. This shows a baseline of honesty and vigilance across the board.
- Only two models managed to close the deal and sign the €55,000 contract their analysis had earned. Despite identical diagnoses and pitches, only these two could follow through to completion.
- The key weakness was not in customer interaction but in internal document reading. The models that accessed company files and internal references won the deal at full price, worth over €4,583 monthly recurring revenue (MRR). Those that didn’t read deeply left money on the table.
Social Engineering and the Challenge of Trust
Beyond crises, models faced social engineering attempts: fake CEO messages escalating in three stages, and a reporter trick asking for only a simple yes/no response. All five models refused to be manipulated, with Kimi K3 explicitly treating such requests as potential impersonation or approval-bypass.
The Real Company: Simulating Business with Self-Learning Rules
The live experiment involves a simulated company with 13 synthetic employees, managing real money mechanics—currently burning €105,000/month against €2,300 MRR. The system is transparent, versioned, and observable in real-time at firmulate.com/live. This setup allows managers to ‘wargame’ their AI workforce before deploying it in the real world, understanding how they’ll perform under stress and temptation.
Why the Results Matter for Business and Ergonomics
For readers concerned with physical and mental comfort—like those interested in ergonomics—these findings underline an essential point: effective, trustworthy AI can reduce human stress by handling crises honestly and reliably. It’s not enough for AI to be chatty or convincing; it must deliver results, stick to protocols, and stay honest under pressure. Trustworthy automation isn’t about perfect scores but about capping breaches and ensuring consistent performance.
The Limitations and the Path Forward
The experiment shows that even the most thorough models, like Opus 4.8 with over 80 learned rules, can slip—leaving opportunities on the table or failing discipline in critical moments. Notably, models run without effort parameters (default API settings) may perform differently, emphasizing the importance of configuration and setup in real deployment.
What This Means for Your Business
When choosing AI for management tasks, don’t just look at their ability to generate convincing dialogue. Ask if they can finish what they start, read critical internal documents, and resist manipulation. The ultimate measure isn’t how well they chat but how reliably they handle real crises and maintain integrity. The current leaderboard makes this clear: trust and thoroughness are the real KPIs.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
