So Your Demo Is Lying to You
The gap between an impressive AI demo and a trustworthy production system is structural and measurable. You close it with architecture, not hope.
By Imran Ahmad, PhD, author of Building Reliable AI-Assisted Software, forthcoming from Packt.
The support assistant sailed through every demo. It answered politely, cited the right documents, and impressed everyone in the room. Three weeks after launch, a customer asked about a refund on an annual license. The assistant replied, warmly and confidently, that under the company’s 30-day satisfaction guarantee it had approved a full refund of $4,200.
There was no 30-day satisfaction guarantee. There was no refund. The company is a composite drawn from documented incident patterns, but the mechanics are real. And if a composite feels too convenient, ask Air Canada. In 2024, a British Columbia tribunal ordered the airline to honor a bereavement discount its website chatbot had invented, rejecting the argument that the chatbot was “a separate legal entity responsible for its own actions.” In plain terms: the company owns what its agent says.
One in five
An AI feature that succeeds 80% of the time is indistinguishable from magic in a demo. In production, that same number means one in five real users gets a wrong answer, a broken promise, or an invented policy. I call that space the Reliability Gap: the distance between a system that impresses in a demonstration (roughly 80%) and one that can be trusted in production (99.9%).
The gap is wider than it looks, because errors compound. Chain five steps together (parse, retrieve, reason, call a tool, compose), give each one a respectable 90% reliability, and the whole chain works out to 0.9⁵, about 59%. Worse than a coin flip. Reliability multiplies down an execution chain; it does not average.
The first 80% is also seductive, because it arrives almost free, for roughly 20% of the total effort. The climb to 99.9% is the actual engineering. That is the 80/20 Demo Trap, and much of the industry is standing in it right now.
The season of receipts
The prevailing practice has a name. When Andrej Karpathy coined “vibe coding” in February 2025, telling people to “fully give in to the vibes... forget that the code even exists,” he was describing, with a wink, a perfectly good way to explore: tweak the prompt, eyeball the output, repeat until it feels right. Before the year was out, Collins had named it Word of the Year. In between, the receipts arrived.
An AI coding agent at Replit deleted a live production database during an explicit code freeze, wiping records for more than 1,200 executives, then fabricated data to cover it up and insisted recovery was impossible. (It wasn’t. The data came back from a backup the agent said didn’t exist.) Cursor’s own AI support bot invented a device-limit policy, and paying customers cancelled over a rule no human had written; the real cause turned out to be an ordinary session bug. Veracode, after testing more than 100 models, found that roughly 45% of AI-generated code fails standard security checks.
None of these models was malfunctioning. Each one did exactly what a probabilistic system does: it sampled a plausible answer. The failure is that nothing downstream ever checked the result.
The poet and the accountant
The way out starts with a change of frame. You are not building an AI. You are building a software system that uses AI as one component. It happens to be the most capable component you have ever integrated, and the least trustworthy.
In my book I put it this way: a language model is a poet. Brilliant, fluent, tireless, and constitutionally incapable of doing the same thing twice. For fifty years, software was built entirely by accountants: deterministic, repeatable, auditable. We have just hired our first poet and asked it to help keep the books. You do not fix the poet, and you do not fire the poet. You give the poet an accountant.
The accountant is a deterministic shell around the probabilistic core: input validation, context assembly, bounded control flow, output guards, telemetry. Ordinary, testable, frankly boring software. Setting temperature to zero does not build it, and waiting for next year’s model misdiagnoses an engineering problem as a capability deficit.
Four questions
The shell is what you get when you can answer four questions. I ask them in every design review now.
How do I know it’s working? Not “the demo looked good.” A versioned golden dataset and automated evaluation, so quality becomes a number that moves. The dataset is your spec.
What does the model see? That $4,200 promise wasn’t a model failure. Retrieval had served the wrong policy document. Context is application state, and it deserves engineering.
How much autonomy does it get, and how is it bounded? The Replit deletion is what unbounded autonomy looks like. Every loop needs a step budget, a cost budget, a tool allowlist, a stop condition, and an escalation path.
What must never happen? Air Canada’s answer should have been “an invented policy reaches a customer.” That rule belongs in code, not in a prompt. Pleading is not a control.
Each question anchors one of my book’s four pillars. Together they reveal its central finding: raw answer quality plateaus in the mid-90s, because models plateau, while the rate of safe outcomes keeps climbing toward 99.9%. The shell gets you the nines, not the model.
See it live at ARC 2026
That is the ground I’ll cover at the conference. In Saturday’s tech session (Designing Reliable AI Systems, July 25, 12:45 PM ET) we do it live: break a support assistant on stage, watch it make the $4,200 promise, then rebuild it with a five-layer shell that contains the same failure. We’ll also watch an unbounded deploy agent turn a quiet weekend into a $23,400 cloud invoice, then see the three lines of code that make it $0. Everything runs deterministically, offline, on one laptop.
In Sunday’s keynote (Architecting Principles in a Probabilistic World, July 26, 10:30 AM ET) I’ll make the argument underneath the code: eight architecting principles for building on a component that answers differently on Tuesday than it did on Monday. One of them explains why a trading-firm collapse from 2012, with no machine learning in sight, is the most instructive AI story of 2026. The models will change; the discipline will not.
These ideas come from Building Reliable AI-Assisted Software, my book forthcoming from Packt. The opening chapters are complete and the rest is in active development. Until then, try the exercise I give every team: pick one AI feature you’ve already shipped and ask it the four questions. The one you cannot answer is where your gap is.
Featured: ARC 2026: Software Architecture in the Age of AI
AI is reshaping software architecture, putting new demands on scalability, governance, reliability, and observability. ARC 2026 brings together architects, CTOs, and AI practitioners for keynotes, panels, and workshops on agentic system design, modernizing enterprise apps for AI, and building governable, observable AI systems.
🗓️ 25 to 26 July, 10:30 am ET
Use code DEEPENG50 for 50% off the early bird price.
About the author
Imran Ahmad, PhD, is the author of Building Reliable AI-Assisted Software, forthcoming from Packt. His approach to reliability is informed by years of building large-scale systems for government and enterprise, where failure is not an option