From AI Demo to Production: Why Most Business AI Projects Stall
Why do most AI projects fail to make it to production?
The widely-cited statistic that 95% of AI projects fail comes from an MIT report — but the author has since said his words were misinterpreted by the media.
As Vladimir notes, "the author of this research just released a statement a couple of days ago saying that his words were completely transformed by the media." The real failure mode is more practical.
Abraham from Google points out that it's easy to create an AI project , which is part of why the failure rate looks so high — the definition is loose.
The harder problem is that teams don't think about evaluation before they dive in: There are millions of ways a user could interact with a system, and you can't humanly think of all the edge cases.
Without structure and testing upfront, projects "get wild and get very crazy very quickly." That's when a lot of people quit these projects .
What is the gap between an AI demo and a deployed workflow?
The gap isn't the model — it's everything around it.
Abraham's framing: before building anything, ask "how would you guys evaluate the system, whatever it is.
But knowing how you're gonna evaluate at scale is really important." If your current evaluation depends on one domain expert eyeballing outputs, that won't scale.
The real project becomes two things at once: Build the AI system.
Replicate your evaluation person — by sitting down and documenting their domain understanding.
Eliott from VanMoof frames the production bar differently for hardware and high-stakes domains: 86% accuracy on complex design decisions "will kill us." A bike has to be consistent within tight tolerances 99.9% of the time, and the 0.1% has to be caught before delivery.
For finance, accounting, and taxes, "it has to be the truth." The demo-to-production gap is the work of closing that last accuracy mile — or designing around it.
How should teams prioritize which AI workflows to build first?
Both work streams — internal productivity tools and strategic company-level AI projects — have more opportunities than any team has resources for.
So prioritization and accountability matter even when building is fast.
Charlotte from ScorePlay uses a simple heuristic: most time-consuming, smallest value add .
That's where automation pays off without sacrificing the human judgment you actually need elsewhere.
For ScorePlay, that meant automating proposal-building work like adjusting Canva templates — "low value add from the cells, highest time spent" — while keeping objection handling human, because that's where understanding the prospect matters most.
Vladimir notes this always comes back to a simple impact-vs-effort matrix.
Once you automate one step, the bottleneck shifts to the next step — so anticipate where the next constraint will land.
“knowing how you're gonna evaluate at scale is really important”
How do you build trust in AI tools across an organization?
Trust comes from treating AI as an experiment, not a mandate.
Abraham's framing: "culturally, you don't want your company to be scared of AI.
It's a tool." Many people don't know what they're doing yet, but they're willing to experiment and see where it falls behind.
Practical patterns that build organizational trust: Make it a low-barrier experiment.
Processing your own documents with AI is a five-minute exercise you can repeat every two months to see how far it's getting.
Give non-engineers access to the codebase.
At Lemlist, all PMs have a dev setup identical to the tech team and can ask questions of the codebase to understand a decade of legacy decisions.
Be honest about what AI can't do yet.
Eliott: AI is incredible as a research tool and good at copy, image, and audio, but he doesn't trust it 100% yet because "you still get random errors… little inaccuracies."
What does it mean to design a 'stochastic-first' organization?
Vladimir frames the second half of AI adoption as organizational: "how can you create a company that is stochastic first, where basically, like, you factor in the fact that AI isn't predictable, and you have to shape your operations accordingly." Eliott agrees that some businesses can do this and some can't.
Real-world hardware companies and anything touching financial decisions, accounting, or taxes can't absorb stochastic error — the output has to be the truth.
Other businesses can absorb a known error rate and design review steps around it.
The decision of whether to go stochastic-first isn't a tooling decision; it's a business-model decision about how much variance your output can tolerate.
Frequently asked questions.
- Is it true that 95% of AI projects fail?
- The 95% figure comes from an MIT report, but the author of that research released a statement saying his words were completely transformed by the media and that his analysis was misinterpreted. Vladimir notes the paper has been around for almost a year and the narrative is now entrenched, even though the research isn't scientifically sound enough to support that conclusion. The real issue is that it's very easy to create an AI project, so the failure rate depends heavily on how you define one.
- What's the first question to ask before building an AI system?
- How will you evaluate it at scale. Abraham from Google frames this as the most important upfront question: if your current evaluation depends on one domain expert reviewing outputs, that won't scale. The real project becomes building the AI system plus replicating your evaluation person — which means sitting down and documenting that person's domain understanding so the evaluation can be automated alongside the system itself.
- How do you decide which workflows to automate first?
- Charlotte from ScorePlay uses a simple rule: target the most time-consuming, smallest value-add tasks. For her sales team, that meant template-adjustment work in Canva — high time spent, low judgment required — while keeping objection handling human because that's where understanding the prospect matters most. Vladimir frames it as an impact-vs-effort matrix, and notes that once you automate one step, the next bottleneck appears immediately, so you should anticipate it.
- Can every business adopt a stochastic-first operating model?
- Eliott from VanMoof argues that hardware companies and businesses making financial, accounting, or tax decisions can't absorb AI's error rate — a bike has to meet tight tolerances 99.9% of the time, and tax filings have to be the truth. Other businesses can shape their operations around AI's unpredictability by adding review steps and factoring failure modes into the workflow. The decision isn't about tooling — it's about how much output variance your business model can tolerate.
- How can non-technical teams build trust in AI tools?
- Treat AI use as an experiment, not a mandate. Abraham from Google recommends low-barrier exercises like processing your own documents — a five-minute task you can repeat every couple of months to see how capability is evolving. At Lemlist, every PM has a developer setup identical to the engineering team's, giving them direct access to the codebase so they can ask it questions and understand a decade of legacy decisions. The cultural goal is that the company isn't scared of AI.
- Where does today's AI fall short for production use?
- Eliott from VanMoof says AI today is incredible as a research tool and strong at copywriting, image generation, and audio — good enough to deliver professional outputs if trained for it. Where it falls short is reliability: you still get random errors and little inaccuracies. It can do a thorough analysis of an Excel document and produce a model that looks right, but you have to be ready for the fact that people will notice when AI is used, and you can't fully trust the output yet.