In July 2025 MIT’s Project NANDA published a study of generative AI in business. It drew on 52 executive interviews, a survey of 153 leaders, and 300 public deployments. The headline finding: 95% of pilots produced no measurable impact on profit and loss. Over 80% of the organisations surveyed had tried tools like ChatGPT or Copilot. Nearly 40% said they had deployed something. Almost none could point to a number that moved.
The report is an industry study from an MIT Media Lab initiative rather than peer-reviewed research, and the 95% figure has been argued over since. But the pattern it describes matches what we see, and the more useful part of the report is what distinguished the 5%.
The pilot that succeeds is a boring one
The deployments that produced returns were overwhelmingly in the back office: document processing, procurement, customer service routing, finance operations, the repetitive and measurable work that no customer ever sees.
The pilots that failed were disproportionately the visible ones. Sales and marketing tools, customer-facing assistants, the executive-sponsored chatbot that demoed well at the offsite. These got the budget because they photographed well, and they stalled because the gap between a good demo and a system you can trust in front of a customer is enormous.
We made the same argument from a different angle in where AI earns its keep: the tasks where a wrong answer is obvious are nearly free money, and the tasks where a wrong answer looks correct need verification that costs more than the model. The MIT data is what that looks like at the scale of an organisation.
Buying beat building, by a lot
The report found that purchased tools and partnerships reached production about two thirds of the time. Internally built systems reached it about one third of the time.
- Bought from a vendor or built with a partner67%
- Built in-house33%
View as table
| Item | Reached production |
|---|---|
| Bought from a vendor or built with a partner | 67% |
| Built in-house | 33% |
This surprises technical teams, who assume a bespoke build will fit better. It fits better on the whiteboard. In practice an internal AI build competes for the same engineers as every other priority, has no vendor whose job depends on it working, and tends to be scoped as a demo rather than a product. The vendor’s tool has already survived the three things that kill internal builds: integration with the systems the business runs on, a plan for what happens when the model is wrong, and somebody paid to keep it running.
The exception is when the task is specific enough that no vendor covers it, which is real but rarer than teams believe.
Pilots die in the gap between demo and workflow
Usually the model was accurate enough. The pilot was never wired into the way work flows.
A summarisation tool that requires someone to copy text into a separate window and paste the result back is a second job. The people it was meant to help use it for a fortnight, then stop, and the pilot is quietly declared a learning experience.
The systems that stick live inside the tools people already have open. The triage runs when the ticket arrives rather than when somebody remembers to run it. The draft appears in the document. The extracted fields land in the CRM instead of a spreadsheet somebody has to reconcile. Getting that integration right is most of the engineering, and it is the part the demo skipped.
Nobody owns it after the launch
The report calls the core problem a learning gap. Most deployed tools do not retain context, do not improve from feedback, and are not adapted to the workflow over time. They are launched as finished products when they should be launched as the first version of something that will need adjusting for months.
This is an ownership problem before it is a technical one. A pilot has a champion for the duration of the pilot. Then the champion moves to the next thing, and the tool sits there behaving exactly as it did on launch day while the business around it changes. Prompts drift out of date. The model gets replaced by its provider. The one person who understood the setup leaves.
Every AI system we have shipped has a named owner and a scheduled review, because we have watched what happens without one. Our own tooling runs the same problem past several models and flags where they disagree, which we described in why we make four AI models argue. That is a form of ongoing ownership built into the system, and it exists because a single model’s answer ages badly when nobody is watching.
Measure a number before you build
The 95% figure exists partly because most pilots never defined what success would look like in a form that could fail.
Before writing a line of code, name the metric. Hours of manual processing per week. Median time to first response on a support ticket. Percentage of invoices coded without human correction. Whatever it is, record the current value, decide what change would justify the cost, and agree who will read the number in three months.
A pilot that cannot fail against a metric cannot succeed against one either, and it will drift into the 95% by default.
What to do with this
If you have a pilot running now, ask three questions. Is it wired into the tools people already use, or does it sit beside them? Does a named person own it for the next twelve months? And is there a number, written down, that it was supposed to move?
Fix any no before adding a feature.
If you are choosing a first project, choose the boring one. The invoice matching, the enquiry routing, the report that takes someone a day a week to assemble. Buy it if a vendor does it well. Build it only if the task is genuinely yours. Scope it as a product with an owner. A demo with a deadline dies in a drawer.
That is how we approach AI solutions work, and a good part of what we do is telling clients which of their ideas belong in the 5%. Get in touch if you want that conversation.
References
- MIT report: 95% of generative AI pilots at companies are failing, Fortune via Yahoo Finance, August 2025
- MIT finds 95% of GenAI pilots fail because companies avoid friction, Forbes, August 2025
- The AI Divide, DX, analysis of the NANDA report