A pilot can look good in the demo and stop when it meets the real process. One explanation is that the model is not smart enough yet, or that the team needs a better tool. My read is that this is the wrong explanation.
A 2026 preprint by Jeanne McClure and Gregg Gerdau also looks past the model. They synthesized 19 industry and academic sources, including surveys of nearly 10,000 organizational leaders. Their abstract opens: “Global corporate AI investment reached $252.3 billion in 2024, yet only 6% of firms report significant earnings impact.” Then it names the cause: “AI project failure is fundamentally an organizational learning problem rather than a technology deficit.” Those are the authors’ figures from the sources they pooled, in a paper I have not seen peer reviewed. They are not a measurement of procurement.
What the demo left out
A demo runs on sample data someone cleaned, and it asks a question with one right answer. The real process is different. My read is that it runs on part numbers that do not match across systems, prices that live in email, and rules that exist only in someone’s head.
A model cannot retrieve what was never recorded. It cannot apply a rule nobody wrote down. It looks like the model failed at judgment. It is an undocumented policy.
None of this is new to machine learning. In 2015, D. Sculley and nine co-authors argued that “it is dangerous to think of these quick wins as coming for free,” and found it “common to incur massive ongoing maintenance costs in real-world ML systems.” Their list of risk factors includes data dependencies. The paper is about the system around the model, not the model.
What a shared record looks like
A standard shows what the fix is made of. GS1 describes a GTIN as “a GS1 identification key used to identify a trade item,” and as “a globally unique number.” The same page says other information is shared with trading partners through the GS1 Global Data Synchronization Network.
That is the shape of a record that works. One key. Everyone reads the same one. A plant can run on keys it made itself, one per system. Nobody has to decide that two keys mean the same item. They just accumulate.
The order of work
Start with the decision, not the tool. Four steps, in this order.
- Write the rule down. One sentence a new buyer could follow. If you cannot write it, the model cannot apply it.
- Fix the record the rule needs. Only the fields the rule reads. Do not clean the whole database.
- Then automate the retrieval. My read is that finding the numbers is the part a model does well, once the numbers exist in one place.
- Keep a person on the decision. The agent prepares and evidences it. A human approves it.
This is what building an agentic app means, as opposed to building an agent. Fixed rules where rules exist. Judgment where it is needed. A person approving the decision. The model is the easy part.
Three places show where the work is missing. Retrieval: people spend the day finding numbers instead of deciding with them. The wait: finished work sits until someone approves it. The re-argument: the same decision gets made from scratch because nobody wrote the rule down. My read is that a better model fixes none of the three. A written rule and a matched record fix the first and the third, and they make the second short.
Before the next purchase, answer six questions. Who it serves. Why it is worth it. What process changes. Where it applies. When you decide. How you will know. Ask them before the demo, not after.
One thing to check this week
Take the last ten times your team approved a price change or a supplier substitution. Write the rule behind each one in a single sentence. Then, for each, list the fields that rule needed: part number, current price, last price, quantity, source.
Count how many of those fields you could pull from one system without opening email, and how many part numbers matched across the systems involved. That count is your AI readiness. It is also your first project.
Sources
- Jeanne McClure and Gregg Gerdau, Why AI Readiness Is an Organizational Learning Problem, Not a Technology Purchase, arXiv preprint 2604.16369, submitted 22 March 2026.
- D. Sculley et al., Hidden Technical Debt in Machine Learning Systems, Advances in Neural Information Processing Systems 28, 2015.
- GS1 US, What is a GTIN?, undated.