Why most AI pilots never reach production

A pilot that works is not evidence that a system will work. It is evidence that a model produced acceptable output on a curated sample, in a notebook, for an audience that wanted to be impressed. Nearly every AI program I have been asked to review had already cleared that bar. Very few had cleared the next one.

The gap is almost never the model. Foundation models are good enough for the overwhelming majority of use cases a mid-market company will attempt this decade, and the ones that are not good enough are usually obvious in week one. What separates the pilots that ship from the pilots that quietly stop being mentioned is organizational, and it shows up in three specific places.

Nobody owns it after the demo

Pilots are typically run by whoever was curious — a data science team, an innovation function, sometimes a single motivated engineer. That is a fine way to start and a terrible way to continue, because none of those people control the surface the feature has to land on. The moment the work needs a change to the checkout flow, the support tooling or the merchant onboarding path, it enters a queue owned by someone who was not in the room and has no incentive to prioritize it.

The fix is unglamorous: name the executive who owns the outcome, not the experiment, and give them the roadmap slot before the pilot starts. If no one will accept the outcome on their own scorecard, that is the finding. Stop there and save the quarter.

If no executive will put the outcome on their own scorecard before the pilot starts, that is the finding.

The data works in the pilot because someone cleaned it by hand

The second failure is a data contract that does not exist. In a pilot, inputs are assembled once, by a human, from whatever sources happened to be reachable. In production, those inputs arrive continuously from systems owned by teams who never agreed to keep the schema stable, never agreed to a freshness guarantee, and will change both without telling you.

This is the failure I have seen do the most damage, because it degrades silently. The model keeps returning confident output; the output is just progressively less connected to reality. By the time anyone notices, trust in the feature is gone and it is very expensive to get back.

Before the pilot, write down for each input: who owns the source, what the freshness guarantee is, what happens when it is missing, and who gets paged when it breaks. If that document is hard to write, you have found the real project — and it is a data project, not an AI project.

There is no operating cadence to carry it

The third failure is the quietest. A pilot runs on enthusiasm, which is abundant for about six weeks. Production runs on cadence: a weekly review where the metric is on the wall, a named person answers for the number, and decisions get made in the room rather than deferred.

At Wish, the work that actually moved the business was not a model. It was listing quality, refund rate and time to door — the operational substrate underneath discovery. Getting the refund rate from 40% to 3.5% and time to door from 35 days to 10 days is what moved NPS from negative territory to +40. The AI-driven discovery experience mattered, but it mattered on top of an operational floor that had to be rebuilt first, and rebuilding it took a cadence, not an insight.

That is the general shape. AI amplifies whatever operating discipline you already have. If the discipline is absent, the pilot is a very expensive way to discover that.

What to do instead

• Name the executive who owns the production outcome before the pilot starts, and secure the roadmap slot on the surface it will ship to.
• Write the data contract for every input — owner, freshness, failure behavior, escalation path — and treat difficulty writing it as the finding.
• Put the metric in an existing weekly review rather than creating a new forum for it. New forums decay.
• Set the kill criterion in advance. A pilot with no defined way to fail will not be stopped; it will be starved, which is slower and more demoralizing.
• Sequence for the operational floor first. If the substrate is broken, fix the substrate — the model will still be there in a quarter, and it will be cheaper.

None of this is an argument against moving quickly. It is an argument that the constraint is rarely where teams look for it. The organizations that get AI into production are not the ones with better models. They are the ones that treated the pilot as the beginning of a delivery problem rather than the end of a research one.