Skip to content
Chukwudum Chukwuedo

Field note

A prompt is not an AI system

The demo is the easy part. The system is the part that has to survive Monday.

6 min read

AI systems, not AI demos.

The claim

A prompt is not an AI system. A prompt is a sentence that produces an output. An AI system is the arrangement of people, data, permissions, evaluation, monitoring and fallback that decides whether that output can be used, by whom, and under what conditions. The distinction sounds pedantic until something goes wrong and the organisation discovers it has demonstrated a model but not built a capability.

Most AI demonstrations are designed to make the model look good. A production system is designed to make the organisation look after itself when the model does not.

The mental model

I think about an AI system as four layers that have to agree with each other.

The first is the model: its accuracy, its failure modes, its sensitivity to inputs that were not in the demonstration. The second is the data and evaluation layer: where the evidence comes from, who owns it, how it is refreshed, and what “good” means in the language of the people who will use the result. The third is the authority layer: who can see the output, who can act on it, what the system is allowed to do without asking, and what has to be approved. The fourth is the operating layer: monitoring, escalation, rollback, audit, and a named owner when the system behaves unexpectedly.

A prompt can sit inside all four layers and still leave every one of them unresolved. That is why “the model scored well” is not an answer to “is this ready?”

The useful question is not “what can the model do?” It is “what happens on day ninety, when the data has drifted, the original team has moved on, and someone is relying on the output in a meeting they cannot attend?”

The evidence

On a national public-health publications service, the model was never the hard part. The service had to move from Alpha to Beta, pass its GDS assessments first time, and reduce inbound publication and support requests from around fifty a week to fewer than five. That work depended on semantic search and personalisation, but it also depended on content ownership, publishing workflows, accessibility, security, and the confidence of the people who had to use it.

On a national surveillance programme, the pipeline design had to handle around 250,000 records a day today and a surge envelope of two to ten million. The interesting constraints were not model choices. They were data contracts, environment separation, Caldicott and DTN obligations, a DPIA, change advisory board reviews, and the discipline of testing across environments. A system that only works on the demonstration data is not a system; it is a rehearsal.

In both cases, the evidence that mattered was operational: who used it, what changed in their work, what happened when the data was late, and what the team could prove after the pilot.

The implication

If you are evaluating an AI proposal, ask for the system, not the prompt. Ask what the model is allowed to decide, what a person is allowed to override, where the evidence is stored, who is accountable for a wrong answer, and how the organisation would know. Ask what happens when the model is unavailable. Ask what the team would switch off first if the risk changed.

Those questions are not blockers. They are the design. The demo can be a sentence. The system has to be an argument that survives contact with operations.

I have watched capable teams lose months because the demonstration created an expectation that the surrounding work was optional. The model was convincing. The data ownership was unclear. The review process had no authority. The monitoring was a dashboard nobody had agreed to watch. None of those failures were model failures. They were system failures that the model made visible.

That is the difference I care about: not whether the model can produce a convincing answer once, but whether the organisation can trust the answer repeatedly, under pressure, with its own name attached.