The North Sea at night under a low sky, one distant light on the horizon

Field notes Operations

The model is the easy part. Running AI in production is an operations job.

November 2026  ·  6 min read

AI systems are bought on the demonstration: one task, one good answer, one impressive afternoon. Nothing in a demonstration runs twice. Production is the same system running every hour of every day, against real queues, real schedules and real people's attention, and the problems that decide whether it is worth having are almost never about the model.

They are operations problems. Work raised twice. Alerts that fire, clear and fire again until nobody reads them. Jobs that stop without failing. Queues that grow because nobody owns the bottom of them. A monitor that has stopped looking exactly like a system that is healthy. None of these is new. Automation makes each of them arrive faster and at a volume people cannot absorb by hand, so each needs a control designed in, not a person told to keep an eye on it.

This note sets out the operating disciplines we hold an AI system to before we call it running, and the question that tests each one.

A duplicate is the first failure, not the last

Any system that observes and reports will rediscover the same thing. A check that runs every hour finds the same misconfiguration every hour. Ten processes reading the same signal raise ten versions of the same concern, each worded slightly differently. Without a gate, the work queue fills with copies, and the people reading it learn the lesson that matters most: most of what arrives here can be ignored. After that, the one item that cannot be ignored is ignored too.

The control is a deduplication gate in front of the queue, not a clean-up behind it. Each observation carries a stable key: what was seen, by which process, about which thing. A rediscovery updates the existing record, marking when it was last seen, and does not raise a new item. The gate also keeps a record of what it turned away, because a gate that rejects nothing is not deduplicating, and one that rejects everything has stopped reporting.

Ask: what stops your system raising the same issue twice, and can you show me what the gate rejected last week?

An alert that cycles teaches people to ignore alerts

An alert that fires, clears itself and fires again is the most expensive signal in operations. Each repetition costs a moment of attention and returns nothing, and the cost compounds: by the tenth time, the alert has trained its readers to dismiss it on sight. Automated systems produce these at a rate no human operation would, because they check more often and recover faster, so the same fault flaps on and off all day.

The fix is not a higher threshold. Alert on a condition that has persisted, not on a single reading. Require the condition to clear properly before the alert can fire again. And treat an alert that recurs as a defect with an owner and a deadline, not as background noise to be muted. The measure that matters is not how many alerts fired but how many led to someone doing something.

Ask: which of your alerts fired most often last month, and what changed because of it?

Stuck is a state, and it needs its own alarm

A component that crashes raises an alarm. Work that neither finishes nor fails raises nothing. A job claimed and never completed, a task waiting on an approval nobody was told about, a retry loop that keeps the item looking busy: on every dashboard these read as work in progress. In an AI operation they are common, because a model call can hang, a tool can return nothing, and an agent can wait patiently for an answer that will never come.

The control is an age limit on everything in flight. Each kind of work has an expected duration, and anything that exceeds it is raised as stuck, with the item, its owner and how long it has been waiting. The same reasoning applies to absence: our checks assert that scheduled work actually ran, because a job that was never scheduled leaves every dashboard green.

Ask: what is the oldest item in progress in your system right now, and how would you know if it had stopped moving?

Every queue needs an owner and an empty state

AI systems are good at producing work for people: findings, drafts, proposals, items needing approval. They are indifferent to whether anyone reads it. A queue without an owner grows until it is declared a backlog, and a backlog is where an approval request goes to be missed.

Each queue needs a named owner and a rule that every item ends in one of three states: acted on, deferred with a reason and a date, or closed. A queue that is growing faster than it empties is a capacity signal and is reported as one, not discovered at quarter end. Approvals in particular need a route that is acted on in time; if the person who can say no never sees the request, the approval step protects nobody.

Ask: who owns each queue your AI system writes to, and how long does an approval request wait before someone sees it?

Watch the watcher

A monitor that has stopped looks identical to a system with nothing to report. This is the failure every other discipline here depends on, and the easiest to miss: the health check whose schedule was removed in a change, the collector that lost its credentials, the alert route pointing at a channel nobody reads any more.

The control is a heartbeat that is expected, not merely accepted. Each monitor reports that it ran, and the absence of that report is itself an alert, raised by something outside the monitor. Silence is never treated as a pass. Alert routes are tested end to end on a schedule, so the first time anyone learns a route is broken is not the day it was needed.

Ask: if your monitoring stopped tonight, what would tell you, and how soon?

Schedules are a contract

An AI operation runs on a timetable: overnight checks, daily summaries, weekly evaluations. Each scheduled job makes a promise about when something will have happened, and other work relies on it. A schedule that drifts, overlaps itself or runs twice after a restart breaks that promise without breaking anything visible.

Each job needs a stated cadence, a rule for what happens when a run is missed (catch up once, or skip and report), protection against two copies running at once, and a record of every run with its outcome. The record is what makes the schedule checkable rather than assumed.

Ask: for any scheduled job, can you show me every run in the last month and what each one did?

Why this is a governance question, not only an engineering one

Each of these failures ends the same way: a system that appears to be supervised and is not. A board that approved an AI system on the understanding that people review what it does is relying on the queue being read, the alert being believed and the monitor still running. For providers of high-risk systems, the EU AI Act's Article 72 makes post-market monitoring an obligation; for everyone else, it is simply the difference between operating a system and owning one that runs unattended.

None of this appears in a demonstration, and all of it can be asked about. A supplier who answers these questions from their own records runs an operation. One who answers from a slide is describing an intention.

We design these controls into an AI operation from the first service, or run the operation for you to agreed service levels: deduplicated queues, alerts that mean something, age limits on work in flight, owned queues with an empty state, monitoring that is itself monitored, and a run record for every schedule. If your AI system is in production and nobody can tell you the oldest thing it is waiting on, talk to us.

All field notes

Tell us what you are accountable for.

The useful first conversation is usually about the thing that worries you, not about our capabilities. If it turns out we are the wrong firm for it, we will say so.

One reply from a person who has read what you sent.

Colophon

  • Plate 06, the sea at night: generated, Azure OpenAI gpt-image-1.5, 3 November 2026.

Every plate on this page is a generated stand-in for commissioned photography. The prompt that made each one is embedded in its file. How this site was made.