Write the standards, not the steps

Early this year an autonomous research agent had to work in a domain it was never built for: different records, different rules, different questions. I budgeted two weeks for the rework and thought I was being generous.

It took two days. Plenty of it got edited: the section on how identity and duplication worked was rewritten, the tool registrations were updated for the new use case, the knowledge base changed. But nothing had to be rethought.

That story gets told as “the architecture generalised”, which is true and tells you nothing. The commits say something narrower and more useful. Almost the entire prompt described how an analyst is supposed to behave, and almost none of it described the domain. Behaviour is portable in a way that domain knowledge never is, and the ratio between the two in that one file is why the port was cheap.

What it was

An autonomous research agent, which in practice meant this: you hand it an open-ended question (are these two records the same thing, why is this group failing its checks, how far does this problem actually extend) and it comes back with an evidence-backed report that someone with domain expertise can act on. Not a chatbot in any useful sense. The unit of work was an investigation that used to eat an analyst’s afternoon, sometimes their whole day.

One decision behind it took real pushback at the time, and it’s the one that mattered. The frontier models were getting better every few weeks, so spending the prompt teaching the model how to investigate felt like a losing bet: you would be re-teaching a method to something that had already outgrown the version you wrote it for. So the file mostly doesn’t. It says who the model is and what it’s accountable for, and leaves the method alone.

That was the intent, anyway. The first version hedged on it badly, and I’ll come back to that.

The file opens on a job description. You are an investigative analyst. You do not blindly trust the records you are given, and you do not blindly trust the automated systems either; when your evidence contradicts one of them, you are expected to override it and say so out loud. Thoroughness over speed. Keep going while each step is still adding evidence, and stop when it stops rather than when you hit a limit somebody else picked.

Then the standards: how to weigh a piece of evidence before leaning on it, how to calibrate confidence, what to do when the evidence refuses to resolve, and how to report a null result instead of manufacturing a finding to justify the run. There is an estimation audit, which asks the model to label every number counted, verified or extrapolated before writing it down, and never to dress an extrapolation up as a count. There is an adversarial self-check: if the main conclusion is wrong, what is the likeliest way it’s wrong, and was that signal already sitting there, dismissed?

Read that list again and notice what’s absent. None of it is about the domain. Those are the standards of a careful analyst and they hold whatever the records happen to describe, so crossing into a new domain cost nothing to re-specify. The only work was pointing them at different data.

The proportion is the finding

The domain-specific part was a single section on how identity and duplication worked among that particular set of records. Everything else was accountability, judgement and standards. Call it nine parts standards to one part domain, and that ratio is why the port wasn’t a rewrite: rewrite that one section, swap the knowledge base, update the tool registrations for the new use case.

One thing helped mechanically, though it isn’t the lesson. The prompt never names its tools. It describes categories by purpose (something to ground you in the rules before you query, something to map distributions across a large dataset, something to inspect one record in depth, something to check the outside world) and then says tools are registered dynamically, go discover what you have and read each description. That started as laziness on my part. Tools were being added and renamed constantly and a hand-maintained list in prose would have been stale within the week. It happened to mean that changing the tool registrations broke nothing in the prose around them, which was convenient, and the prompt would have ported without it.

A prompt that teaches method ages with the model. A prompt that sets standards outlives it.

The scaffolding that came out

The first version was not this clean. It had plan, execute and verify as three separate, sequential stages: the decoupled-planning family, which ReWOO (Reasoning WithOut Observation, 2023) formalised as planner, worker, solver. Write the plan up front, run the tools against it, compose the answer at the end, and never let an observation reach back into the planning. The opposite pole is ReAct, where reasoning and acting interleave and every observation shapes the next thought.

Staging felt responsible at the time. It’s legible, you can point at which box failed, and there is real comfort in that when you’re about to hand serious work to something non-deterministic.

It was also a ceiling. A model made to walk plan, then execute, then verify will walk them in that order even when the evidence in front of it argues for going back to the plan, or for verifying one thing before executing the next. Those stages were doing what a nervous author wanted, and I was the nervous author.

So they came out. What replaced them is closer to ReAct with standards bolted on: the model interleaves however it likes, and the prompt constrains what counts as acceptable work rather than the order the work happens in.

Worth noticing, because it complicates the argument: those stages ported to the new domain perfectly well. They were domain-neutral. Portability and quality are two different questions, and a prompt can be portable and still be holding the model back.

Evaluation, or the lack of it

The first version’s whole output contract was one page of markdown, the way a good analyst hands over a one-pager. Because the deliverable was a document rather than a label, there was nothing to score it against. There is no ground truth for “was this a good investigation”; two competent analysts write different one-pagers about the same problem and both can be right.

So the method was crude. Every run asked whoever requested it whether the output was useful, and roughly 98% came back approved.

That number is hard to read, and I’ve stopped pretending otherwise. Maybe the agent was excellent. Maybe the bar was soft. The most defensible reading is that it measured legibility rather than correctness: a one-pager is compact enough that someone who owns the problem can tell inside ninety seconds whether the thing understood what they asked. Approval on that says a great deal about relevance and readability, and very little about whether the findings were true.

It held up exactly as long as the agent stayed advisory. The day people started acting directly on the output, a thumbs-up per run stopped being enough, and closing that gap has been most of the work since. Starting crude still looks right, though. Building a formal eval harness first would have meant a quarter spent measuring something nobody had confirmed they wanted.

The transferable bit

Write down the standards, not the steps.

The standards are the part you know and the model doesn’t: what counts as evidence here, when to distrust your own data, what you are never allowed to guess, what to do when the honest answer is “I can’t tell.” Those survive the next domain, the next team and the next model release.

The steps are the part the model can work out for itself, and every one you pin down is a decision taken away from something that may well reason better than you about that specific case.

Nine-tenths of that prompt was standards, and that’s why it moved in two days.

One last thing: none of this would have worked on a weaker model. That first version shipped on Claude Opus 4.6, and the bet only pays off if the thing on the other side can actually be trusted with the method. It could. Thanks are due for that.