Verification Requires Domain Knowledgeauto-ingested

You can only catch an AI's mistakes in areas you already know. So the work you delegate because you lack the expertise is exactly the work whose errors you are least able to see — and those errors cost the most (Anthropic — Investigating Vault Particles with Claude).

The observation

Anthropic's biology team took on a project outside their expertise ("none of us had worked with a protein assembly this large," "limited experience in structural biology") precisely because Claude could cover the gap. Their own summary of how that went:

"In areas we knew well, we caught mistakes quickly, but they cost us time in areas we didn't."

The two documented failures fit the pattern. A protocol specified a growth medium additive that interferes with getting DNA into cells — a wet-lab detail an experienced molecular biologist would flag, and it cost two weeks. In cryo-EM processing, where the team was least experienced, Claude skipped a standard step and produced a map blended with its own mirror image — a classic artifact in that field, caught only after the fact. Their corrective was a knowledge-free heuristic precisely because they couldn't evaluate the improvisation on its merits: "Please let's do normal things unless they don't work."

Why this is a paradox, not just a caution

The case for delegating is strongest where your knowledge is weakest — that is where the AI adds the most. But [[Critical Thinking Is Domain-Bound|judgment runs on domain knowledge]], so that is also where your checking is worst. The checking capacity and the reason to delegate move in opposite directions. Three consequences:

• "Human in the loop" is not self-executing. Human-in-the-Loop by Exception and The Human Sandwich both reserve judgment for a human. This page says the reserved slice only works if that human holds the relevant knowledge. A loop staffed by someone who can't evaluate the output is ceremony, not oversight.
• The cheapest errors to catch are the ones you needed least help with. Net value from delegation is therefore highest where you are competent but slow, and most variable where you are genuinely ignorant.
• It is a mechanism for the erosion worry. Critical Thinking Is Domain-Bound asks whether offloading knowledge to AI thins out judgment. This is that question with a measured cost attached: not a loss of judgment later, but a loss of error-catching now.

What makes it survivable

The vault project still succeeded, and the team's verdict was that benefits "far outweighed the drawbacks." The conditions that seem to matter:

• Ground truth outside the model. Cryo-EM maps, gels and particle counts answer back. The mirror-image map was eventually caught because the physical world kept score. Work with no external check has no such backstop (AI Use Tracks AI Capability: capability tracks verifiability, r = 0.78).
• Conventional-first as a default. "Do normal things unless they don't work" is a cheap substitute for expertise: it constrains the model to the part of the space where published standards exist and errors are recognizable.
• Consultants for the knowledge you don't have. The team brought in Dr. Rome, the vault's co-discoverer, to advise on the manuscript.
• Feeding the errors back. Lab notes went to the researchers training Claude, so the errors became training signal rather than a recurring tax.

Where the knowledge requirement can be engineered down

Scope narrowing (auto-ingest 2026-11-08): Anthropic — Progress on Provable Inference for Large Language Models shows the exception. Anthropic's team says Claude "often makes decisions that are hard to evaluate, especially in large complex systems," the same problem the vault team hit. Their answer was not more expertise in the reviewer but a trusted verifier: Lean statements of the security properties plus a provably correct verifier, so "you can fully distrust the prover if you trust the verifier." The reviewer then has to audit only the statements ("dramatically less onerous than auditing proofs"), not the derivation. That still takes knowledge, so this page's claim holds, but the amount needed shrinks to "enough to read the spec." It works only where the claim can be formalized, which the vault lab's failures could not be. See Provable Inference.

A second, informal route (auto-ingest 2026-11-08): Anthropic — Teaching AI to Build Scientific Models asks AI model-builders to deliver not one model but "an account of which parts are pinned down by the data, which are still open, which alternatives failed." That is the knowledge Hodgkin and Huxley got by building the model themselves. Handing it over explicitly gives the reviewer something to check — did the simpler alternative really fail? — without rebuilding the model. It doesn't need formalization, only that the AI report its dead ends (Distributions over Best Fits).

Open questions

• Is the two-week loss an acceptable cost of the trade, or a sample of a tail that gets worse as projects get further from the operator's knowledge? One case can't say.
• Does "do normal things unless they don't work" generalize? It blocks exactly the kind of unconventional move that worked here (loading raw lysate onto the column). The same project shows both the value of overriding convention and the cost of letting the model do it unsupervised.
• Who holds the knowledge in a wiki-mediated workflow? See Method Reflection T3.

In cybernetic form

Ashby — An Introduction to Cybernetics states this page's claim as a law. "Only variety in R can force down the variety due to D; variety can destroy variety," and "R's capacity as a regulator cannot exceed R's capacity as a channel of communication." A reviewer is a regulator of the agent's output. Where the reviewer's own repertoire of possible readings (their variety) is smaller than the output's possible errors, the errors pass. That is why the vault team "caught mistakes quickly" in areas they knew and lost two weeks where they did not: not a lapse of attention but a bound on regulation. It also says what the engineered routes above do: a trusted verifier or a list of dead ends reduces the variety the human must match, rather than raising the human's.

Connections

• Communication and Control Are Intertwined — the cybernetic bound this page is an instance of.
• Critical Thinking Is Domain-Bound — the cognitive account this instantiates: checking is thinking, and thinking is bound to knowledge.
• AI Commoditizes Expertise — the counterweight: the team did do expert work without the expertise. The two pages together give the terms of the bargain.
• The Human Sandwich — names the judge-and-extend slice; this page names its precondition.
• Human-in-the-Loop by Exception — the craft-domain twin: a human on the exception only pays off if that human can adjudicate the exception.
• AI Use Tracks AI Capability — verifiability as the axis AI capability follows; here it is also the axis human oversight follows.
• Falsification Is Asymmetric — the chromatography episode: the model's confident objection was a hypothesis, and one experiment beat it.
• AI-Accelerated Science — the acceleration scenarios net out errors like these only implicitly.
• Method Reflection — T3: whether an Agent-Maintained Wiki builds the human's knowledge or substitutes for it.
• Provable Inference — the engineered exception: a trusted verifier plus audited statements shrink how much knowledge checking needs.
• Distributions over Best Fits — the informal route: make the AI report the alternatives it ruled out, so the reviewer checks the eliminations.

Sources

• Ashby — An Introduction to Cybernetics — the Law of Requisite Variety (§11/6–11/11).
• Anthropic — Investigating Vault Particles with Claude — the "areas we knew well" finding and both documented failures.
• Willingham — Critical Thinking, Why Is It So Hard to Teach — why checking can't be a content-free skill.
• Anthropic — Progress on Provable Inference for Large Language Models — the formal-verification route and the trust gaps it leaves.
• Anthropic — Teaching AI to Build Scientific Models — the "report what failed" route: understanding came from building the model, so an AI builder must hand over its dead ends.