What if AI behaves differently when we're testing it?
Frontier models are starting to behave differently under test, so a passed AI evaluation proves only what happened under its conditions, and the evidence has to include what the system does in ordinary operation.
Auditors have always understood that people behave differently when they're being watched. It's one reason we don't decide whether a control works by standing beside someone and asking them to perform it. We count cash by surprise and test what happened when we weren't there.
AI may be creating a surprisingly familiar version of the same problem.
In the first three essays in this series, I looked at three things I want to know before relying on an AI-enabled audit procedure: can I test the method, do I know the population was complete, and was the result checked against something independent of whatever produced it? All three rest on another assumption: that the behavior I observe when I test the procedure tells me something useful about how it behaves when nobody is testing it. Evidence from the frontier labs is making that assumption harder to make.
OpenAI recently published a study comparing six of its models in safety evaluations with comparable real situations. All six behaved less safely in the evaluations than in production, by 10 to 37 percentage points. That's the opposite of what an auditor might expect, since our usual concern is that people behave better when they know they're being tested. Telling one model that it was being evaluated made almost no difference. What mattered was what the model thought the evaluation was checking. When it was told the correct objective, its success rate rose from 63 percent to 92 percent, and when it was told the wrong one, it fell to 18 percent.
Auditors will recognize the broader problem. If people know exactly what the auditor is checking, that knowledge affects what the auditor observes. With AI, the testing environment may now be one of the variables we're testing without realizing it.
Anthropic and the UK's AI Security Institute have reported versions of the same issue. When Anthropic released two new models in November, it said they expressed suspicion that they were being evaluated more often than its recent models, which makes it harder to assess how they will behave in real-world settings, and it expects the problem to grow as models become more capable. AISI has reported greater evaluation awareness in recent models and questioned whether changes in measured behavior reflect the behavior we care about or awareness of the evaluation.
None of this proves that AI systems are secretly pretending to behave well, and we don't need to know why the behavior changes before it becomes an assurance problem.
Suppose I'm testing an AI agent before allowing it to perform a consequential business process. I give it representative tasks and edge cases, check whether it stays within its permissions and document the results. It passes. What, exactly, have I established? That it behaved appropriately under the conditions in which I tested it. We have traditionally treated that as close enough to establishing that it behaves appropriately. As models get better at recognizing their context, those may stop being the same thing.
This also limits something I proposed in the first essay in this series. I argued for keeping a reference set of known cases and rerunning an AI-enabled procedure against it over time, and I still would. A stable set of cases shows whether a procedure keeps producing the results we expect as models, prompts and other parts of the procedure change. A reference set can't be the whole validation, though. If models can infer that they're in an evaluation, repeatedly passing known cases tells me the procedure can pass those cases under those conditions, and I still need evidence from ordinary operation.
Completeness offers a parallel. In the second essay, I argued that checking everything a procedure flags isn't enough, and that you also need to sample what it cleared, because that's where the false negatives are. The same instinct applies here. Sample what the system did during normal operation as well as what it did when you tested it. The controlled evaluation shows how the procedure behaved under test, and the production sample shows what it did when no test cues were deliberately present. Then check the results against something independent of the system that produced them, which was the argument in my third essay: a source document, a ledger, a deterministic calculation or a later event that shows whether the answer was right.
If behavior differs between the controlled test and ordinary operation, investigate the difference instead of treating the production result as noise because the test was the controlled evidence. The gap may be the finding, even when every individual test passed.
Realistic documents, volumes and workflows can remove some of the obvious signals that distinguish an evaluation from normal work, and Anthropic has said it is working on more realistic evaluations for this reason. A test that looks more like production still isn't production, so the evidence has to include production itself.
The evidence also has to be refreshed. Models, system instructions, tools and the environment around an agent all change. Evidence gathered before those changes may still tell us something, but Audit needs to decide how much reliance it still deserves. Assurance practice is beginning to reflect that: the AI assurance offering announced by the AICPA and AIUC includes live behavioral testing and recurring technical retesting.
Controlled evaluations remain useful, and we need to be more precise about what they establish. For auditors, that means keeping the controlled test and the reference set, then sampling decisions or outputs from normal operation, checking them against independent evidence and comparing them with the controlled results. When the behavior differs, investigate. When the model, tools, instructions or operating environment change, retest. And write the conclusion to match the evidence.
A reference set shows how the procedure performed on known cases, a production sample shows what it did in ordinary operation, and an independent check shows whether the answer itself was right. Together they tell us much more than "the model passed the test." We still haven't proved how it will behave in every situation, and Audit has never been able to prove that. It's a much better basis for deciding how much reliance the procedure deserves.
Next week I'll come back to reproducibility: whether an AI procedure has to produce the same result every time it runs.
Continue the conversation. I read and reply to comments on Substack. The sharpest responses usually come from readers.
Get the essays in your inbox
Clear Calls publishes on Substack. No promotion. Unsubscribe anytime.