What counts as evidence when AI did the work?
When AI reasons across live data, there is no sample to keep. The evidence has to come from the method.
When we test a sample, we keep the sample, and that is what has made audit work repeatable. Anyone can go back to the same items and reach the same conclusion. The IIA's Global Internal Audit Standards are built on that. Documentation has to be good enough that an informed, prudent internal auditor “could repeat the work and derive the same engagement results.”
Full-population testing and AI-assisted analysis break that model. A tool that reasons across every transaction in a live system is not working from a fixed set of items. The data keeps moving, so the same method run next month can reach a different conclusion, and it should, because the population has changed. To repeat the work exactly, you would need a copy of everything the tool looked at. For most organizations that means duplicating vast amounts of data, much of it personal or confidential, and keeping it for years. That is expensive and cuts against the data-minimization principles in privacy law. I also don't think it is what the Standard is really trying to protect.
What the Standard protects is confidence that a conclusion followed from sound work. I think we should consider getting there by preserving and testing the method rather than the data. The profession has accepted this logic before. Under the benchmarking provisions of the PCAOB's internal control standard, an auditor who has tested an automated control can rely on it in later years without repeating the test, as long as the control hasn't changed and the controls over changes to it are still working. Nobody re-runs last year's transactions to prove the system still calculates correctly. They confirm the logic was tested and that nobody has altered it since.
External audit already asks for something closer to this. The PCAOB's documentation standard requires that an experienced auditor with no previous connection to the engagement be able to understand “the nature, timing, extent, and results of the procedures performed, evidence obtained, and conclusions reached.” Understanding the work is a different test from repeating it, and for AI-assisted procedures it is the more workable one.
Preserving the method doesn't mean keeping nothing about the data. A reviewer still needs to know what the procedure ran on, and that can be recorded without copying the population: when the data was extracted and from which systems, and control totals that tie back to the source. The exceptions are different. The items the procedure flagged, and the evidence behind any finding, should be kept in full, because those are what a conclusion rests on and what someone may challenge later.
The method itself needs the most care. If the criteria for an exception live in a set of instructions, those instructions are the test design, and they deserve the same retention and review as any other audit program. So do the configuration of the tool and the version of the model that ran. Small changes in wording can change which items get flagged, and a reviewer looking only at the output has no way to know which version of the procedure produced it. COSO's guidance on internal control over generative AI, published in February, points the same way for management's own controls, calling for logs of model versions and prompts and for change control over prompts and configuration.
Then there is how anyone knows the method works. Before relying on an AI-assisted test, someone needs to have run it against cases where the answer is already known and compared the results with work done another way. This is also where repeatability can survive. A reference set of known cases, kept and reused, does what the sample used to do: anyone can re-run the method against it later and see whether it still gets the same answers. If it does, and nothing in the method has changed, a reviewer has good reason to trust what it found in data that no longer exists in the same form. A probabilistic tool may not word its output the same way twice, and I will come back to what that means for re-performance in a later essay, but its results on the reference set should hold.
After that comes what happened to the exceptions. AI-assisted procedures can return far more items than a sample ever did, and investigating them is where much of the real audit work now sits. The PCAOB's 2024 amendments on technology-assisted analysis set out what an auditor should consider when investigating items identified this way. I would look at every item the procedure flags, and at a sample of the items it passes. Most attention goes to the flags, but the passed items are where you find out what the tool is missing. The sample is small enough to keep along with the flags, so a reviewer later has real items to look at even though the full population is gone. The file should show how each exception was resolved and on what evidence.
Doing this every time the procedure runs keeps people looking at actual transactions as well as the tool's summary of them. When the review turns up something the tool missed, or a run of false alarms, that is the signal to change the instructions or the thresholds and test the method again. In response to my last essay, Andrew Struthers-Kennedy suggested that some friction in how people work with AI may be a good thing, because it forces attention. Reviewing the flags and a sample of the passes builds in that kind of friction, and what people find goes straight back into improving the method. It should also show where the auditor disagreed with the tool and what changed as a result, because the output alone can't show how the conclusion was formed.
None of this applies with the same force everywhere. When AI only points an auditor toward an issue, the auditor can verify the issue with other evidence, and the main thing to record is what the tool couldn't see, so that an area it skipped isn't mistaken for an area found to be clean. When it supports a judgment, the reviewer needs enough to judge how much weight it adds. When it drives the conclusion, everything above is needed, and the method should be validated before anyone relies on it.
The risk I would watch most closely is a procedure moving between those roles without anyone deciding that it should. Suppose a team builds a tool to help plan where to focus testing. It works well. A year later, its output has become the main reason a control is rated effective, and nobody went back to ask whether the evidence behind the tool was strong enough for that use. The tool itself hasn't changed, only how much the conclusion depends on it, and the evidence should have kept up with that.
The standards haven't settled any of this yet. The PCAOB's 2024 amendments said plainly that they did not address artificial intelligence specifically, and when it asked for input on its standard-setting agenda in June 2026, commenters raised what counts as sufficient evidence for AI-assisted procedures and how they should be documented.
The IAASB's draft revision of its audit evidence standard, open for comment until December 15, gets a good deal of this right. It deliberately avoids rules for particular technologies. Its explanatory memorandum says a single workable model would be premature and notes that generative AI tools “challenge elements of the ‘input-output model,’” because it is not always possible to fully evaluate a tool's inputs or confirm that it operated as designed. Its guidance on technological tools also recognizes that some information changes over time, and that running a procedure at an earlier or later date “may render a different outcome.” I agree with all of that.
Where I would push is on what follows from it. If an auditor can't always evaluate a tool's inputs or see how it operated, the evidence has to come from somewhere else, and the draft's guidance doesn't yet say where. I would like it to. A tool can be validated against cases where the answer is already known, and changes to it can be controlled and recorded the way changes to an automated control are. When the full population can't be kept, the file can still record what the procedure ran on and keep the exceptions and a sample of what it passed. That is the feedback I would want the IAASB to hear before its comment period closes.
In my own function, we built verification into how the work flows: what gets checked, and against what evidence, before a finding turns into a decision. I think every function using these tools will have to make the same kinds of choices, probably before the standards catch up.
Standard 14.6 asks the right question, which is whether another professional could rely on the work. For a sample, keeping the data was the way to answer it. For full-population and AI-assisted work, the answer has to come from the method. If another professional can re-run it on known cases and get the same results, and can see exactly what it ran on, they can rely on the conclusion without a copy of the data. That is the change I think the Standards should consider.
Continue the conversation. I read and reply to comments on Substack. The sharpest responses usually come from readers.
Get the essays in your inbox
Clear Calls publishes on Substack. No promotion. Unsubscribe anytime.