When AI Meets Audit · October 2026

Assurance over 100% of what?

Testing every record shows only that a procedure ran on the records it received, so the conclusion needs evidence that the population was complete and a check on what the procedure cleared.

When I took over as CFO of several companies that had been acquired, the finance team showed me a polished report of results. When I asked about the actuals, I got blank looks: they had built their report from the forecast. The team received an automated report of actual results every month, but because they did not recognize the numbers, they ignored it. We had to map where every number came from, which systems fed the report and how each field mapped, before we could get to the truth. I have seen smaller versions of the same problem during ERP changes, when a report suddenly looks slightly off. Much of the judgment a finance leader builds is noticing when something does not fit and knowing when to dig in.

AI raises the stakes on that skill. We went from data puddles to data lakes in the cloud, and now we are in the ocean. In the middle of the Pacific you need a feel for the currents, and you still need to be able to dive down to a specific problem when something looks wrong.

In audit, that shift shows up in how we test. When audits relied on samples, the limitation of a procedure was written into the procedure. You tested a sample, and everyone understood that you had not looked at the rest. Sampling risk was measured and stated in the conclusion, so anyone reviewing the work could see how much of the population had been examined and how much of the conclusion was inference.

Full-population testing removes that limitation, and it is one of the real gains of the last few years. It can also create the false comfort I saw in that finance team: a belief that everything has been evaluated when nobody has checked where the data came from. When a procedure runs on every record, it is easy to say we tested everything, but that means only that the records received were tested. That statement does not tell us whether the procedure received the right records, or whether it would have recognized everything in them that mattered.

A finance person's instinct with any number is to ask whether it makes sense and whether it fits the pattern of everything else known about the business. That check depends on having the right population. A listing of vendor payments is useful only if it agrees with the ledger, includes every required entity and day, and excludes records that do not belong. Auditors have long applied the same discipline to information produced by the company. The PCAOB's audit evidence standard, AS 1105, requires the auditor to test the accuracy and completeness of that information, or the controls over it, before relying on it.

What has changed is where the population comes from and how it fails. In a traditional audit, the population was often a report someone ran and handed over, and you could count it and tie it out. In an AI-enabled procedure, the population more often comes through a live feed that joins several systems and applies filters someone wrote months ago. A connection can time out and return only part of the data without raising an error. A join can drop records when an identifier or date is formatted differently. A faulty filter can exclude legitimate transactions or include records that do not belong. Missing records leave gaps, and extra records dilute the exception rate so the result looks stronger than it is. In every case, the procedure can report truthfully that it reviewed every record it received. The UK's Financial Conduct Authority found a version of the missing-record problem at Metro Bank, which it fined in 2024. An error in how data was extracted for the bank's transaction monitoring system kept accounts opened and used on the same day out of the system. Between 2016 and 2020, that error and related failings left more than 60 million transactions unmonitored, and until December 2020 the bank had no routine way to check that every record sent to the system had arrived.

Suppose part of a company moves to a new ERP system. A payments procedure built before the migration continues to run weekly across every record in its feed, but the feed still reads from the old system. The migrated business shows no exceptions, and on a dashboard that looks exactly like a clean result. Years ago, a fraud case I remember came down to one missing line on a spreadsheet that was otherwise accurate. A missing feed is the same kind of problem at a much larger scale.

A full-population procedure also finds only what it was built to find. Anything it flags gets reviewed, while what it clears goes out with millions of legitimate records and nobody looks at it again. With a rule, you can usually see what it will miss. A duplicate-invoice check may not catch a number with an added suffix. With a model, it can be harder to know what will slip through, especially when the exception is unlike anything the model has seen before. Sampling had one modest advantage here: an experienced auditor working through a sample would sometimes notice something that was not on the test plan.

The standard setters have started to look at what a procedure clears. When the PCAOB adopted amendments in 2024 on audit procedures that involve technology-assisted analysis, some commenters asked whether a full-population analysis that identified no items for investigation would be sufficient evidence on its own, and one suggested the amendments address the items not identified. The final requirements, which took effect for audits of fiscal years beginning on or after December 15, 2025, are principles-based and center on how the auditor investigates the items a procedure does identify. That is where the profession is: we have much clearer expectations for what a procedure flags than for what it clears. The IAASB's proposed revision of its audit evidence standard, out for comment until December 15, 2026, is explicit that when a tool scans an entire population to pick out unusual items, testing those items gives evidence about those items, and the misstatements found in them cannot be projected to the rest of the population.

We need to treat the population itself as evidence. Each time a procedure runs, compare what it received with record counts from the source systems and totals that agree to the ledger. The comparison should identify both missing and extra records. If the procedure runs continuously, this check has to run with it, so a sudden change in volume from one business unit does not pass unnoticed. Metro Bank's eventual fix was simple: a daily count of the records sent to the monitoring system, compared with the records it accepted or rejected, and an alert whenever the totals did not match.

Sampling also comes back, in a different role. A sample drawn from the records the procedure cleared shows whether it missed problems among the records it received. If a separate method or an experienced reviewer finds something the procedure should have flagged, that tells us what reviewing the flagged items alone cannot. Sampling cannot find records the procedure never received or determine by itself whether irrelevant records were included. The population check and the sample therefore answer different questions, and a sound conclusion needs both.

When a procedure runs across ten million records and finds nothing, first ask whether the population was the right one. If it finds four exceptions, look hardest at any that do not fit a category the procedure was built to recognize.

We should also decide in advance what kinds of failure could change the conclusion. If the conclusion is that no payments went to sanctioned parties, the failures that matter are specific, such as a payment system that is not connected or name matching that misses transliterated spellings. Each of those can be tested directly, and a reviewer then has something concrete to challenge.

Full-population testing is still valuable, but testing every record is not enough. The conclusion should explain how we know no required records were missing, no irrelevant records were included, and whether the procedure missed problems in the records it cleared.

Continue the conversation. I read and reply to comments on Substack. The sharpest responses usually come from readers.

Julia Denman is Corporate Vice President and Chief Risk and Audit Officer at Microsoft, a former CFO, and a director of The Clorox Company. Her book, The Clarity Quotient, publishes early 2027 with Fast Company Press.

Take the five-minute self-assessment →

Get the essays in your inbox

Clear Calls publishes on Substack. No promotion. Unsubscribe anytime.

Subscribe on Substack →

All essays Clarity Review About Julia