Pattern Chaser: Audit Analytics for Internal Audit

PATTERN CHASER

Audit Analytics for Internal Audit in Financial Services

Your test is 95% accurate

And still 98% wrong. Why your exception report is mostly noise, and the one move that fixes it.

Hey Reader 👋

Right now, somewhere, an auditor is opening a file they didn’t ask for.

Around fifty thousand transactions in it, every one flagged by the exception testing the firm just paid good money for. Three people to work through them. It’s due Friday.

The tool worked exactly as designed. That’s the problem.

Because almost none of them are real. Not because the test is broken. Because when you hunt something rare, even a good test buries it in false alarms. Almost nobody is taught that.

In today’s issue:

↳

Why a good test on a rare thing is mostly noise, and the maths nobody teaches

↳

A duplicate-payment hunt built in the open: what fails, and the one clue that fixes it

↳

Why the answer is to rank the list, not clear it

THE BIG IDEA

Why a good test drowns you

Start here. Almost every payment a company makes is fine. The duplicate, the fraud, the error you’re hunting is a needle in a very large haystack. Rare, by definition. If it were common you wouldn’t need audit to find it.

Now run a test across that haystack. Say it catches 95% of the real ones and correctly clears 95% of the clean. That sounds like a good test. Run it across a million payments and the report comes back with around fifty thousand alerts. Fewer than a thousand are real.

You’ve built a car alarm. It goes off in the car park all day, and everyone learns to walk past it.

That’s the trap, and the tool didn’t cause it. When the real thing is rare, the false alarms bury it, however clever the test. The accuracy score on the box never warns you, because it cannot see how rare your target is.

And it’s worse in real life, because you rarely know how rare up front. Nobody hands you the true number of frauds in a ledger. You can estimate it, but not for free, and until you do you cannot even tell how bad your list is. You just get a long one, and the quiet hope that long means thorough. It usually means the opposite.

UNDER THE HOOD

So I went looking, and let you watch

That was the arithmetic. To see it happen, I built a payments ledger from scratch and hid 120 duplicate payments inside it. Each one is a real payment, paid a second time, keyed a little differently: a vendor spelt two ways, a dropped “Ltd”, an invoice number fat-fingered. I know where all 120 are, because I put them there, so I can score every method honestly.

Then I hunted them the way most teams do. Exact matches first. Same vendor, same amount, days apart. It found about a quarter. The rest slipped by, because someone typed “Northwind Ltd” one week and “Northwind Limited” the next, and an exact match can’t see past a typo.

So you loosen the rules. Close-enough names, roughly the same amount, near enough in time. You catch more of them. And the list explodes.

Almost everything it now flags is innocent. Fewer than three alerts in a hundred are real. Instalments, rent, subscriptions: same vendor, same amount, days apart and completely fine. To a test that only knows vendor, amount and date, a duplicate and a rent payment are twins.

Loosening caught a few more and buried them in noise. Tightening just went back to missing the real ones. The rules themselves were too blunt.

What fixed it wasn’t a cleverer rule. It was a better clue. A real duplicate is the same invoice paid twice. An instalment is different invoices for one bill. So the invoice number is the clue that tells them apart, and the first test never looked at it. Add it to the score.

# naive: vendor + amount + date

                  score_naive <- 0.55*vendor_sim + 0.30*amount_sim + 0.15*date_prox
                  # tuned: one more signal, the one that splits a duplicate from an instalment

                  score_tuned <- 0.45*invoice_sim + 0.25*vendor_sim + 0.20*amount_sim + 0.10*date_prox

Add that one clue and the top of the list flips from almost all noise to almost all real.

The false-positive cliff. The naive matcher (red) stays near zero precision at every setting; adding the invoice-number clue (gold) holds precision near 100 per cent until the match is forced loose.

Precision at every match setting. The naive matcher (red) never lifts off the floor. Add the invoice-number clue (gold) and it holds near 100% until you force the match loose.

Fair warning, since you’d ask it of anyone else. I made my duplicates share an invoice number, because that is what a real duplicate is, so the clue that works here is the one that works in a real ledger. Live invoice numbers are messier, reused and truncated and mistyped, so you’d match those loosely too.

Then stop trying to clear the list

Stop asking for every duplicate. You will never clear fifty thousand lines and you shouldn’t try.

Rank every candidate by that combined score, most likely first, and work down. In my ledger the top fifty were every one a real duplicate. Take the review budget your team actually has, say the top few hundred, and read down until the hits fall away.

The haystack: every candidate pair by score. The crowd sits in grey, the real duplicates glow gold on the right, and a cyan line marks the top of the list the team reviews.

Every candidate pair by score. The real ones glow gold. You review the top, not the lot.

Notice what you never needed: the total number of duplicates in the ledger. You worked from the top and stopped when it stopped paying. That is how you get on with the job without the number nobody can give you.

One honest caveat, because a good auditor will raise it. Ranking is triage, not assurance. It tells you where to spend your team’s hours, not what you missed. If you need the total error rate, that is a separate job with a separate tool: take a fresh random sample of the whole ledger, work it properly and let it tell you the rate you couldn’t see. Two different questions. Answer them separately and don’t let anyone pretend one is the other.

WHEREVER YOU SIT

The same idea lands differently depending on your chair. Here’s yours.

If the numbers scare you

The whole thing sits on one sentence. The rarer what you’re hunting, the less a single alert is worth. Hold that and you’ll see through an exception list faster than people twice your grade. You never needed to code to do it.

If you build the tests

You know the thresholds cold. The catch is the threshold was never the fix. When a test drowns you, you’re usually short a signal, not a setting. Go find the one field that splits the real thing from its double. Mine was the invoice number.

If you’re clearing the list

You were never meant to clear fifty thousand lines. Rank them, take the top few hundred, stop when the hits fall away. That isn’t cutting corners. It’s the only honest way to spend the hours you actually have.

If it’s your name on it

Next time a vendor or a keen analyst brings you a smarter model or a longer report, ask two questions. How rare is what we’re hunting? And how far down this list can my team really reach? Anything that can’t answer both is a louder alarm dressed up as assurance.

An exception list was never assurance. It is a pile of maybes.

The job was never making the list. It was knowing which few to trust and being straight about the rest.

Less talk about the work. More of the work.

Every number here came from one short script. It’s yours to run, and the ledger lands in your inbox soon so you can hunt it yourself.

Before you go

Where does your team’s judgement actually sit?

Ranking the list is one skill. The Future-Ready Auditor Scorecard shows you which of the five that carry an analytics function you should build first. Two minutes, five dimensions, no audience.

Get your score

WHAT’S ON MY RADAR

Worth your time this week.

🔍 The Pattern

The ’99 Percent Accuracy’ Paradox. A test can be 99% accurate and still be wrong almost every time it fires, because the thing it hunts is rare. The exact trap this issue is about, named in the mainstream press three months ago.

🧠 The Mind

Base Rate Fallacy. The name for a familiar failure: a sharp team stares at a 95%-accurate test and still trusts the alert, because we fixate on the accuracy on the box and ignore how rare the target is.

📊 The Proof

Alarm Fatigue (Making Healthcare Safer III). Hospitals found between 72% and 99% of clinical alarms are false, so staff learn to walk past them. The same false-alarm collapse this issue describes, measured in a field where the ignored alert can kill someone.

🗞️ Where It’s Heading

Moving Beyond Automated Transaction Monitoring. The banks’ own standard-setter now argues what you just read. Stop scoring monitoring by alert volume, move to a risk-based approach and chase useful outcomes, not longer lists.

Know a colleague drowning in an exception report? Send it on.