Pattern Chaser: Audit Analytics for Internal Audit

PATTERN CHASER

Audit Analytics for Internal Audit in Financial Services

Unusual Is Not the Same as Dangerous

Your anomaly detection is very good at finding what is unusual. That is not the same as finding what matters.

Hey Reader 👋

A monitoring system can be switched on, visibly active and firing alerts every day. And still be blind to almost everything that matters.

Busy is not the same as watching. A full alert queue is not the same as coverage. One feels like control. The other is control.

This is the trap in most audit anomaly detection. It is very good at finding what is unusual. It is poor at finding what matters.

An outlier is a maths problem. A material anomaly is a risk problem. Most functions are solving the first and reporting it as the second.

In today’s issue:

↳

Why a flood of exceptions hides the one that matters

↳

The metric that should replace coverage

↳

A triage layer you can put between the model and the queue

Named enforcement examples are drawn from public regulatory notices and court findings. Other examples are composites from industry experience and peer conversations.

DEEP DIVE

The Alerts Were Firing. The Bank Was Blind.

One bank showed exactly how wide that gap can get.

Between 2018 and 2024, TD Bank’s automated monitoring failed to cover 92 per cent of transaction volume. The bank knew about the gap. It did not fix it.

The networks moving illicit money did not need to be clever. They needed the part of the flow the system never scored. More than 670 million dollars moved through TD Bank accounts across three money laundering networks.

In October 2024 the bank pleaded guilty and agreed to pay roughly 3.09 billion dollars across coordinated resolutions with the Justice Department, FinCEN, the OCC and the Federal Reserve. The FinCEN penalty alone was the largest the US Treasury’s financial crime unit has ever placed on a bank. But the penalty is not the lesson. The lesson is that activity is not coverage. A system can be live, visibly active and firing alerts, and still leave most of the risk outside its field of view.

TD is the simplest version of the problem. The risk was never in scope, so nothing ever scored it. No anomaly detector flags what it never sees.

But scope is only half the trouble. Even when the data does reach the model, finding what is unusual and finding what is dangerous are two different jobs. Most systems are built for the first and then graded on the second.

Two definitions, no relationship

A statistical outlier is defined against a distribution. It is a point that sits far from the rest of the data. A material anomaly is defined against a risk objective. It is an event that signals fraud, control failure or regulatory breach with consequence above a threshold that matters.

These two definitions share no necessary relationship. A transaction can be perfectly legal and statistically extreme. A genuine year-end payment will trip every threshold you set and mean nothing.

The reverse is worse. Sophisticated fraud is built to sit inside the normal distribution. Structuring, layering, splitting across accounts. Every technique exists to keep each individual transaction unremarkable. The scheme is invisible precisely because none of its parts are outliers.

A 2021 review of sixteen studies on anomaly detection in auditing reached a blunt conclusion. On its own, anomaly detection returns too many results of no interest to auditors. It raises the cost of the audit without a matching rise in risk caught. Ranking the anomalies helps, but the most anomalous item can still be a false positive. The algorithm finds the unusual. It cannot tell you what is dangerous.

Why More Anomalies Is Not More Coverage

The instinct is that flagging everything unusual means you cannot miss anything material. That instinct fails on three fronts.

Failure 1

Alert Fatigue Amplifies Risk

In rule-based monitoring, false positive rates are often estimated at 90 to 95 per cent. When most of what lands is noise, every genuine signal competes with thousands of non-events for the same scarce review hours. High volume does not raise scrutiny. It dilutes it.

Failure 2

The Credibility Tax

A long list of anomalies with no risk ranking, no ownership and no decision logic has a name. Exception dumping. Raise 400 explainable exceptions and the business learns one lesson: the model cries wolf. When the one that matters finally lands, it gets the same shrug as the noise.

Failure 3

The Base Rate Problem

In many monitoring populations, genuine fraud or material control failure is rare. When the true rate is that low, even a highly accurate detector produces far more false positives than true ones. Without a risk-weighted prior, the maths guarantees you drown. No amount of algorithmic sophistication fixes a missing prior.

So the question is not how many anomalies you flagged. It is how many of them became a finding that changed something. Call it the alert-to-finding conversion rate. A rate in the low single digits is not rigour. It is indiscipline at scale.

And coverage is the wrong comfort too. A model can score every record it is given and still see almost nothing, if most of the risk sits in a population no one fed it. One hundred per cent coverage is a processing statistic. It is not assurance.

The regulators already say this out loud. The FCA’s January 2025 review of monitoring in wholesale markets found that automated systems used in isolation provide limited success. They have to be combined with customer risk, know-your-customer data and front and middle-office input. The Basel guidance goes further and frames eliminating false positives as a supervisory expectation, not an efficiency nicety. The standard is not more alerts. It is better ones.

WHAT’S POSSIBLE

The Risk-Ranked Exception Queue

Most exception queues are ordered by when the alert fired or how far a value sits from the mean. Neither has anything to do with risk.

Now picture the queue ordered differently. Each exception scored before a human ever sees it. Financial value. How often the pattern recurs. The strength of the control wrapped around it. A prior weight for the fraud typology it resembles. The investigation list arrives sorted by expected materiality, not statistical distance.

Card fraud teams have worked this way for years. Every transaction carries a risk score and the queue is worked in that order. Audit analytics still flags a pile of exceptions and works them roughly first in, first out.

None of this needs a new model. It needs a scoring layer sitting between detection and the queue. That layer is the part most functions never build, because the model felt like the hard bit and the queue felt like admin.

The team still investigates 30 items this week. The difference is they are the right 30.

You did not train as an auditor to clear a queue.

You trained to find the thing hiding inside what looks normal. The outlier is easy. The system was built to surface it.

The anomaly that matters is the one nobody flagged. Go and find it.

What is your alert-to-finding conversion rate right now? Hit reply, I read every one.

WHAT’S ON MY RADAR

Worth your time this week.

🔍 THE PATTERN

Alarm Fatigue in Intensive Care: A Systematic Review. Hospitals learned the hard way that when almost every alarm is a false alarm, staff stop hearing the real ones. Your exception queue is the same machine with a different label.

🧠 THE MIND

The Base Rate Fallacy. The cleanest explanation of why a rare event hidden in a huge population produces mostly false positives, no matter how good your detector is. The maths every alerting system runs into.

📊 THE PROOF

Alert Fatigue in Security Operations Centres. Security operations are the closest mirror to audit analytics, and the research shows alerts fired and then ignored in most breaches because analysts were buried. The price of a queue with no triage.

🗞️ WHERE IT’S HEADING

AI in Financial Services Risks Serious Harm Without Action. The UK Treasury Committee is now warning Parliament on AI risk across financial services. A clear signal that how your function governs and explains its models is about to face real scrutiny.

THAT’S A WRAP