Pattern Chaser: Audit Analytics for Internal Audit

PATTERN CHASER

Audit Analytics for Internal Audit in Financial Services

Your AI Problem Isn’t the Model

The data underneath is where most audit analytics fail. Fix the pipes before you buy the machine.

Hey Reader 👋

Half term is over. The house is quiet again. Perfect time to write about the thing everyone’s ignoring.

Everyone wants to talk about AI models.

Almost nobody wants to talk about the extract feeding them.

That is where I have seen audit analytics fail most often. Not in the algorithm. Not in the dashboard. In the quiet handoff between source system, staging table and whatever tool the team has learned to trust.

The dangerous part is that bad data does not look broken. The dashboard still refreshes. Alerts still fire. Someone still takes comfort from the output.

Then months later you find the system was only seeing part of reality.

In today’s issue:

↳

Why data quality failures cost more than AI investments

↳

The four failure patterns that keep repeating

↳

A pre-model checklist you can run this week

Named enforcement examples are drawn from public regulatory notices. Other examples are composites from industry experience and peer conversations.

DEEP DIVE

What 60 Million Missing Transactions Looks Like

Metro Bank had an automated transaction monitoring system. It ran every day. It produced alerts. People investigated those alerts.

The problem was what it never saw.

The system rejected records it classified as “bad data.” No effective completeness reconciliation. Inadequate exception handling for the records that mattered. For years, rejected records sat in Bad Data folders. Staff raised concerns. The control environment did not respond.

Over 60 million transactions. £51 billion in value. Never monitored.

The FCA fined them £16.7 million. But the fine is not the point. The point is that everyone thought the system was working. Alerts were firing. Dashboards were green. The comfort was real. The coverage was not.

This is not isolated. NatWest had cash deposits misclassified as cheques, which meant lower risk scores and less scrutiny. One customer deposited around £264 million in cash despite multiple red flags, internal suspicion reports and automated alerts. HSBC had incomplete data feeding key systems, with transactions “in the millions” monitored incorrectly or not at all. The ECB reviewed 25 significant institutions for data quality. Zero had fully implemented the expected standards.

The pattern is the same every time: the system runs, partial outputs create false comfort, and nobody asks what the system is not seeing.

The Four Failure Modes I Keep Seeing

When I look at where data quality breaks audit analytics, it is almost always one of these.

Pattern 1

Ingestion Failure

Complete source data never reaches the model. Records are dropped, rejected or filtered out before the analytics layer ever sees them. The system runs. It just runs on an incomplete picture.

Pattern 2

Broken Reconciliation

Rejected records are known but not governed. Someone somewhere knows they exist. No one owns fixing them. No one tracks how many. The gap compounds quietly over months.

Pattern 3

Wrong Labels

Customer risk ratings, transaction types, product classifications. If the label is wrong at source, everything downstream inherits the error. NatWest’s cash-as-cheques problem was a label problem.

Pattern 4

False Comfort from Partial Outputs

Some alerts fire. Some dashboards update. Stakeholders assume coverage is intact because something is working. This is the most dangerous pattern. Partial functionality creates the illusion of control.

These were not just model failures. They were data quality, rule design, reconciliation and governance failures. The algorithm logic was weak in some cases. The inputs were incomplete or mislabelled in others. Either way, the output created more comfort than the control deserved.

WHAT’S POSSIBLE

The Pre-Model Data Check

Before I trust any model, alert rule or full-population test, I run these five checks. They take an hour. They tell you whether your data is fit for purpose before you waste a week building something on top of it.

Check

What You Are Looking For

Completeness

Compare record counts at source vs staging vs model input. If numbers do not match, find out where records are dropping.

Missingness

Profile null rates by field and by segment. Concentrated missingness usually means a source problem, not random noise.

Duplicates

Check for duplicate business keys. Duplicates inflate metrics, distort model probabilities and create false positives.

Labels

Sample classification fields and trace back to source. If the label is wrong at source, everything downstream inherits the error.

Rejects

Ask: where do rejected records go? Who owns them? How many are there? If no one can answer, you have found your first audit issue.

This is not exploratory analysis. It is a control discipline. If your team cannot answer these five questions before running the model, the model output is not evidence. It is a guess dressed up as analytics.

The audit functions that will matter in three years are not the ones buying the best AI.

They are the ones who know what their data actually looks like before they feed it to anything.

Fix the pipes before you buy the machine.

WHAT’S ON MY RADAR

Worth your time this week.

🎯 PATTERN

AI models are choking on junk data. Fortune on why the quest for more training data has created a glut of low-quality inputs that could derail physical AI. The problem this newsletter is about, in mainstream business press.

📚 MIND

Reasoning models don’t always say what they think. Anthropic tested whether AI “chain of thought” reflects actual reasoning. Claude mentioned the real hint only 25% of the time. The explanation you see may not be the explanation underneath.

⚡ TOOL

A single real-world data point may stop AI model collapse. Counterintuitive research finding: one human data point can anchor an entire synthetic training set. Practical implication for anyone building with AI.

🎲 WILDCARD

How OpenAI monitors coding agents for misalignment. Their agent quietly deleted code that checked its own work. Another extracted encrypted credentials from the macOS keychain without being asked. The kind of thing that makes you stop scrolling.

THAT’S A WRAP