You're reading Part 5 of Achieving Audit Leadership Through Analytics and AI, a six-part series from The Internal Audit Collective and Supervizor on how audit teams shift from delivering findings to owning business outcomes.
New to the series? Start with the earlier installments
Missed the beginning of the series? Catch up here:
-
Part 1: Going from zero to one: launching an audit analytics program that actually sticks
-
Part 2: From Point-in-Time Audits to Continuous Auditing: Choosing What to Run Repeatedly and Why
- Part 3: The Handoff: Moving from Continuous Auditing to Continuous Monitoring Owned by the Business
- Part 4: The Audit Report Is Not the Product Anymore
Tom O'Reilly of the Internal Audit Collective tells the story with the specificity of something that cost him.
It was 2014. He was CAE, newly armed with enterprise Qlik licenses and a mandate to modernize. He chose travel and expense (T&E) for the first analytics audit – a natural fit. His team of eight would build dashboards, surface spending patterns, and show the business something it couldn't see before.
That 400-hour audit became something closer to 2,000 hours.
The data queue at IT ran longer than expected. Once the data arrived, the scope had shifted. They brought in a consultant. They restarted twice. As they started building, they realized the data they'd asked for wasn't everything they needed – so they went back to request fields they hadn't known to ask for the first time. Getting a handle on the full data ecosystem took far longer than anyone had budgeted. The rest of the audit plan got reshuffled. Tom spent months apologizing to stakeholders and renegotiating commitments he'd made in good faith.
"After we were at like 2,000 hours in this audit," he says, "...I'm never using analytics again."
They finished it. The analytics worked – they even pushed the output to the business. But Qlik went unused after that. Other demands filled the calendar. The urgency that had launched the initiative quietly dissipated. More than a decade later, the memory is still vivid.
That's the part worth sitting with. Not the hours. The durability of it. A bad early experience with analytics doesn't just disrupt one audit. It shapes how a CAE thinks about analytics for years. It becomes the internal reference point when someone proposes a new initiative: "We tried this. It was a nightmare." The analytical ambition that started the program doesn't disappear – it just becomes easier to justify deprioritizing. Tom estimates roughly 80% of chief audit executives who've tried to launch an analytics program have run into some version of this. The details vary. The shape of the failure is consistent.
Programs don't fail because CAEs lack commitment or vision. They fail in specific, predictable places. Here are the six most common signals of failure – and how to know if you're seeing one right now.
Failure Signal 1: You Underestimated How Long Data Analytics Actually Takes
There are actually two time problems hidden inside this one, and most CAEs budget for neither.
The first is the IT queue problem. Most IT organizations are managing a three-to-four month backlog. The queue is prioritized around revenue: sales needs a new reporting dashboard, operations needs a data feed for a new system, finance needs its month-end extracts. Internal audit's requests for historical transaction data fall at the back of that queue by default. Not because IT is hostile, but because the function doesn't generate revenue, and in any resource allocation conversation, that matters.
The second problem with time is harder to see in advance: the data discovery process itself. When Tom's team finally got their T&E data, they realized it wasn't enough. The fields they'd specified didn't capture everything that mattered. They had to go back. Then they had to go back again. This isn't a mistake – it's the normal experience of working with enterprise data for the first time. You don't know what you don't know about your data until you start trying to build with it. T&E data alone might live across an expense management system, a corporate card platform, a reimbursement module, and a general ledger, each with different field names, different data quality, and different refresh frequencies. Understanding that landscape takes time that almost no analytics project budget accounts for.
The warning signal is straightforward: you're thinking about data access when the audit starts, not three months before.
You're in this situation if the analytics portion of your audits regularly runs past the planned fieldwork window. If you've apologized to an audit customer for a missed date. If your team is still having discovery conversations with the data team after fieldwork has officially begun.
Tom's rule, drawn directly from the Qlik experience: double the time estimate for any first-time analytics project. Only reduce that buffer once you've successfully closed one under the original estimate. The practical fix is simple but rarely done: publish your audit plan on January 1st, then brief IT that same week on every analytics project for the year, in priority order. You're not requesting data yet – you're getting into the queue twelve months in advance. Done consistently, that one conversation changes your entire relationship with the data team. If you're still building the foundation for your program, Article 1 covers how to sequence these early decisions before even thinking about your first analytics audit.
Failure Signal 2: The Business Owner Wasn't in the Room
Audit designs the analytic. Audit builds it. Audit runs it. Audit delivers a list of exceptions. The business owner spends most of their time explaining why the exceptions don't mean what the output suggests. After two cycles, the report stops being reviewed.
This pattern appears in analytics programs at every maturity level, and it usually comes from a legitimate place: the independence instinct. Auditors are trained to protect objectivity by maintaining separation from the business. That instinct, applied too rigidly to the analytic design phase, produces the exact problem it was trying to prevent. By the time you're running the analytic and sharing results, you've already lost the business owner's engagement – because you never established what "normal" looks like in their process.
There's a practical consequence that compounds over time. When the business owner sees exceptions they can't explain, they file them away as "audit things." The analytic gets labeled "the audit tool." Investigation response times slow down. And when you eventually find something that genuinely matters, you're starting that conversation from a posture of low credibility because the prior outputs were mostly noise to the people who know the process best.
The diagnostic question is simple: was the business owner in your first scoping conversation? Not the one where you told them audit was coming – the one where you defined what the analytic should flag, what transactions look genuinely anomalous versus expected, and what a complete investigation would need to establish. If the answer is no, you're building a tool for yourself.
You're in this situation if the business owner regularly responds to exceptions with "that's expected because..." and you didn't know that context in advance. If investigation response time is declining each cycle. If no one from the first line has ever asked to see results before you sent them. If the framing Tom raises in the context of every analytics relationship – what's in this for them? – doesn't have a clear answer.
The fix is a single scoping session before you build anything, with the business owner at the table, specifically to define what normal looks like. Not to share audit's view – to learn theirs. Article 3 covers what genuine first-line partnership looks like, and Article 4 gets at the deeper shift in how business leaders experience audit when that relationship is working. This is what gets you to the starting point.
Failure Signal 3: One Person Holds the Whole Program Together
Your analytics program is working. Exceptions are being reviewed, value is being delivered, and the business is starting to engage. Then your analytics lead gets a better offer and gives three weeks' notice.
You inherit dashboards no one else can run, extracts no one else can pull, and analytic logic that exists only in one person's head. The program has a single point of failure. And it just failed.
This is the modern expression of a pattern that's been visible in audit analytics for years. Ask a CAE in 2018 about their analytics capability and you'd get: "We have ACL licenses." Press on actual usage: "We haven't really run anything in three or four years." The investment existed on paper. The muscle was never built. Today you see the same underlying problem not in shelf licenses but in concentrated expertise – one person who is, effectively, the entire program.
You're in this situation if only one person on your team can run or explain any given analytic. If onboarding a new hire to the analytics program takes months. If exceptions pile up uninvestigated whenever that person is absent.
Tom O'Reilly points to one audit function he knows of as a model worth following. A small dedicated analytics group handles the complex work, but their explicit mandate is also to bring the broader team along – training every auditor on basic data literacy: how to read transaction-level output, how to investigate an exception with rigor, how to recognize when data quality undermines a finding. No single departure can hollow the program out because the capability is distributed. The dedicated analytics function handles complexity. The distributed capability handles daily production and critical thinking.
Failure Signal 4: The Team Can't Read What the Analytics Are Telling Them
The routines are running. Exceptions are being generated. Investigation tickets are being opened. Six months in, nothing material has been found.
The program isn't broken technically. It's broken in translation.
The warning signal is how your team describes exceptions. If the language is system-framed – "the routine flagged a variance in field X" – rather than business-framed, you have a readiness gap. The data is coming out. The interpretation isn't happening.
This produces a failure mode that's hard to see from the outside because the program looks active. Exceptions are logged. Tickets are opened and closed. Reports are produced. But the underlying analytical work is documentation rather than investigation. The same exception types appear cycle after cycle without root cause conclusions, because nobody on the team has asked why that exception keeps appearing or whether fixing the underlying condition would make it disappear. The program has become an activity, not a function.
There's a structural reason this happens. Good analytics work requires two things that rarely live in the same person at the right level: deep ERP and data literacy, and the audit judgment to understand what an exception means for business risk. Your data-fluent team member may not have the process knowledge to understand that the exception in field X is almost always a timing artifact of how the business books intercompany transactions at quarter end. Your most experienced auditor may not be comfortable enough with the data to question whether the extract is pulling the right population in the first place. Neither person alone can close the loop.
You're in this situation if exceptions are being documented without a clear explanation of business impact. If the same exception patterns appear in cycle after cycle without root cause conclusions. If team members escalate every interpretive question to one senior person.
The test is specific: pull your last 20 investigated exceptions and look at what's written in the notes. If you can't find clear explanations of business impact, root cause analysis, or conclusions about whether the exception represents a control failure or a legitimate transaction, you have this problem. An analytic isn't production-ready until your team can explain in plain language why a given exception matters – and what an investigation needs to establish before it can be closed.
Failure Signal 5: A License Is Not a Strategy
The purchase feels like progress. Leadership approved the budget, the contract is signed, onboarding is scheduled. Ten months later, the license is barely in use, implementation is stalled, and renewal is approaching.
This is what buying before the use case looks like from the inside. The tool was acquired before anyone could clearly answer: what specific problem are we solving, which audits will be materially better because of this, and what will look different in six months?
What makes this pattern expensive is the renewal trap. After a year of limited usage, the conversation with leadership becomes uncomfortable. The team defends the investment by describing future plans rather than current value. A second year gets approved on the basis of potential rather than performance. The underlying problem – no clear use case, no operational integration, no ownership of which audits this changes – doesn't get fixed. It just gets funded for another cycle.
Celia Viguier, US General Manager at Supervizor, describes a pattern she's seen repeat across organizations: a team purchases a tool for largely aspirational reasons – "we want analytics" – and ends up with a list of exceptions every month that no one knows what to do with. The issue isn't the tool. It's the absence of a prior decision about what the tool was supposed to surface, who was supposed to investigate it, and how that investigation was supposed to connect to audit findings or business change.
You're in this situation if the tool has been live for more than six months with fewer than a handful of analytics in production. If the team isn't sure which audit to apply it to first. If there's no documented answer to what success looks like before renewal.
As Tom puts it: if you're not solving a business problem, you'll struggle to succeed – and you'll struggle even more to get the support from people outside of audit that you need to make it work. The roadmap comes before the tool. The specific use case comes before the build. Leadership buy-in comes before the budget conversation. Article 1 covers how to structure that foundation from the start.
Failure Signal 6: Right Vision, Wrong Scale
This one is less common than the others, but more expensive when it arrives – because it looks like success until it isn't.
The program launches with maximum ambition: dozens of analytics routines, multiple systems, global coverage, everything running simultaneously. The goal is comprehensive risk visibility. The vision is sound. Within the first few months, the team is underwater – not because the analytics aren't working, but because there are too many of them to act on.
Consider a team that launched with 300 analytics routines running across 150 countries and three ERPs. The objective was comprehensive fraud detection: broad enough that when a fraud eventually surfaced anywhere in the enterprise, the data would be there to find it. The logic was impeccable. In practice, each monthly cycle produced an unmanageable volume of exceptions with no prioritization framework. The team flagging and investigating 300 routines across 150 countries had no clear starting point. When a real risk eventually surfaced, it was invisible in the noise. The program technically worked. Nobody could use it.
Here's the fraud detection paradox that this example illustrates: comprehensive coverage without investigation capacity produces worse fraud detection than targeted coverage with disciplined follow-through. The instinct to see everything is right. But every exception your program generates that goes uninvestigated is an implicit decision that it doesn't matter. High exception volume without prioritization logic creates investigator fatigue. Fatigue creates systematic under-investigation. And systematic under-investigation means your effective coverage rate is lower than a well-designed analytic on a fraction of the population would produce.
The warning signal is the gap between exceptions generated and exceptions actually investigated. If your investigation rate sits consistently below 20 to 30 percent of what's flagged, you have a volume problem masquerading as a coverage program.
The fix isn't a better triage system. It's proving adoption before expanding scope. Every time you add a new analytic to an already-stretched team, you're not adding capacity – you're diluting the investigation quality of everything that's already running. Start with three analytics with clear owners and measurable follow-through. Only expand when you've proven that every exception the current set generates is being investigated to a conclusion. Article 2 covers the framework for deciding which analytics are worth running repeatedly – and why that selection decision matters more than coverage volume.
The Common Thread
Every one of these signals has the same underlying structure: someone tried to move faster than the organizational conditions allowed, or built something technically sound that the people responsible for using it weren't yet equipped to act on.
What makes these situations particularly costly is that they tend not to appear in isolation. A team that hits Failure Signal 1 (data timing) and Failure Signal 2 (no business owner) simultaneously on their first analytics audit is likely to walk away from the program entirely. And because the experience lands as a single confusing event – the audit blew up, the team was overwhelmed, the business owner was disengaged – the CAE often can't identify the specific conditions that caused it. Without that diagnosis, the conclusion becomes: analytics doesn't work here. That attribution error is what Tom is describing when he says the memory stays with you for a decade.
The right response isn't to restart the program. It's to name the specific signal, fix that condition, and prove one clean cycle before expanding. Not fix everything at once. Not overhaul the approach. Find the specific thing that stalled and repair it.
Programs that survive their first two years don't do so because they avoided all of these moments. They survive because the CAE was willing to diagnose what was actually happening rather than attribute the difficulty to analytics in general – and because they treated the recovery as evidence that the program was alive, not proof that it had failed.
The next article in this series covers what's possible once you've gotten through these moments: an audit function operating at a level of analytical maturity that opens doors well beyond transaction testing.
.jpg){kind=icon}