AI in Auditing: Are We Auditing the Machines?

We’ve trained ourselves to audit documentation, audit codes, audit teams – but now we need to ensure we have added something else to the list:

Audit the AI.

That’s right. artificial intelligence isn’t just something we use to assist documentation or streamline chart review. This may be new to some, but AI is shaping decisions in real time, and quietly inserting itself into the audit trail.

And if no one’s reviewing those outputs before they move downstream, we’re not just working faster; we’re working blind.

We’ve all seen AI show up in documentation workflows, coding suggestions, and even chart prioritization.

But now that these tools are becoming embedded into operational systems, we have to move from asking “how can AI help?” to something more urgent:

How do we validate what AI is doing?

Because let’s be clear: these tools are not neutral.

They are scoring documentation risk, pre-filtering audit queues, and suggesting what deserves attention.

That’s useful, but it’s also powerful. And like anything powerful, it needs governance.

So, when a compliance team says “we audit 5 percent of discharges,” it’s time to ask: which 5 percent?

If that sample is based on AI flags, your audit pool is already filtered.

And unless your team knows how the tool flagged those encounters, you could be leaving behind entire categories of risk that simply didn’t make the list.

And here’s where we need to pause and ask the bigger question:

As artificial intelligence becomes part of the audit trail, who’s reviewing the reviewers?

That doesn’t just mean spot-checking an output or nodding at a dashboard.

It means making sure there are real people and defined processes in place to regularly evaluate the logic, challenge questionable flags, and track unintended drift – before a payor or auditor points it out for you.

Now, most vendors won’t hand over the algorithm.

You may not get the full logic, and that’s expected.

But what you should be able to ask is:

• What patterns are driving these flags?
• When was this logic last reviewed?
• Who’s monitoring it for drift, bias, or misalignment?

That’s not just a workflow question; it’s an information governance responsibility.

And that perspective is now being backed by national and international guidance.

Frameworks from NIST, ISO, and EUAIML all emphasize the importance of auditability, explainability, and human oversight, even if full transparency into proprietary systems isn’t possible.

At the same time, agencies such as the U.S. Department of Health and Human Services (HHS) Office for Civil Rights (OCR) and Office of Inspector General (OIG) have flagged concerns about automation that operates without clear policy alignment, particularly when it affects decision-making in healthcare.

And if you’ve built your own internal audit triggers or filtering logic, then that responsibility lives with your team – which means you have the ability to make it stronger.

Try This practice prompt:

“Act as a healthcare compliance auditor. Based on this documentation, would you escalate the chart for review? Why or why not?”

Here’s a fictional (but familiar) example:

“Patient admitted with fall and confusion. CT head negative. Provider notes ‘likely encephalopathy’ and starts antibiotics for pneumonia. No neuro consult. Discharge summary includes ‘encephalopathy resolved.’ Code billed: metabolic encephalopathy.”

Would your AI tool flag that case?

Would your team agree with the diagnosis, or question whether it’s clinically supported?
Would this slide through coding and billing if no one challenged the narrative?

This is how we sharpen oversight: not by resisting automation, but by thinking around it.

Three steps to take right now:

1. Audit the logic influencing your audit program.
   Even without full access, you should know what triggers are in play – and whether they still match payor expectations and clinical guidance.
2. Don’t let automation shrink your scope.
   Intentionally rotate in non-flagged cases. That’s how you catch what the system overlooks – and what it overconfidently approves.
3. Strengthen your information governance lens.
   Oversight of AI tools belong not just to IT or vendor teams, but to compliance, CDI, and clinical documentation leaders. Make sure your policies reflect that.

Because speed doesn’t guarantee accuracy.

And automation without validation is just a fancier version of guessing.

So, I’ll leave you with this:

If AI is now auditing your data, who’s auditing the AI?

And how confident are you in what it’s not showing you?

Stay sharp. Stay curious.

And let’s keep leading this next chapter, strategically and responsibly.