The Myth About Pass/Fail Rates for Providers

Projecting results of an audit to a larger population of providers can be a serious step.

Many of the organizations with which I engage feel this pressing need to use the results of individual provider audits to infer an overall pass/fail rate, either to everything that provider does or to the organization.

I get the reason for this; leadership wants a general idea of what their compliance risk and exposure looks like, and they often want to know which providers are contributing most to that risk. The problem with inferring those audit results to a larger population has to deal with both sampling methods and sample sizes for those audits. And much of that is driven by the model used to construct the audit. For example, some organizations are still caught in the random probe audit cycle, something that should have died years ago. The problem with a probe audit is that, unless it is targeted to a specific code, it misses the overwhelming majority of risk events. Take an internal medicine provider. If you look at just the Medicare submission database, IM docs billed 140 unique procedures/services (based on the code), of which 92 make up the top 80 percent. So let’s take a look at how this might break down. Suppose an organization using a legacy model, such as a probe audit, might have a mandate to audit 10 encounters per provider, per year. If it’s not a focused audit (in essence, a general random probe audit), the maximum number of codes you would be able to review would be 10, or 7 percent of all the unique codes the provider reported. That means that you just ignored 93 percent of the compliance risk events. And by the way, the chance of getting 10 unique codes in a random pull is something like 1.2 x 1020, or 1.2, followed by 20 zeros. You probably have a better chance of getting eaten by a shark while being struck by lightning than getting 10 unique codes. In this case, you would still only end up with one encounter to audit, and who believes that it is OK to extrapolate from one encounter?

How about “nobody?”

Maybe it’s a focused audit, so you audit 10 encounters for that provider for a specific code, say, 99233. The results of the audit reveal that 3 of those 10 encounters failed the audit for one reason or another. What’s easy is to say that this provider’s error rate is 30 percent, and if you were talking about the number of encounters that failed, you would be correct, because we are not inferring the results (yet), but rather describing the results. Where we go wrong is when we infer that 30 percent of all of the provider’s 99233 codes are coded wrong. Why can’t we do that? Well, it is based on the idea of sample error, which occurs with every single sample, as defined by the standards of statistical practice. While three out of 10 is 30 percent, from a descriptive perspective, three out of 10 is actually somewhere between 6.7 percent and 65.2 percent when inferring the results to a larger population. Many would know this process as extrapolation, and it is an inherent problem with extrapolations are conducted.

So, how many encounters would I have to audit in order to be able to extrapolate the results to the population of codes, or claims, or beneficiaries, or whatever the unit being audited? That depends on your tolerance for error. I am a fan of the 95 percent confidence interval. This is a way of measuring the relative error in our point estimate (30 percent is the point estimate, and the range is the margin of error), and it is a commonly accepted metric. Here’s what it means: if I were to pull 100 samples of 10 of the 99233 codes, in 95 of those 100 (or 95 percent), the actual error rate would be somewhere between 6.7 and 65.2 percent. That’s factual, but is it useful? That’s for you to decide, but in my world, that large of a range (or sampling error) is pretty much useless. So, back to the question of how many encounters are needed. Well, let’s say we pulled 100 encounters, and 30 of those were in error. In describing the error rate for that sample, we can still say that the point estimate is 30 percent, and we would be right on the mark. Inferentially, however, the 95 percent confidence interval is now 21.2 percent to 39.9 percent, and if you are OK with a plus-or-minus 10 percent, then you have your number. If not, then you will have to do more. How many more depends on a few assumptions, and you can use a sample size calculator (many can be found online) to calculate the sample size you need for your purposes.

Let’s say that you audited 10 of the 99233 codes for 100 providers. Now you have a sample size of 1,000. Let’s say again that 300 of them were found to have been coded in error. Now your range is from 27.1 percent to 32.9 percent, and I wouldn’t have any problem saying that there is a general error rate among all providers for the 99233 code of 30 percent, but I would not be comfortable predicting that for any individual provider, since their sample size is simply too small. Heck, you could also then analyze the averages of the results from those 100 samples, and having satisfied the Central Limit Theorem, the foundational axiom behind inferential statistics, you could be quite accurate in your projection. And you would likely have a normal distribution, which can be used for a lot more fun calculations. But more on that in another article.

I know that this can get a bit tedious, but projecting the results of an audit to a larger population of codes or providers can be a pretty serious step. For one, you may infer that a given provider has a bigger problem than it really does. Or you may push an organization into a more expensive (yet statistically valid) review of some codes or providers when it wasn’t necessary in the first place.

I am all for using statistics to estimate error rates, but I am not for extrapolating those error rates when doing so is not justified. The risks almost always outweigh the benefits. One of my favorite quotes is from George Box, a famous statistician, who said that “all models are wrong, but some are useful.”

And that’s the world according to Frank.

Program Note:
Listen to Frank Cohen report this story live during Monitor Monday, May 20, 10-10:30 a.m. ET.

Facebook
Twitter
LinkedIn

Frank Cohen

Frank Cohen is Senior Director of Analytics and Business Intelligence for VMG Health, LLC. He is a computational statistician with a focus on building risk-based audit models using predictive analytics and machine learning algorithms. He has participated in numerous studies and authored several books, including his latest, titled; “Don’t Do Something, Just Stand There: A Primer for Evidence-based Practice”

Related Stories

Leave a Reply

Please log in to your account to comment on this article.

Featured Webcasts

Enhancing Outcomes with CDI-Coding-Quality Collaboration in Acute Care Hospitals

Enhancing Outcomes with CDI-Coding-Quality Collaboration in Acute Care Hospitals

Join Angela Comfort, DBA, MBA, RHIA, CDIP, CCS, CCS-P, as she presents effective strategies to strengthen collaboration between CDI, coding, and quality departments in acute care hospitals. Angela will also share guidance on implementing cross-departmental meetings, using shared KPIs, and engaging leadership to foster a culture of collaboration. Attendees will gain actionable tools to optimize documentation accuracy, elevate quality metrics, and drive a unified approach to healthcare goals, ultimately enhancing both patient outcomes and organizational performance.

November 21, 2024
Comprehensive Inpatient Clinical Documentation Integrity: From Foundations to Advanced Strategies

Comprehensive Outpatient Clinical Documentation Integrity: From Foundations to Advanced Strategies

Optimize your outpatient clinical documentation and gain comprehensive knowledge from foundational practices to advanced technologies, ensuring improved patient care and organizational and financial success. This webcast bundle provides a holistic approach to outpatient CDI, empowering you to implement best practices from the ground up and leverage advanced strategies for superior results. You will gain actionable insights to improve documentation quality, patient care, compliance, and financial outcomes.

September 5, 2024
Advanced Outpatient Clinical Documentation Integrity: Mastering Complex Narratives and Compliance

Advanced Outpatient Clinical Documentation Integrity: Mastering Complex Narratives and Compliance

Enhancing outpatient clinical documentation is crucial for maintaining accuracy, compliance, and proper reimbursement in today’s complex healthcare environment. This webcast, presented by industry expert Angela Comfort, DBA, RHIA, CDIP, CCS, CCS-P, will provide you with actionable strategies to tackle complex challenges in outpatient documentation. You’ll learn how to craft detailed clinical narratives, utilize advanced EHR features, and implement accurate risk adjustment and HCC coding. The session also covers essential regulatory updates to keep your documentation practices compliant. Join us to gain the tools you need to improve documentation quality, support better patient care, and ensure financial integrity.

September 12, 2024

Foundations of Outpatient Clinical Documentation Integrity: Best Practices for Accurate Coding and Compliance

This webcast, presented by Angela Comfort, DBA, RHIA, CDIP, CCS, CCS-P, a recognized expert with over 30 years of experience, offers essential strategies to improve outpatient clinical documentation integrity. You will learn how to enhance the accuracy and completeness of patient records by adopting best practices in coding and incorporating Social Determinants of Health (SDOH). The session also highlights the role of technology, such as EHRs and CDI software, in improving documentation quality. By attending, you will gain practical insights into ensuring precise and compliant documentation, supporting patient care, and optimizing reimbursement. This webcast is crucial for those looking to address documentation gaps and elevate their coding practices.

September 5, 2024

Trending News

Featured Webcasts

Patient Notifications and Rights: What You Need to Know

Patient Notifications and Rights: What You Need to Know

Dr. Ronald Hirsch provides critical details on the new Medicare Appeal Process for Status Changes for patients whose status changes during their hospital stay. He also delves into other scenarios of hospital patients receiving custodial care or medically unnecessary services where patient notifications may be needed along with the processes necessary to ensure compliance with state and federal guidance.

December 5, 2024
Navigating the No Surprises Act & Price Transparency: Essential Insights for Compliance

Navigating the No Surprises Act & Price Transparency: Essential Insights for Compliance

Healthcare organizations face complex regulatory requirements under the No Surprises Act and Price Transparency rules. These policies mandate extensive fee disclosures across settings, and confusion is widespread—many hospitals remain unaware they must post every contracted rate. Non-compliance could lead to costly penalties, financial loss, and legal risks.  Join David M. Glaser Esq. as he shows you how to navigate these regulations effectively.

November 19, 2024
Post Operative Pain Blocks: Guidelines, Documentation, and Billing to Protect Your Facility

Post Operative Pain Blocks: Guidelines, Documentation, and Billing to Protect Your Facility

Protect your facility from unwanted audits! Join Becky Jacobsen, BSN, RN, MBS, CCS-P, CPC, CPEDC, CBCS, CEMC, and take a deep dive into both the CMS and AMA guidelines for reporting post operative pain blocks. You’ll learn how to determine if the nerve block is separately codable with real life examples for better understanding. Becky will also cover how to evaluate whether documentation supports medical necessity, offer recommendations for stronger documentation practices, and provide guidance on educating providers about documentation requirements. She’ll include a discussion of appropriate modifier and diagnosis coding assignment so that you can be confident that your billing of post operative pain blocks is fully supported and compliant.

October 24, 2024
The OIG Update: Targets and Tools to Stay in Compliance

The OIG Update: Targets and Tools to Stay in Compliance

During this RACmonitor webcast Dr. Ronald Hirsch spotlights the areas of the OIG’s Work Plan and the findings of their most recent audits that impact utilization review, case management, and audit staff. He also provides his common-sense interpretation of the prevailing regulations related to those target issues. You’ll walk away better equipped with strategies to put in place immediately to reduce your risk of paybacks, increased scrutiny, and criminal penalties.

September 19, 2024

Trending News

Happy National Doctor’s Day! Learn how to get a complimentary webcast on ‘Decoding Social Admissions’ as a token of our heartfelt appreciation! Click here to learn more →

👻Spooky Sale is Back!👻 Get 31% off all three Medlearn brands, using code SPOOKY24.