Many of us are competitive by our nature and one way that we feel the thrill of success is by saying that we have performed better than another. In the recent Olympics, champions received medals to demonstrate that they are the best-in-class. In healthcare as in many other industries, we look to measure ourselves against benchmarks.
The National Institute of Health (NIH) defines benchmarking as “a strategic and analytical process of continuously measuring an organization’s products, services and practices against a recognized leader in the studied area for the purpose of improving business performance.”1
Nearly every KHC client asks, “How is our performance compared to your other clients? Do you have a benchmark for that?” Hospitals and other providers have troves of data and a seemingly endless way of measuring their organization’s performance. The other day, I worked through a limited dataset live with a compliance officer and in one hour and we identified five new metrics for observation services. However, clients always want to know how they compare to others in order to measure success.
When making the leap directly to an external benchmark to determine if you perform better or worse than a peer, the context is overlooked. Every company and situation is unique and external benchmarks do not contain enough information to determine if the data that supports the metric was determined in a manner that is consistent with how the company will utilize it that benchmark.
When considering a benchmark, there are key questions to help determine if it is meaningful to you:
- What is the source of data? Self-reported data is subjective as different respondents may interpret questions or data points differently.
- What is the lag time between data collection and reporting? Is a benchmark that is reported in June based on data from the prior year helpful or even relevant?
- What key factors were considered in developing the benchmarks? Payer mix? Payer contract terms? Geographic location? Provider specialty?
- What industry or market circumstances might be reflected in the data that does or does not apply to my organization?
- What is not included in the benchmark?
I was recently digging through the Centers for Medicare and Medicaid Services (CMS) Part D Prescriber Provider Utilization and Payment (PUF) Data to find some contextually relevant benchmarks for a client. The data is accompanied by a 28-page methodology document that outlines what is included and what is NOT included in the various data sets.2 Although there was a sufficient amount of data available, what was not included impacted any benchmarks in such a way that they were no longer meaningful.
I have also found that a significant amount of time is spent justifying the benchmark and explaining why it does not consider the specifics to the internal organization. Importantly, benchmarks do not help identify the impact of changes made to the organization.
Operationally, it is more effective to measure the current performance against the desired results. External benchmarks can be used to provide guideposts as to what the desired result is and a month over month analysis against an internal benchmark can demonstrate performance improvement in relevant areas based on the organization’s individual and unique context and goals. This also considers the reality that the team that performs the work drives productivity, quality, and the final outcome.
Our auditing clients frequently request comparative error rates. As each audit has a specific goal and each organization has different coding policies; coding team responsibilities; and physician involvement; comparative error rates can only be utilized as a high-level guideline to help establish what the desired result should be for that organization.
Consider physician compensation as another example. HFM Magazine’s Summer 2021 issue has an article by Stuart Schaff, “We must stop relying so heavily on benchmark tables to set physician pay.” This statement succinctly explains why external benchmarks expose a hospital to risk because the hospital faces “difficulties not only in foreseeing the changes in benchmark data” and also details how the benchmarks generally used in physician compensation are based on a voluntary response and are self-reported. Mr. Schaff instead proposes a physician compensation model that accounts for the physician’s specialty and other aspects of the physician’s work and pay.
In KHC compliance and litigation work, opposing experts frequently lose when we are able to explain why the analytics are not relevant to the case. A few good examples are when are when an opposing expert used lower prices from 2018 to reprice claims from nearly a decade earlier, using Medicaid volumes from geographic locations with a lower Medicaid population to attempt to prove fraud or comparing nursing home ratings in one state to another to establish performance metrics. In each of these examples, the benchmarks that were used were not of the same context as the measurement against which they were compared.
As one that lives in data, benchmarking is an analytics tool that I use daily as part of my work. However, time must be spent refining the benchmark calculation to ensure that it is within the context of the problem. Frequently, as I spend more time refining the calculation, I’m able to identify more than one benchmark that needs to be measured in order to measure against the goal.
Resources:
- National Institutes of Health, Office of Management: https://ors.od.nih.gov/OD/OQM/benchmarking/Pages/benchmarking.aspx.
- Medicare Fee-For Service Provider Utilization & Payment Data Part D Prescriber Public Use File: A Methodological Overview, https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Medicare-Provider-Charge-Data/Downloads/Prescriber_Methods.pdf.
Data problem? Analytics? Regulations? Data privacy? Humanity? Coffee and talk? You can reach me at jleventhal@kohlerhc.com or 312.933.2752.
Josh Leventhal is an expert in health and is Managing Director with Kohler HealthCare. He has over 15 years’ hands-on experience in healthcare data and analytics solving problems for providers, payers and life science organizations. Josh started his career in management consulting analyzing data for the largest joint defense litigations in the country before using his skills and expertise at local startups to assist the Medicaid managed care organization and medical research industries. His experiences as a consultant, product manager and developer allow him to work effectively with both business and technology stakeholders.
