|
A correction has been published: N Engl J Med 2008;359(5):546.
|
Background In the English pay-for-performance program, physicians use a range of criteria to exclude individual patients from the quality calculations that determine their pay. This process, which is called exception reporting, is intended to safeguard patients against inappropriate treatment by physicians seeking to maximize their income. However, exception reporting may allow physicians to inappropriately exclude patients for whom targets have been missed (a practice known as gaming).
Methods We analyzed data extracted automatically from clinical computing systems for 8105 family practices in England (96% of all practices), data from the U.K. Census, and data on practice characteristics from the U.K. Department of Health. We determined the rate of exception reporting for 65 clinical activities and the association between this rate and the characteristics of patients and medical practices.
Results From April 2005 through March 2006, physicians excluded a median of 5.3% of patients (interquartile range, 4.0 to 6.9) from the quality calculations. Physicians were most likely to exclude patients from indicators that were related to providing treatments and achieving target levels of intermediate outcomes; they were least likely to exclude patients from indicators that were related to routine checks and measurements and to offers of treatment. The characteristics of patients and practices explained only 2.7% of the variance in exception reporting. We estimate that exception reporting accounted for approximately 1.5% of the cost of the pay-for-performance program.
Conclusions Exception reporting brings substantial benefits to pay-for-performance programs, providing that the process is used appropriately. In England, rates of exception reporting have generally been low, with little evidence of widespread gaming.
A key consideration for pay-for-performance programs is avoiding inappropriate treatment of patients to whom a quality indicator does not apply or for whom other considerations take precedence.11,12,13,14 There are three approaches to this problem. The first is to risk-adjust the quality indicators15 — for example, by constructing indicators that allow for as many combinations of coexisting illnesses as possible.16 This approach makes indicators very complex, and it is never possible to allow for all eventualities. The second approach sets maximum-achievement thresholds below 100% — in other words, physicians can earn the maximum financial rewards without achieving the targets for all patients. In the United States, this approach has been used for the Bridges to Excellence program, which sets thresholds of 25 to 80% for aspects of cardiac and diabetic care,17 and the Medicare Physician Quality Reporting Initiative.18 However, determining which proportion of patients would not meet the criteria for each quality indicator is extremely difficult. The third approach, adopted in some U.S. programs,19,20 allows physicians to use their clinical judgment to remove inappropriate patients from calculations of quality achievement. In the United Kingdom, the pay-for-performance program adopts the second and third approaches. The removal of inappropriate patients in the United Kingdom is called exception reporting (Table 1).
|
Methods
Study Design
In the United Kingdom, primary care is provided by primary care practices, comprised of up to 10 primary care physicians (mean, 3.5) who work from common premises. The contract for providing medical care and all quality payments relate to the practice rather than to the individual physician. There is a single common electronic medical record for all practices, and data on quality of care are extracted automatically from clinical computing systems of practices and are collated in the central National Health Service Quality Management and Analysis System (QMAS) database.
The analyses that we describe here relate to all 65 clinical indicators that were used in the second year of the U.K.'s pay-for-performance program. Data were available for the prevalence of each of the 10 diseases listed in the program, since practices are required to maintain registers of patients with relevant diagnoses and the quality points earned for each indicator. Quality points translate into physicians' income through a formula that takes into account the prevalence of each disease.21 These data are published nationally by the National Health Service Information Centre,22 which also provided data on the number of patients who were excluded for each indicator according to medical practice. Although the reasons for exception reporting are recorded on the computing systems of practices, they are not collated in the QMAS database and hence were not available.
For each practice, we calculated the rate of exception reporting, which was defined as the number of patients who were excluded for each indicator as a proportion of the number of patients who were eligible for the target. We constructed summary scores for each disease and each type of clinical activity using data for the relevant indicators (Table 2). An overall score was constructed as an unweighted mean across all indicators.
|
The QMAS database contains data from 8409 family practices in England. Practices were excluded from the study if they had fewer than 1000 patients (44 practices), if one or more disease registers were missing (107 practices), or if data regarding exception reporting were missing or incomplete (175 practices). Our main results are drawn from 8105 practices, which provide care for more than 49 million patients. Complete socioeconomic data were not available for 476 practices (5.9%), so these practices were excluded from the regression analyses. As a group, these practices did not differ significantly in terms of the rates of exception reporting. Since this study was based on publicly available data, it did not require approval from institutional ethics committees.
Statistical Analysis
We analyzed the effect of the characteristics of patients and medical practices on rates of exception reporting, using a multiple linear regression model reporting least-square means with robust estimates of error variance. Area indicator variables were included to allow for unobserved effects of local National Health Service policies. All explanatory variables were divided by their standard deviation, so reported coefficients show the increase in the standard deviation of rates of exception reporting for each 1-SD increase in predictor variables.
We also analyzed the financial gain from exception reporting (i.e., the number of quality points earned by practices that could be attributed to exception reporting). Analyses were performed with the use of Stata software, version 9. Further details regarding the methods used for calculating rates of exception reporting and financial gain are reported in the Supplementary Appendix, available with the full text of this article at www.nejm.org.
Results
Clinical Targets
Overall, medical practices achieved clinical targets for a median of 84.6% of eligible patients (interquartile range, 81.5 to 87.1) on their chronic-disease registers. Of the patients who were not excluded, practices achieved clinical targets for a median of 90.4% of patients (interquartile range, 87.4 to 92.5).
Rate of Exception Reporting
The median rate of exception reporting for all practices in 2005–2006 was 5.3% (interquartile range, 4.0 to 6.9) (Table 3). This rate ranged from 0 to 28.3% according to practice (Figure 1). The rates of exception reporting for all 65 individual indicators are listed in the Supplementary Appendix.
|
|
In terms of type of activity, the lowest median rate of exception reporting was for offering treatment (1.4%), and the highest was for providing treatment (12.6%). Almost half the indicators involved regular review of patients with chronic diseases (e.g., routine measurements of blood pressure and cholesterol). This group of indicators had relatively low rates of exception reporting (median, 2.5%). The exception was the cancer indicator, which required a comprehensive review of patients who potentially could be seriously ill. Rates of exception reporting for diagnostic and referral activities were generally higher than average (median, 6.5%), the exception being peak flow measurements for patients with asthma, which could be carried out in practice without referral to an external agency. The fifth group of activities — achieving intermediate outcomes — was the only one that involved outcomes rather than processes (e.g., achieving target levels of cholesterol and blood pressure), and rates of exception reporting for these activities were moderately high (median, 7.1%).
There was no correlation between the mean rate of exception reporting and the number of points (and hence financial rewards) available for individual indicators (Spearman's correlation coefficient, 0.06). The individual indicators with the highest median rates of exception reporting were prescribing beta-blockers for patients with coronary heart disease (24.7%), vaccinating patients with asthma against influenza (16.0%), and keeping patients with epilepsy free from convulsions (13.8%).
Characteristics of Patients and Practices
Increased overall rates of exception reporting were associated with practices located in densely populated areas and those with relatively small proportions of patients under the age of 16 years or over the age of 64 years (Table 4). Large practices also excluded a higher proportion of patients. Logistic-regression analysis identified the latter two variables — having a large practice or one with a skewed age range — as the only significant predictors of practices being in the top 1% of those reporting exceptions. All these effects were modest. For example, an increase of 1000 patients in the practice population was associated with an increase of 0.04% in the rate of exception reporting, and the variables that were included in the multiple regression model explained only 2.7% of the variance (R2). Although there were some significant deviations for individual diseases and types of activity, these differences did not conform to any meaningful pattern.
|
Financial Gain
Out of a maximum 492 points for the clinical-activity indicators, practices gained a median of 13.9 points (interquartile range, 8.6 to 21.2) from exception reporting. As a proportion of the total points available for these indicators, this score translated into a median of 2.8% (interquartile range, 1.7 to 4.3). The percentage gain ranged from 0% percent (in 15 practices) to 25.4% (in 1 practice) (Figure 2). Given an average reward of £125 (approximately $250, in U.S. dollars) per point, practices gained a median of £1,738 ($3,476) as a result of exception reporting, with a maximum financial gain for an average-size practice of £15,500 ($31,000).
|
When the United Kingdom's pay-for-performance program was introduced, there was uncertainty about the effects of allowing physicians to exclude patients from the reporting system. Data were not available for exception reporting during the first year of the program, and rates could be estimated for only 30 of the 76 clinical indicators.10 Our study describes a large-scale analysis of actual data on exception reporting, although the data are limited by the absence of reasons for exclusions and are available only at the practice level and not at the patient level. We also cannot differentiate between patterns of reporting by physicians, nurses, and practice managers, all of whom have the power to exclude patients.
Overall levels of exception reporting were generally modest in 2005–2006, averaging less than 6%. This rate is similar to the levels we estimated for the first year of the pay-for-performance program in 2004–2005 for a subgroup of clinical indicators.10 However, the maximum rate of exception reporting appears to have decreased substantially, from an estimate of 86% in 2004–2005 to 28% in 2005–2006. This reduction in rates mainly affected practices with very high rates of exception reporting, since the 75th percentiles were similar in 2004–2005 and 2005–2006 (7.7% and 6.9%, respectively).
We estimate that exception reporting increased the income in practices by an average of £1,738 ($3,476) and the income of individual physicians by about £500 ($1,000), which is less than 0.5% of the average net income of a family practitioner. The total cost to the Department of Health in England was approximately £17.2 million ($34.4 million) in 2005–2006. Given that the overall cost of the pay-for-performance program in that year was approximately £1.15 billion ($2.3 billion) (Vickerman M: personal communication), all exception reporting accounted for 1.5% of the cost of the program.
The generally low overall rates of exception reporting masked some wide variations — for example, in rates of exception reporting for the same activity in different groups of patients. These variations were greatest for the targets for influenza vaccination, with an increase of 60% in rates of exception reporting for patients with asthma, as compared with patients with coronary heart disease. This difference might be explained in part by the different age profiles of these two groups of patients. There were also high exclusion rates for targets on reviewing the support needs of patients with mental-health issues or cancer, differences that might reflect long stays in secondary care or transfers to hospices and informal care providers outside the practice locale.
There was even greater variation in rates of exception reporting according to the type of activity. More straightforward activities, such as performing measures of disease and offering advice on smoking cessation, had very low rates for most disease groups, whereas rates for activities associated with administering treatments and achieving intermediate outcomes were increased by a factor of 6. Gaming is a possible explanation for these differences, since the activities with the highest rates of exception reporting tended to be more demanding of physicians. However, there are legitimate reasons for higher rates of exception reporting for treatment and intermediate outcome targets. Extreme frailty, for example, is a greater impediment to the successful treatment of hypertension than it is to simple monitoring of the condition.
The potential for gaming remains an argument against allowing exception reporting. We found only minor and inconsistent associations between practice performance in the first year and exception reporting the following year, but any tendency to inflate exclusions fraudulently would be affected by the maximum-achievement thresholds. Analysis of data from Scotland has shown that rates of exception reporting in 2005–2006 were higher for practices that had levels of achievement below the maximum thresholds in the previous year than for those that had levels above the maximum thresholds.24,25
This finding raises the possibility that some practices may have gamed exceptions in order to receive maximum remuneration. However, we found no association between the rate of exception reporting and the size of the financial incentive available for individual indicators. Nor was it the case that activities for which it would be more difficult to fraudulently inflate achievement rates (due to the involvement of an external agency, such as a pathology laboratory or pharmacy) always had higher rates of exception reporting. These findings, along with the generally low levels of exclusions, suggest that extensive gaming of exception reporting is unlikely to have occurred.
Monitoring arrangements may have contributed to this finding. In England, Primary Care Trusts (statutory bodies that are entrusted with monitoring) annually inspect all practices as part of the pay-for-performance program and can access the reasons used for exception reporting. However, available resources currently limit these trusts to carrying out more detailed inspections on statistical outliers.
Another argument against allowing exception reporting is that excluded patients might be those in greatest need. In earlier studies, which involved smaller numbers of practices and either a limited number of quality indicators26 or indirect measures of exception reporting,27 practices in financially deprived areas were more likely to exclude patients. However, we found that the characteristics of practices and patients, including whether they were in an area of socioeconomic deprivation, had only marginal effects on rates of exception reporting. As compared with our analyses for 2004–2005,10 this model explains less of the variation between practices. However, after accounting for extraneous error due to missing data, we found that the levels of variance explained by sociodemographic predictors were similar in both the first year and the second year of the program (for details, see the Supplementary Appendix). Our findings indicate that the high rates of achievement that were reported by practices across the sociodemographic spectrum10 were not achieved because practices in deprived areas excluded many more patients.
An argument in favor of exception reporting is that when the provision is properly used, only patients for whom the targets are inappropriate are excluded. Maximum thresholds lack this precision and are fixed for all practices. If arbitrary thresholds are set too high, they may result in inappropriate treatment for some patients; if they are set too low, they allow physicians to receive maximum remuneration without treating all eligible patients. If exception reporting is allowed, maximum thresholds are superfluous for the purpose of avoiding inappropriate treatment, although they might still be required for indicators for which 100% achievement is unrealistic. If exception reporting is not allowed, then the adoption of maximum-achievement thresholds below 100% would be necessary to avoid inappropriate treatment. The results of our study give some indication of what the thresholds should be in this case.
The United Kingdom's experience with pay-for-performance programs provides some important lessons for other countries. As a system for safeguarding against inappropriate treatment, exception reporting has substantial benefits: it is precise, it increases acceptance of the program by physicians, and it ameliorates perverse incentives to refuse care to "difficult" patients.28,29 The feared drawbacks of exception reporting — that the provision would substantially increase the cost of the program and would be widely abused — appear to be unfounded. This finding may be related to peculiarities of the program in the United Kingdom, but there is no reason to believe that exception reporting would fail in other pay-for-performance programs, provided that effective systems to monitor for inappropriate exclusions and penalties for gaming are in place. Further longitudinal research is now required to study the complex links between exception reporting and levels of achievement, maximum thresholds, and size of incentives, as well as to explain why, rather than simply how often, physicians exclude certain patients from their reporting.
The National Primary Care Research and Development Centre receives core funding from the U.K. Department of Health. In 2002 and 2003, Dr. Roland was a member of a small team of academics who advised the government and professional negotiating teams that developed the U.K. pay-for-performance program. No other potential conflict of interest relevant to this article was reported.
The views expressed in this article are those of the authors and not necessarily those of the U.K. Department of Health.
Source Information
From the National Primary Care Research and Development Centre, University of Manchester, Manchester (T.D., C.F., D.R., M.R.); and the National Primary Care Research and Development Centre, Centre for Health Economics, University of York, York (H.G.) — both in the United Kingdom.
Address reprint requests to Dr. Doran at the National Primary Care Research and Development Centre, Williamson Bldg., University of Manchester, Manchester M13 9PL, United Kingdom, or at tim.doran{at}manchester.ac.uk.
References
| |||||||||||||||||||||||||||||||||||||||||||||||||||
Related Letters:
Pay-for-Performance System for English Physicians
Rosof B. M., Sirio C. A., Kmetik K. S.
Extract |
Full Text |
PDF
N Engl J Med 2008;
359:2176-2177, Nov 13, 2008.
Correspondence
This article has been cited by other articles:
HOME | SUBSCRIBE | SEARCH | CURRENT ISSUE | PAST ISSUES | COLLECTIONS | PRIVACY | TERMS OF USE | HELP | beta.nejm.org Comments and questions? Please contact us. The New England Journal of Medicine is owned, published, and copyrighted © 2009 Massachusetts Medical Society. All rights reserved. |
Previous
