|
|
Background A pay-for-performance scheme based on meeting targets for the quality of clinical care was introduced to family practice in England in 2004.
Methods We conducted an interrupted time-series analysis of the quality of care in 42 representative family practices, with data collected at two time points before implementation of the scheme (1998 and 2003) and at two time points after implementation (2005 and 2007). At each time point, data on the care of patients with asthma, diabetes, or coronary heart disease were extracted from medical records; data on patients' perceptions of access to care, continuity of care, and interpersonal aspects of care were collected from questionnaires. The analysis included aspects of care that were and those that were not associated with incentives.
Results Between 2003 and 2005, the rate of improvement in the quality of care increased for asthma and diabetes (P<0.001) but not for heart disease. By 2007, the rate of improvement had slowed for all three conditions (P<0.001), and the quality of those aspects of care that were not associated with an incentive had declined for patients with asthma or heart disease. As compared with the period before the pay-for-performance scheme was introduced, the improvement rate after 2005 was unchanged for asthma or diabetes and was reduced for heart disease (P=0.02). No significant changes were seen in patients' reports on access to care or on interpersonal aspects of care. The level of the continuity of care, which had been constant, showed a reduction immediately after the introduction of the pay-for-performance scheme (P<0.001) and then continued at that reduced level.
Conclusions Against a background of increases in the quality of care before the pay-for-performance scheme was introduced, the scheme accelerated improvements in quality for two of three chronic conditions in the short term. However, once targets were reached, the improvement in the quality of care for patients with these conditions slowed, and the quality of care declined for two conditions that had not been linked to incentives. Continuity of care was reduced after the introduction of the scheme.
We have previously reported on the quality of clinical care in 2005, the year after the pay-for-performance scheme was introduced.3 We found a modest acceleration in the rate of improvement in the quality of care for asthma and diabetes but not for heart disease. There had been rapid improvement in the quality of care for all three conditions before the introduction of pay for performance. This article extends these analyses to include performance data in 2007. We used an interrupted time-series analysis to examine trends in the quality of clinical care from 1998 through 2007, a period spanning the introduction of pay for performance. We also report on trends in patient reports on communication with their physician, on access to care, and on continuity of care across the same period.
Methods
Data Collection
Trained research staff abstracted clinical data from the medical records kept by 42 nationally representative family practices. In each practice, data were collected for nonoverlapping random samples of patients (20 in 1998 and 12 each in 2003, 2005, and 2007) who had heart disease, asthma, or diabetes; the data were collected with the use of quality indicators.4,5 The methods used to collect data in 2007 were consistent with the methods used in 1998, 2003, and 2005.3
For patient evaluation, a version of the General Practice Assessment Questionnaire (www.gpaq.info) was mailed, with one follow-up reminder, to a random sample of 200 registered adult patients (age, 
18 years) in each practice.6,7 Rapid access to any doctor within 48 hours was associated with an incentive under the pay-for-performance scheme, and our questionnaire included two items addressing the patient's ability to get an appointment within 48 hours with "any doctor" and with "a particular doctor." Because of concern that the scheme's focus on clinical indicators might lead practitioners to neglect other aspects of care,8 we also analyzed communication with physicians and continuity of care. Communication was assessed by asking seven questions, with the answers scored on a six-point scale ranging from "very poor" to "excellent"; continuity of care was assessed with the use of the same six-point scale and a single question: "How often do you see your usual doctor?" All scores were rescaled to range from 0 to 100. The rate of response to the survey was 38% in 1998, 47% in 2003, 45% in 2005, and 38% in 2007. In all cases, higher scores indicate higher quality of care. The research protocol was approved by the North West Research Ethics Committee.
Statistical Analysis
As we had done previously,3,9,10 we computed an overall clinical quality score for each patient in 1998, 2003, 2005, and 2007, which was based on the number of indicators for which appropriate care was provided, divided by the number of indicators relevant to that patient. This score represents the percentage, from 0 to 100%, of "necessary" or "indicated" care provided to the patient. Practice-level quality scores were computed as the mean of individual patient scores in each practice. We computed separate quality scores for the subgroups of indicators that were assigned incentives under the pay-for-performance scheme and for the subgroups that were not assigned incentives.
Data on quality of care had been collected in the same practices in 1998.9 When a pay-for-performance scheme was announced for commencement in 2004, we designed an interrupted time-series study whereby data on quality of care would be collected at two points before the scheme was introduced (1998 and 2003) and at two points after its introduction (2005 and 2007). We use the term "pre-introduction period" to refer to the period from 1998 through 2003, "introduction period" for 2003 through 2005 (from the year before to the year after the implementation of pay for performance), and "post-introduction period" for 2005 through 2007.
We analyzed the data as an interrupted, or segmented, time series. In this model, the within-practice variation was partitioned into three main components to provide independent tests of the slope in scores for the pre-introduction period (test 1); the change in level during the introduction period, allowing for the trend before pay for performance (test 2); and the change in slope from before to after pay for performance (test 3).11 Practice was treated as a random effect, and robust standard-error estimates were used (see the Supplementary Appendix).
The analysis for each outcome measure was conducted in two steps. In step 1, we used the interrupted time-series analysis to look for evidence that pay for performance was having an effect on the trend in scores over time, as indicated by a statistically significant result with respect to either the change in level or the change in slope (tests 2 and 3). The results of these tests determined step 2: if the results of neither test were significant, there was no evidence that pay for performance had affected the preexisting trend and we conducted no further analyses; if the results of either test were significant, there was evidence of an effect and we investigated this further by using the coefficients from the time-series analysis to compare the immediate- and long-term effects of the scheme (i.e., compare the slope during the introduction period with the slope during the post-introduction period) and to estimate the size of the effect on mean quality scores in 2005 and 2007.
We compared the trends in quality scores for the subgroups of indicators associated with incentives and indicators not associated with incentives by means of interactions between indicator set and the changes in level and slope (as defined above) within a regression analysis. If either interaction was significant, we took this as evidence that the trends varied by indicator set and next tested the interaction between indicator set and the change in slope from the introduction period to the post-introduction period.
The quality scores based on medical records and those based on patient evaluation are subject to ceilings of 100%, and many practices achieved this level on at least one indicator. The ceiling necessarily limits any linear trend in improvement, since a score on quality cannot exceed 100%. Analyses were therefore conducted on scores transformed to a logit scale, which has no ceiling, as described previously.3 The transformation increases the weight given to score changes near the ceiling or floor — for example, score changes from 97 to 98% and from 55 to 65% are numerically equivalent (0.41) after transformation. However, where possible, results are re-expressed in original units to facilitate interpretation.
To assess the sensitivity of the findings to our statistical assumptions, we varied the method of statistical inference with the use of a bootstrap method, using 1000 bootstrap samples, and we assumed a linear model for the trend by repeating the analysis on untransformed scores (for details see the Supplementary Appendix). We report any results that differ from those of the primary analysis.
Results
Coronary Heart Disease
The quality of care for coronary heart disease had been improving before the pay-for-performance incentives were introduced (Table 1 and Table 2 and Figure 1). The rate of increase was equivalent to an average of 3.5% per annum from 1998 through 2003 (95% confidence interval [CI], 2.8 to 4.2; P<0.001). In 2005, after the introduction of pay for performance, scores on quality rose slightly, but not significantly, higher than expected, as compared with the trend before the introduction of pay for performance (P=0.06). Subsequently, the rate of improvement dropped below the improvement rates for both the pre-introduction period (P=0.02) and the introduction period (P=0.001), and the overall quality score in 2007 (84.8; 95% CI, 82.2 to 87.4) was similar to that in 2005 (85.0; 95% CI, 83.0 to 87.1) (Table 2 and Table 3 and Figure 1).
|
|
|
|
The quality of care for asthma was improving during the pre-introduction period, at an average rate of 2.0% per annum (95% CI, 0.9 to 3.1; P<0.001), and there was a significant change in the level of quality over and above this trend in 2005 (P<0.001) (Table 1 and Table 2 and Figure 1). However, this accelerated rate of increase was not maintained after 2005 (P=0.001). The trend after 2005 did not differ significantly from the trend before the introduction of pay for performance (P=0.16), although in absolute terms, overall quality hardly changed between 2005 (84.3; 95% CI, 80.6 to 88.1) and 2007 (85.0; 95% CI, 82.2 to 87.8) (Table 2 and Table 3 and Figure 1).
Diabetes
The quality of care for patients with diabetes was improving in the pre-introduction period, at an average rate of 1.8% per annum (95% CI, 1.1 to 2.4; P<0.001) (Table 2 and Table 3 and Figure 1). Diabetes care, like asthma care, showed a significant change in the level of improvement after the introduction of pay for performance that was well above the preexisting trend (P<0.001). As with asthma care, this accelerated rate of improvement was not maintained after 2005 (P<0.001); instead, the rate fell back to the pre-introduction level (P=0.91) (Table 2 and Table 3 and Figure 1).
Effect of Incentives on Quality Scores
Mean quality scores for aspects of care that were linked to incentives were higher than those for care that was not linked to incentives, and this pattern applied to all conditions at all four time points (Figure 2). Allowing for these overall differences, there were further differences over time in the scores for aspects of care that were linked to incentives as compared with those that were not. For heart disease, the scores for aspects of care that were linked to incentives showed a bigger immediate increase when the pay-for-performance system was introduced (P=0.05), although this trend was not significant as calculated in the linear model (P=0.46). The long-term trends (scores in the post-introduction period vs. scores in the pre-introduction period) did not differ significantly (P=0.06). However, the difference was significant when calculated with the use of the bootstrapping method (P=0.05) or the linear model (P=0.03), and in absolute terms, the mean quality score for aspects of care for heart disease that were not linked to incentives declined after 2005, whereas the quality score for care that was linked to incentives increased. For asthma, the immediate effect of pay for performance did not differ between care that was and care that was not linked with incentives (P=1.00), but the trends subsequently diverged (post-introduction period vs. pre-introduction period, P=0.006; post-introduction period vs. introduction period, P=0.05), with the mean score for care that was not linked to incentives declining after 2005, and the mean score for care that was linked to incentives increasing. Trends in diabetes care did not differ at any time according to whether the care was linked to incentives.
|
The percentages of patients able to see a physician within 48 hours, as well as the mean scores on the physician-communication scale, showed no significant changes in trend. Continuity of care declined significantly after the introduction of pay for performance (P<0.001) and remained at this lower level (Table 1 and Table 2 and Figure 2).
Estimated Overall Effect of Pay for Performance
For outcomes in which there was evidence that pay for performance altered the trend in quality improvement, we used coefficients from the interrupted time-series analysis to compute estimates of the increase in scores beyond that expected from the trend in the pre-introduction period (back-transforming the results from the logit analysis, with estimated 95% confidence limits). As compared with the expected level of improvement based on the pre-introduction trend, the pay-for-performance scheme was associated with an improvement in the quality of care for diabetes of 7.5 percentage points in 2005 (95% CI, 4.7 to 10.4) and 6.9 percentage points in 2007 (95% CI, 3.8 to 10.0). For asthma, the increase in quality potentially attributable to pay for performance was 9.4 percentage points in 2005 (95% CI, 3.9 to 15.0) and 5.5 percentage points in 2007 (95% CI, –1.0 to 12.1). For heart disease, pay for performance in 2005 was associated with a nonsignificant improvement in quality above the levels expected (2.8 percentage points; 95% CI, –0.1 to 5.8), and in 2007, it was associated with a nonsignificant reduction in quality from that expected (0.8 percentage points; 95% CI, –4.7 to 3.1). The results of patient evaluations of continuity of care were 4.1 percentage points lower than expected in 2005 (95% CI, –6.1 to –2.0) and 4.3 percentage points lower in 2007 (95% CI, –6.9 to –1.6).
Discussion
We previously found that there were improvements in some aspects of clinical care over and above the underlying trend after the introduction of a pay-for-performance scheme.3 Our current findings suggest that although these initial improvements were maintained, for two of the three conditions studied (heart disease and asthma), improvements in the quality of care reached a plateau a year after the scheme's introduction. Allowing for ceiling effects, care for diabetes continued to improve, but it did so at a rate equivalent to that of the rate in the pre-introduction period.
Within these overall trends for care, we found significant differences between aspects of care that were linked to incentives and aspects of care that were not linked to incentives. For asthma and heart disease, we found a significant difference in the effect of pay for performance on these two groups of quality indicators; for both conditions, mean quality scores for aspects of care that were not linked to incentives dropped between 2005 and 2007, whereas mean scores for aspects of care that were linked to incentives continued to increase. This widening gap in quality came on top of already lower levels of care for indicators not linked to incentives.
For all aspects of care — whether associated with incentives or not — and for all three conditions, rates of quality improvement slowed considerably after 2005. There are several possible explanations. The first is that near-maximal scores had been achieved. However, whereas achievement was high for some indicators (e.g., smoking status recorded for more than 98% of patients for all conditions), the logit transformation theoretically eliminates ceiling effects, and we observed the same plateau effect for indicators reflecting lower levels of achievement. A second explanation is that once initial gains had been made, subsequent gains were more difficult to achieve. A third explanation is that the structure of the pay-for-performance scheme did not reward further improvement once targets had been attained. This explanation is supported by the fact that family practices in our study gained, on average, 96.9% of available clinical-quality payment points in 2005 and 97.8% in 2007 (which were similar to the average gains of 97.1% and 97.5%, respectively, for all family practices in England12,13) — that is, there was little financial incentive for further improvement. A fourth explanation is that family practitioners had sufficient income and had little personal motivation to improve performance and income further (the target-income hypothesis); this explanation would be consistent with the 30 to 40% gains in family practitioners' net income from the 2002–2003 period to the 2005–2006 period.14
Our data cannot be used to ascertain the relative merit of these explanations. However, government negotiators in England appear to endorse the third explanation (too many physicians achieving maximal or near-maximal payments for quality of care). Alterations of the pay-for-performance scheme in 2006 introduced higher thresholds for maximal clinical-quality payments and a wider range of indicators (see the Supplementary Appendix).
This study suggests that continuity of care declined after pay for performance was introduced. One possible explanation is that practices focused on meeting rapid-access targets in which access to any doctor in the practice within 48 hours was linked to incentives but access to a particular physician was not,15 making it more difficult for patients to see their own doctor. This could be an unintended and perverse effect of the scheme and is a concern, since continuity is an aspect of family practice that patients value.16 Another explanation is that there were increases in the size of practices, and many practices introduced nurse-led clinics for management of individual chronic diseases. Although this may have been an important part of improving the quality of care, it may have made continuity of care harder to achieve.
Other studies suggest that financial incentives result in small improvements in quality.17,18 Our data suggest that the pay-for-performance scheme in England attained its quality-improvement goals but that the pace of improvement was not sustained once these goals had been reached. There may be unintended consequences for aspects of care other than those studied, which may be influenced by differences in the operational details of apparently similar incentive schemes.19 An unanticipated benefit of the scheme in England has been a reduction in sociodemographic inequalities in the delivery of health care.20
Our study has several limitations. First, the pay-for-performance scheme was introduced throughout the United Kingdom, thereby precluding a controlled trial and making the use of an interrupted time series the best evaluation method available. The only other time-series analysis of the quality of primary care in England suggests that pay for performance has had a more modest effect than that suggested by our results.21 Second, because practices were observed at only two time points before the introduction of pay for performance, we cannot say whether the rate of improvement was already accelerating as a result of earlier but still ongoing initiatives. Third, the statistical power of our study was such that only moderate-to-large differences in trend were detectable between indicators that were and those that were not associated with incentives. Fourth, response rates for the patient questionnaire were poor (38 to 47%), although there is no reason to suspect any differences in bias at the four study time points. Finally, we focused on three diseases for which substantial efforts had been made to improve the quality of care before the introduction of the pay-for-performance scheme. Pay for performance might have a greater effect on conditions with lower profiles, including some introduced as the scheme developed (e.g., learning disabilities).
In conclusion, between 1998 and 2007, there were significant improvements in measurable aspects of clinical performance with respect to the care provided for three major chronic diseases. The initial acceleration in the underlying rate of quality improvement after the introduction of pay for performance was not sustained. If the aim of pay for performance is to give providers incentives to attain targets, the scheme achieved that aim. There may have been unintended consequences, including reductions in the quality of some aspects of care not linked to incentives and in the continuity of care.
Supported by the Department of Health for England.
Dr. Roland reports serving as an academic adviser to the government and British Medical Association negotiating teams during the development of the U.K. pay-for-performance scheme in 2001 and 2002. No other potential conflict of interest relevant to this article was reported.
The views presented here are those of the authors and not those of the Department of Health.
We thank the staff of all the practices that participated in the study and also Nan Bailey, Michele Bohan, Nicholas Burr, Ella Gaehl, Nicola Small, and Angela Swallow, who contributed to data collection.
Source Information
From the National Primary Care Research and Development Centre, University of Manchester, Manchester (S.M.C., D.R., E.K., B.S., M.R.); and the University of Cambridge General Practice and Primary Care Research Unit, Institute of Public Health, Cambridge (M.R.) — both in the United Kingdom.
Address reprint requests to Dr. Campbell at the National Primary Care Research and Development Centre, University of Manchester, Oxford Rd., Manchester M13 9PL, United Kingdom, or at stephen.campbell{at}manchester.ac.uk.
References
| |||||||||||||||||||||||||||||||||||||||||||||||||
Related Letters:
Pay for Performance and Quality of Care in England
Kearney L., O'Neill D.
Extract |
Full Text |
PDF
N Engl J Med 2009;
361:1709, Oct 22, 2009.
Correspondence
This article has been cited by other articles:
HOME | SUBSCRIBE | SEARCH | CURRENT ISSUE | PAST ISSUES | COLLECTIONS | PRIVACY | TERMS OF USE | HELP | beta.nejm.org Comments and questions? Please contact us. The New England Journal of Medicine is owned, published, and copyrighted © 2009 Massachusetts Medical Society. All rights reserved. |
Previous
