FREE NEJM E-TOC    HOME   |   SUBSCRIBE   |   CURRENT ISSUE   |   PAST ISSUES   |   COLLECTIONS   |   Search Term  Advanced Search

Sign in | Get NEJM's E-Mail Table of Contents — Free | Subscribe

Previous Volume 335:251-256 July 25, 1996 Number 4 Next

Abstract PDF Letters Add to Personal Archive Add to Citation Manager Notify a Friend E-mail When Cited PubMed Citation

ABSTRACT

Background Reports on the comparative performance of physicians are becoming increasingly common. Little is known, however, about the credibility of these reports with target audiences or their influence on the delivery of medical services.

Methods Since 1992, Pennsylvania has published the Consumer Guide to Coronary Artery Bypass Graft Surgery, which lists annual risk-adjusted mortality rates for all hospitals and surgeons providing such surgery in the state. In 1995, we surveyed a randomly selected sample of 50 percent of Pennsylvania cardiologists and cardiac surgeons to find out whether they were aware of the guide and, if so, to determine their views on its usefulness, limitations, and influence on providers.

Results Eighty-two percent of the cardiologists and all the cardiac surgeons were aware of the guide. Only 10 percent of these respondents reported that its mortality rates were "very important" in assessing the performance of a cardiothoracic surgeon. Less than 10 percent reported discussing the guide with more than 10 percent of their patients who were candidates for a coronary-artery bypass graft (CABG). Eighty-seven percent of the cardiologists reported that the guide had a minimal influence or none on their referral recommendations. For both groups, the most important limitations of the guide were the absence of indicators of quality other than mortality (cited by 78 percent), inadequate risk adjustment (79 percent), and the unreliability of data provided by hospitals and surgeons (53 percent). Fifty-nine percent of the cardiologists reported increased difficulty in finding surgeons willing to perform CABG surgery in severely ill patients who required it, and 63 percent of the cardiac surgeons reported that they were less willing to operate on such patients.

Conclusions The Consumer Guide to Coronary Artery Bypass Graft Surgery has limited credibility among cardiovascular specialists. It has little influence on referral recommendations and may introduce a barrier to care for severely ill patients. If publicly released performance reports are intended to guide the choice of providers without impeding access to medical care, strengthening the collaborative process involving physicians may enhance the credibility and usefulness of the reports.

Proponents of performance reports believe they will lead to the selection of high-quality providers and will motivate hospitals and health plans to improve the quality of care they provide.^5,11,12,13 Others have noted that difficulties in adjusting for differences in case mix, problems with the reliability of the underlying clinical data, and random fluctuation of outcomes from year to year may undermine the validity and credibility of comparative data.^{14,15,16,17,18,19} If report cards are not adequately adjusted to account for variations in the risk of a poor outcome, physicians may avoid caring for chronically or severely ill patients, since such patients have a higher risk of an adverse outcome than do less seriously ill patients.

Since 1992, the Pennsylvania Health Care Cost Containment Council, a state agency, has published four volumes of the Consumer Guide to Coronary Artery Bypass Graft Surgery.^20,21,22,23 Each volume lists, by surgeon and by hospital, the number of coronary-artery bypass graft (CABG) surgeries performed in a calendar year; the actual in-hospital mortality rate among patients treated by each surgeon and hospital; and the expected range of in-hospital mortality rates, derived from statistical models that take into account the severity of the patient's illness and coexisting conditions.^24,25,26,27 Each surgeon and hospital receives a grade indicating whether the actual in-hospital mortality rate is significantly lower than the expected range, within the expected range, or higher than the expected range. The Pennsylvania report and a similar report in New York State²⁸ represent the most sophisticated and widely publicized risk-adjusted data on the performance of hospitals and surgeons.

Despite the controversy surrounding the release of report cards, little is known about their importance to health care providers or the extent to which providers use such information to make referrals for their patients. We report here on a survey of two groups of Pennsylvania physicians who have critical roles with respect to CABG surgery: those who refer patients for possible surgery (cardiologists) and those who perform the surgery (cardiac surgeons).

Methods

Survey Sample

Through the American Medical Association Physician Masterfile, which categorizes physicians according to the specialties they report, we identified all cardiologists residing in Pennsylvania as of December 1994. The Masterfile categorizes physicians according to the self-reported specialty, so those who make referrals for CABG surgery but do not identify themselves as cardiologists are excluded.²⁹ We obtained a list of cardiothoracic surgeons who perform CABG surgery from volume III of the Consumer Guide to Coronary Artery Bypass Graft Surgery.²² These sources identified 1214 cardiologists and 171 cardiothoracic surgeons. We used a random-selection procedure to choose approximately 50 percent of each group for the survey. Cardiologists were excluded if they had made no referrals for CABG surgery in the previous year. In both groups, physicians who had moved out of the state before the survey were excluded. We recorded the sex and age of all the physicians in the sample, as well as the board-certification status of all the cardiologists.

Survey Questionnaire

We designed a written questionnaire in which we first asked all eligible respondents to rate on a five-point Likert scale the importance of risk-adjusted mortality and clinical outcomes other than mortality in judging the quality of a cardiothoracic surgeon's performance. We then asked whether the respondent was aware of the Consumer Guide to Coronary Artery Bypass Graft Surgery. If so, we asked a series of additional questions: "How important is the Consumer Guide to the assessment of the quality of a cardiothoracic surgeon?" "In what percentage of cases did you discuss the Consumer Guide ratings with patients?" and (to cardiologists only) "Has the Consumer Guide had any impact on your referral recommendation to patients?" We asked both groups of physicians to rate the importance of eight potential technical limitations of the Consumer Guide on a five-point Likert scale. We asked cardiologists what proportion of patients did not follow their initial referral recommendation and also asked several questions about practice characteristics that might have an effect on the cardiologist's choice of referral.

We also asked questions about changes over time in the level of access to care for severely ill patients. We asked cardiologists whether there was any change, as compared with three years ago, in the level of difficulty in finding a surgeon willing to operate on their most severely ill patients in need of CABG surgery. We asked cardiothoracic surgeons whether there was any change in their willingness to operate on such patients.

The survey was mailed to physicians during the period from April to September 1995. Three weeks after the first mailing, the physicians were sent a reminder, followed by up to two additional questionnaires. Nonrespondents were contacted by telephone to check their eligibility and determine whether they had received the questionnaire. A final questionnaire was sent to eligible nonrespondents.

Statistical Analysis

The analysis of responses to the questions about demographic and practice characteristics, important factors in assessing performance, and changes in access to care for severely ill patients included all the responses we received. The analysis of responses to questions about the content of the Consumer Guide and its influence was limited to responses from physicians reporting that they were aware of the guide. The significance of differences in responses was assessed by a chi-square test for binary response categories and by a Wilcoxon rank-sum test for pairwise comparisons of ordinal scaled responses. Two-tailed P values are reported for all comparisons. More than 95 percent of the respondents answered each question. Nonrespondents were excluded from the analyses.

Results

Table 1 shows the characteristics of the 697 physicians identified by our random-selection procedure (612 cardiologists and 85 cardiac surgeons). Responses were obtained from 434 physicians and from 18 surrogates (in the case of those who had moved or died), for a response rate of 65 percent. The response rate was 64 percent among the cardiologists and 74 percent among the cardiothoracic surgeons. A total of 110 cardiologists and 5 surgeons were ineligible because they were retired (32), were still in training (16), had moved out of state (14), had a subspecialty other than cardiology or cardiothoracic surgery (13), made no referrals for CABG (36), or had died (4). Completed questionnaires were received from 337 eligible physicians (279 cardiologists and 58 cardiac surgeons).

View this table: [in this window] [in a new window]   Table 1. Characteristics of the Study Population.

 
Eighty-two percent of the cardiologists who responded and all the cardiac surgeons were aware of the Consumer Guide. Among the cardiologists who were unaware of it, a disproportionate number were less than 40 years of age (45 percent, vs. 22 percent of those who were aware of the guide; P<0.01), were not board-certified (35 percent vs. 13 percent, P<0.001), had referred fewer than 20 patients in the preceding year (43 percent vs. 27 percent, P=0.02), or had made referrals to a single surgeon in the preceding year (12 percent vs. 3 percent, P<0.01).

Table 2 summarizes the data on the importance of outcomes and the Consumer Guide. Eighty-four percent of the cardiologists identified risk-adjusted mortality in general as very or extremely important, as compared with only 60 percent of the cardiac surgeons (P<0.01). Eighty-seven percent of the cardiologists, but only 74 percent of the cardiac surgeons, thought clinical outcomes other than mortality were a very or extremely important indicator of the quality of a cardiac surgeon's performance (P<0.01).

View this table: [in this window] [in a new window]   Table 2. Survey Respondents' Views on the Importance of Outcomes and the Consumer Guide in Assessing the Quality of a Cardiac Surgeon's Performance.

 
Most of the cardiologists and cardiac surgeons who were aware of the Consumer Guide (70 percent and 68 percent, respectively) reported that its risk-adjusted mortality ratings were not important or were minimally important in assessing the quality of a cardiothoracic surgeon's performance. The majority of the respondents never discussed the Consumer Guide with their patients undergoing CABG surgery. The large majority of the doctors who discussed the ratings reported doing so with less than 10 percent of their patients. Eighty-seven percent of the cardiologists reported that the Consumer Guide had a minimal influence on their referrals or none. Only 2 percent of the cardiologists responded that the Consumer Guide had a "significant impact" on their referrals. These responses did not vary significantly among the cardiologists according to any of the practice characteristics documented (Table 1).

Table 3 shows the potential limitations of the Consumer Guide that the respondents viewed as very or extremely important. The majority of both groups cited the following three problems as very important limitations: "mortality rates are an incomplete indicator of the quality of a surgeon's care," "risk-adjustment methods are inadequate to compare surgeons fairly," and "hospitals and surgeons can manipulate the data."

View this table: [in this window] [in a new window]   Table 3. Limitations of the Consumer Guide Rated by Respondents as Very or Extremely Important.

 
The cardiologists believed that their recommendation was generally the key factor in patients' decision making. Thirty-nine percent of the cardiologists reported that no patient rejected their initial referral recommendation. Another 56 percent reported that between 1 percent and 10 percent of patients did not follow their initial recommendation for a referral.

A majority of both cardiologists and cardiac surgeons reported increased difficulty in providing CABG surgery for the most severely ill patients who needed it (Figure 1). Fifty-nine percent of the cardiologists reported that it had become more difficult or much more difficult to find a surgeon willing to perform cardiac surgery in severely ill patients in need of such surgery; only 10 percent reported that it had become less difficult or much less difficult. Sixty-three percent of the cardiac surgeons reported that they were less willing or much less willing to operate on the most severely ill patients; none were more willing to perform surgery in such patients.

View larger version (5K): [in this window] [in a new window]   Figure 1. Access to Cardiac Surgery for Severely Ill Patients in Need of Such Surgery, as Compared with Access Three Years Earlier. Access to cardiac surgery was assessed on the basis of cardiologists' ratings of the level of difficulty in finding surgeons willing to operate on severely ill patients in need of such surgery and cardiac surgeons' ratings of their willingness to operate on such patients. The numbers above the bars are percentages of respondents.

 
Discussion

Many health policy experts, employers, and consumer representatives consider reports on the outcomes of medical care a critical tool for improving the quality of care in our increasingly market-driven health care system.^11,12,13 The Pennsylvania Consumer Guide to Coronary Artery Bypass Graft Surgery is an important prototype of such reports and reflects the state of the art in most respects.^25,28,30 Clinical data are abstracted from medical records according to a strict protocol. The council verifies diagnostic codes through computerized checks, manual validation, and independent medical-record audit in selected cases. The risk-adjustment model, which is based on the inclusion of multiple clinical risk factors,²⁷ compares favorably with other risk-adjustment models.^31,32 An advisory group of physicians, statisticians, and experts in quality measurement periodically reviews the process of data collection and reporting. The program has been revised in response to questions raised by hospital and provider groups.²⁵ In spite of these favorable characteristics, the majority of the cardiovascular specialists in Pennsylvania believe that the Consumer Guide is not a clinically credible profile of the quality of care.

The most disturbing finding of our survey of cardiovascular specialists is their belief that access to care has decreased for severely ill patients who need CABG surgery. We lack evidence that these beliefs reflect actual problems with access to care. The results of other studies conflict on the issue of access. A recent study suggests that the movement of severely ill patients to an adjacent state has been a measurable effect of New York State's public reporting of data on CABG surgery.³³ On the other hand, data from New York State also show an increase over time in the average severity of illness and the prevalence of coexisting conditions among patients undergoing CABG surgery,³⁴ suggesting that access to care among severely ill patients may have been maintained.

It should be possible to add other measures of performance to data on mortality, thereby lowering one of the three barriers to acceptance of the Consumer Guide among cardiovascular specialists. The two remaining barriers are doubt about the risk-adjustment methods and concern about the reliability of the underlying clinical data. There are several possible explanations for these findings. First, cardiovascular specialists may be questioning the validity of the data in order to vent their displeasure at being monitored. Second, the risk-adjustment models may be excellent, but the respondents may not understand or appreciate them. Third, even though a risk-adjustment model may appropriately categorize large groups of patients, surgeons may believe that operating on severely ill patients increases the probability that they will receive a negative rating if their total caseload is too small to spread the risk adequately.³⁵ We did not directly assess the providers' knowledge of the risk-adjustment methods used in the Consumer Guide. Finally, despite the extensive auditing of these data, providers may observe subtle biases in the reporting of clinical data. One study corroborates the notion that the collection of data for public release may provoke a biased recording of risk factors.¹⁴

It seems prudent to address providers' skepticism about performance reports for several reasons. First, our data support the clinical impression that cardiologists are highly influential in guiding their patients' choice of surgeon. If changing patients' choices is the goal, cardiologists are an important lever. Second, efforts to improve the quality of care for patients who undergo CABG surgery will surely be more successful if made in collaboration with cardiovascular specialists.³⁶ Third, the development of a risk-adjustment system that has credibility among providers could reduce the tendency to avoid providing care for severely ill patients. The state of Pennsylvania has made an effort to incorporate feedback from cardiovascular specialists into the Consumer Guide. Our results suggest that this effort must continue and perhaps be expanded.

The Pennsylvania program is specifically intended to stimulate "consumer choice." Who the consumer is, however, remains unclear. Is the consumer the patient, the physician, or the corporate benefits manager? Furthermore, how important is choice as a means of improving the quality of care? Our finding that many cardiologists refer patients to multiple surgeons and hospitals implies that referring providers have considerable latitude to shift referrals. However, the literature on the effects of performance reports yields no evidence of a systematic alteration in the choice of providers. Hannan et al. found that New York State's performance reports produced no change in "market share" among ranked surgeons or hospitals (with the exception of low-volume surgeons whose operating privileges had been suspended).^30,37 A study of the effect of the Health Care Financing Administration's public release of data on hospital mortality rates found no appreciable redistribution of market share from hospitals with high mortality rates to those with low rates.³⁸ A recent study from northern New England suggests that regional improvements in mortality rates may be achieved without public release of data on mortality.³⁶

Our study has important limitations. We studied one program in one state. As with all surveys, we relied on self-reports of awareness, use, and changes in access. Although we tried to obtain a balanced sample, it may have been weighted toward specialists with negative views of the Consumer Guide. Also, our study does not address other possible consequences of the Pennsylvania program. Some patients may have used the Consumer Guide without the knowledge of their physicians. The guide may have prompted hospitals in Pennsylvania to change processes of care or curtail surgical privileges in order to reduce mortality rates among patients undergoing CABG surgery. Employers may have used the guide to pressure hospitals to take such action.

In summary, the Consumer Guide is an important prototype of a performance report that has been carefully developed and has many positive features. Nevertheless, cardiovascular specialists in Pennsylvania believe that statewide reporting of risk-adjusted mortality rates among patients undergoing CABG surgery has limited clinical credibility and limited usefulness as an indication of the quality of cardiac surgery; they believe, further, that the Consumer Guide may have adversely affected access to CABG surgery for the most severely ill patients who need it. Technical refinements alone may not be sufficient to address these doubts. If public performance reports are intended to guide decisions about medical care in the future, strengthening the collaborative process involving physicians may enhance their credibility and usefulness.

Supported by the Henry J. Kaiser Family Foundation. Dr. Schneider was the recipient of a National Research Service Award (5T 32 PE 11001-8) from the Department of Health and Human Services.

We are indebted to John Ayanian, M.D., M.P.P., Paul Cleary, Ph.D., Ed Guadagnoli, Ph.D., and Paul Hauptman, M.D., for advice about the survey design; to Lee Beerman, M.D., president of the Pennsylvania Chapter of the American College of Cardiology, Gene Robak of the American Medical Association, and Ernest Sessa and the staff of the Pennsylvania Health Care Cost Containment Council for their support; to the cardiovascular specialists who participated in the survey; and to Lisa Iezzoni, M.D., M.Sc., Anthony Komaroff, M.D., and Barbara McNeil, M.D., Ph.D., for their thoughtful comments on earlier versions of the manuscript.

Source Information

From the Section on Health Services and Policy Research, Division of General Medicine, Brigham and Women's Hospital; the Department of Health Care Policy, Harvard Medical School; and the Department of Health Policy and Management, Harvard School of Public Health — all in Boston.

Address reprint requests to Dr. Epstein at the Department of Health Care Policy, Harvard Medical School, 25 Shattuck St., Parcel B, 1st Fl., Boston, MA 02115.

References

1. Epstein A. Performance reports on quality -- prototypes, problems, and prospects. N Engl J Med 1995;333:57-61. [Free Full Text]
2. Annual report of the California Hospital Outcomes Project. Vol. I. Sacramento: California Office of Statewide Health Planning and Development, 1993.
3. Cleveland-area hospital quality outcome measurement and patient satisfaction report. Cleveland: Quality Information Management Corporation, 1994.
4. Report card on health care. Minneapolis: Medica, 1994.
5. General Accounting Office. "Report cards" are useful but significant issues need to be addressed. Washington, D.C.: Government Printing Office, 1994. (Publication no. GAO/HEHS-94-219.)
6. Welch HG, Miller ME, Welch WP. Physician profiling: an analysis of inpatient practice patterns in Florida and Oregon. N Engl J Med 1994;330:607-612. [Free Full Text]
7. Corrigan JM. How do purchasers develop and use performance measures? Med Care 1995;33:Suppl:JS18-JS24. [CrossRef][Medline]
8. Freudenheim M. People want more information for health choices, survey finds. New York Times. September 17, 1995:I38.
9. General Accounting Office. Health care: employers and individual consumers want additional information on quality. Washington, D.C.: Government Printing Office, 1995. (Publication no. GAO/HEHS-95-201.)
10. Survey on outcomes management. New York: Foster Higgins, 1994.
11. Brook RH. Health care reform is on the way: do we want to compete on quality? Ann Intern Med 1994;120:84-86. [Free Full Text]
12. Topol EJ, Califf RM. Scorecard cardiovascular medicine: its impact and future directions. Ann Intern Med 1994;120:65-70. [Free Full Text]
13. Health care report cards. New York Times. July 10, 1995:A12.
14. Green J, Wintfeld N. Report cards on cardiac surgeons: assessing New York State's approach. N Engl J Med 1995;332:1229-1232. [Free Full Text]
15. Iezzoni LI, ed. Risk adjustment for measuring health care outcomes. Ann Arbor, Mich.: Health Administration Press, 1994.
16. Jost TS. Health system reform: forward or backward with quality oversight? JAMA 1994;271:1508-1511. [CrossRef][Medline]
17. Kassirer JP. The use and abuse of practice profiles. N Engl J Med 1994;330:634-636. [Free Full Text]
18. McNeil BJ, Pedersen SH, Gatsonis C. Current issues in profiling quality of care. Inquiry 1992;29:298-307. [Medline]
19. Park RE, Brook RH, Kosecoff J, et al. Explaining variations in hospital death rates: randomness, severity of illness, quality of care. JAMA 1990;264:484-490. [Abstract]
20. A consumer guide to coronary artery bypass graft surgery. Vol. I. 1990 Data. Harrisburg: Pennsylvania Health Care Cost Containment Council, 1992.
21. A consumer guide to coronary artery bypass graft surgery. Vol. II. 1991 Data. Harrisburg: Pennsylvania Health Care Cost Containment Council, 1994.
22. A consumer guide to coronary artery bypass graft surgery. Vol. III. 1992 Data. Harrisburg: Pennsylvania Health Care Cost Containment Council, 1994.
23. A consumer guide to coronary artery bypass graft surgery. Vol. IV. 1993 Data. Harrisburg: Pennsylvania Health Care Cost Containment Council, 1995.
24. Coronary artery bypass graft surgery: a technical report. Vol. I. 1990 Data. Harrisburg: Pennsylvania Health Care Cost Containment Council, 1992.
25. Coronary artery bypass graft surgery: a technical report. Vol. II. 1991 Data. Harrisburg: Pennsylvania Health Care Cost Containment Council, 1994.
26. Coronary artery bypass graft surgery: a technical report. Vol. III. 1992 Data. Harrisburg: Pennsylvania Health Care Cost Containment Council, 1994.
27. Coronary artery bypass graft surgery: a technical report. Vol. IV. 1993 Data. Harrisburg: Pennsylvania Health Care Cost Containment Council, 1995.
28. Hannan EL, Kilburn H Jr, O'Donnell JF, Lukacik G, Shields EP. Adult open heart surgery in New York State: an analysis of risk factors and hospital mortality rates. JAMA 1990;264:2768-2774. [Abstract]
29. Grumbach K, Becker SH, Osborn EHS, Bindman AB. The challenge of defining and counting generalist physicians: an analysis of Physician Masterfile data. Am J Public Health 1995;85:1402-1407. [Free Full Text]
30. Hannan EL, Kilburn H Jr, Racz M, Shields E, Chassin M. Improving the outcomes of coronary artery bypass surgery in New York State. JAMA 1994;271:761-766. [Abstract]
31. Parsonnet V, Dean D, Bernstein AD. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 1989;79:Suppl I:I-3. 
32. Landon B, Iezzoni LI, Ash AS, et al. Judging hospitals by severity-adjusted mortality rates: the case of CABG surgery. Inquiry (in press).
33. Omoigui NA, Miller DP, Brown KJ, et al. Outmigration for coronary bypass surgery in an era of public dissemination of clinical outcomes. Circulation 1996;93:27-33. [Free Full Text]
34. Chassin MR, Hannan EL, DeBuono BA. Benefits and hazards of reporting medical outcomes publicly. N Engl J Med 1996;334:394-398. [Free Full Text]
35. Asch DA, Hershey JC. Why some health policies don't make sense at the bedside. Ann Intern Med 1995;122:846-850. [Free Full Text]
36. O'Connor GT, Plume SK, Olmstead EM, et al. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. JAMA 1996;275:841-846. [Abstract]
37. Hannan EL, Siu AL, Kumar D, Kilburn H Jr, Chassin MR. The decline in coronary artery bypass graft surgery mortality in New York State: the role of surgeon volume. JAMA 1995;273:209-213. [Abstract]
38. Vladeck BC, Goodwin EJ, Myers LP, Sinisi M. Consumers and hospital use: the HCFA "death list." Health Aff (Millwood) 1988;7:122-125. [CrossRef][Medline]

Abstract PDF Letters Add to Personal Archive Add to Citation Manager Notify a Friend E-mail When Cited PubMed Citation

Related Letters:

Cardiac-Surgery Performance Reports
Krieger L. M., Sessa E. J., Schneider E., Epstein A.
Extract | Full Text  
N Engl J Med 1997; 336:442-443, Feb 6, 1997. Correspondence

This article has been cited by other articles:

• Grant, S W, Grayson, A D, Jackson, M, Au, J, Fabri, B M, Grotte, G, Jones, M, Bridgewater, B (2008). Does the choice of risk-adjustment model influence the outcome of surgeon-specific mortality analysis? A retrospective analysis of 14 637 patients under 31 surgeons. Heart 94: 1044-1049 [Abstract] [Full Text]  
• Hollenbeak, C. S., Gorton, C. P., Tabak, Y. P., Jones, J. L., Milstein, A., Johannes, R. S. (2008). Reductions in Mortality Associated With Intensive Public Reporting of Hospital Outcomes. American Journal of Medical Quality 23: 279-286 [Abstract]  
• Bridgewater, B, Keogh, B (2008). Surgical "league tables": Ischaemic heart disease. Heart 94: 936-942 [Full Text]  
• Segev, D. L., Simpkins, C. E., Thompson, R. E., Locke, J. E., Warren, D. S., Montgomery, R. A. (2008). Obesity Impacts Access to Kidney Transplantation. J. Am. Soc. Nephrol. 19: 349-355 [Abstract] [Full Text]  
• Fung, C. H., Lim, Y.-W., Mattke, S., Damberg, C., Shekelle, P. G. (2008). Systematic Review: The Evidence That Publishing Patient Care Performance Data Improves Quality of Care. ANN INTERN MED 148: 111-123 [Abstract] [Full Text]  
• Simunovic, M., Baxter, N. N. (2007). Lymph Node Counts in Colon Cancer Surgery: Lessons for Users of Quality Indicators. JAMA 298: 2194-2195 [Full Text]  
• Westaby, S., Archer, N., Manning, N., Adwani, S., Grebenik, C., Ormerod, O., Pillai, R., Wilson, N. (2007). Comparison of hospital episode statistics and central cardiac audit database in public reporting of congenital heart surgery mortality. BMJ 335: 759-759 [Abstract] [Full Text]  
• Bridgewater, B., Grayson, A. D, Brooks, N., Grotte, G., Fabri, B. M, Au, J., Hooper, T., Jones, M., Keogh, B., on behalf of the North West Quality Improvement Pr, (2007). Has the publication of cardiac surgery outcome data been associated with changes in practice in northwest England: an analysis of 25 730 patients undergoing CABG surgery under 30 surgeons over eight years. Heart 93: 744-748 [Abstract] [Full Text]  
• Shahian, D. M., Edwards, F. H., Ferraris, V. A., Haan, C. K., Rich, J. B., Normand, S.-L. T., DeLong, E. R., O'Brien, S. M., Shewan, C. M., Dokholyan, R. S., Peterson, E. D. (2007). Quality Measurement in Adult Cardiac Surgery: Part 1--Conceptual Framework and Measure Selection. Ann. Thorac. Surg. 83: S3-S12 [Full Text]  
• Casalino, L. P., Alexander, G. C., Jin, L., Konetzka, R. T. (2007). General Internists' Views On Pay-For-Performance And Public Reporting Of Quality Scores: A National Survey. Health Aff (Millwood) 26: 492-499 [Abstract] [Full Text]  
• Pine, M., Jordan, H. S., Elixhauser, A., Fry, D. E., Hoaglin, D. C., Jones, B., Meimban, R., Warner, D., Gonzales, J. (2007). Enhancement of Claims Data to Improve Risk Adjustment of Hospital Mortality. JAMA 297: 71-76 [Abstract] [Full Text]  
• Epstein, A. J. (2006). Do cardiac surgery report cards reduce mortality? Assessing the evidence.. Med Care Res Rev 63: 403-426 [Abstract]  
• Stevenson, D. G. (2006). Is a public reporting approach appropriate for nursing home care?. Journal of Health Politics, Policy and Law 31: 773-810 [Abstract]  
• Jha, A. K., Epstein, A. M. (2006). The predictive accuracy of the new york state coronary artery bypass surgery report-card system.. Health Aff (Millwood) 25: 844-855 [Abstract] [Full Text]  
• Kesselheim, A. S., Ferris, T. G., Studdert, D. M. (2006). Will physician-level measures of clinical performance be used in medical malpractice litigation?. JAMA 295: 1831-1834 [Full Text]  
• Stensland, J., Winter, A. (2006). Do Physician-Owned Cardiac Hospitals Increase Utilization?. Health Aff (Millwood) 25: 119-129 [Abstract] [Full Text]  
• Guru, V., Anderson, G. M., Fremes, S. E., O'Connor, G. T., Grover, F. L., Tu, J. V., Canadian CABG Surgery Quality Indicator Consensus, (2005). The identification and development of Canadian coronary artery bypass graft surgery quality indicators. J. Thorac. Cardiovasc. Surg. 130: 1257-1257 [Abstract] [Full Text]  
• Schwartz, L. M, Woloshin, S., Birkmeyer, J. D (2005). How do elderly patients decide where to go for major surgery? Telephone interview survey. BMJ 331: 821- [Abstract] [Full Text]  
• Wolff, J. L., Boult, C. (2005). Moving beyond Round Pegs and Square Holes: Restructuring Medicare To Improve Chronic Care. ANN INTERN MED 143: 439-445 [Abstract] [Full Text]  
• Williams, S. C., Schmaltz, S. P., Morton, D. J., Koss, R. G., Loeb, J. M. (2005). Quality of Care in U.S. Hospitals as Reflected by Standardized Measures, 2002-2004. NEJM 353: 255-264 [Abstract] [Full Text]  
• Hilton, C. J., Hamilton, J.R. L., Vitale, N., Haaverstad, R. (2005). Effects of 'Bristol' on surgical practice in the United Kingdom. ICVTS 4: 197-199 [Abstract] [Full Text]  
• Thompson, A M, Stonebridge, P A (2005). Building a framework for trust: critical event analysis of deaths in surgical care. BMJ 330: 1139-1142 [Full Text]  
• Ito, H, Sugawara, H (2005). Relationship between accreditation scores and the public disclosure of accreditation reports: a cross sectional study. Qual Saf Health Care 14: 87-92 [Abstract] [Full Text]  
• Werner, R. M., Asch, D. A., Polsky, D. (2005). Racial Profiling: The Unintended Consequences of Coronary Artery Bypass Graft Report Cards. Circulation 111: 1257-1263 [Abstract] [Full Text]  
• Werner, R. M., Asch, D. A. (2005). The Unintended Consequences of Publicly Reporting Quality Information. JAMA 293: 1239-1244 [Abstract] [Full Text]  
• Narins, C. R., Dozier, A. M., Ling, F. S., Zareba, W. (2005). The Influence of Public Reporting of Outcome Data on Medical Decision Making by Physicians. Arch Intern Med 165: 83-87 [Abstract] [Full Text]  
• Saver, B. G. (2004). Commentary on "Opportunities and Challenges for Measuring Cost, Quality, and Clinical Effectiveness in Health Care": The Fault Lies Not in our Stars but in Our System. Med Care Res Rev 61: 151S-160S  
• Keogh, B., Spiegelhalter, D., Bailey, A., Roxburgh, J., Magee, P., Hilton, C. (2004). The legacy of Bristol: public disclosure of individual surgeons' results. BMJ 329: 450-454 [Full Text]  
• Daly, J. M. (2004). Society of Surgical Oncology Presidential Address: Volume, Outcome, and Surgical Specialization. Ann. Surg. Oncol. 11: 107-114 [Full Text]  
• McKee, M. (2004). Not everything that counts can be counted; not everything that can be counted counts. BMJ 328: 153-153 [Full Text]  
• Shahian, D. M. (2004). Improving Cardiac Surgery Quality--Volume, Outcome, Process?. JAMA 291: 246-248 [Full Text]  
• Liu, J. H., Etzioni, D. A., O'Connell, J. B., Maggard, M. A., Hiyama, D. T., Ko, C. Y. (2003). Inpatient Surgery in California: 1990-2000. Arch Surg 138: 1106-1112 [Abstract] [Full Text]  
• Castle, N. G. (2003). Providing Outcomes Information to Nursing Homes: Can It Improve Quality of Care?. Gerontologist 43: 483-492 [Abstract] [Full Text]  
• Sheeran, T. (2003). A Survey of Psychiatric Outpatients on Quality of Care. Psychiatr. Serv. 54: 1031-1033 [Abstract] [Full Text]  
• Marshall, M. N., Shekelle, P. G., Davies, H. T.O., Smith, P. C. (2003). Public Reporting On Quality In The United States And The United Kingdom. Health Aff (Millwood) 22: 134-148 [Abstract] [Full Text]  
• Jacobson, B., Mindell, J., McKee, M. (2003). Hospital mortality league tables. BMJ 326: 777-778 [Full Text]  
• Mor, V., Berg, K., Angelelli, J., Gifford, D., Morris, J., Moore, T. (2003). The Quality of Quality Measurement in U.S. Nursing Homes. Gerontologist 43: 37-46 [Abstract] [Full Text]  
• TU, J. V., CAMERON, C. (2003). Impact of an acute myocardial infarction report card in Ontario, Canada. Int J Qual Health Care 15: 131-137 [Abstract] [Full Text]  
• Newhouse, J. P. (2002). Why Is There A Quality Chasm?. Health Aff (Millwood) 21: 13-25 [Abstract] [Full Text]  
• Rider, E. A., Perrin, J. M. (2002). Performance Profiles: The Influence of Patient Satisfaction Data on Physicians' Practice. Pediatrics 109: 752-757 [Abstract] [Full Text]  
• Pace, N. (2002). Surgeons should beware of plans to print individual performance. BMJ 324: 979-979 [Full Text]  
• Naylor, C. D. (2002). Public Profiling of Clinical Performance. JAMA 287: 1323-1325 [Full Text]  
• Adab, P., Rouse, A. M, Mohammed, M. A, Marshall, T. (2002). Performance league tables: the NHS deserves better. BMJ 324: 95-98 [Full Text]  
• Shahian, D. M., Normand, S.-L., Torchiana, D. F., Lewis, S. M., Pastore, J. O., Kuntz, R. E., Dreyer, P. I. (2001). Cardiac surgery report cards: comprehensive review and statistical critique. Ann. Thorac. Surg. 72: 2155-2168 [Abstract] [Full Text]  
• Scanlon, D. P., Chernew, M., Doty, H. E., Smith, D. G. (2001). Options for Assessing PPO Quality: Accreditation and Profiling as Accountability Strategies. Med Care Res Rev 58: 70-100 [Abstract]  
• Schneider, E C, Lieberman, T (2001). Publicly disclosed information about the quality of health care: response of the US public. Qual Saf Health Care 10: 96-103 [Abstract] [Full Text]  
• Davies, H T O (2001). Public release of performance data and quality improvement: internal responses to external data by US health care providers. Qual Saf Health Care 10: 104-110 [Abstract] [Full Text]  
• Ayanian, J. Z., Quinn, T. J. (2001). Quality Of Care For Coronary Heart Disease In Two Countries. Health Aff (Millwood) 20: 55-67 [Abstract] [Full Text]  
• Harlan, B. J. (2001). Statewide reporting of coronary artery surgery results: A view from California. J. Thorac. Cardiovasc. Surg. 121: 409-417 [Full Text]  
• Weil, T. P. (2001). Commentary: Public Disclosure in the Health Field: Is There a Relevant Option?. American Journal of Medical Quality 16: 23-33 [Abstract]  
• Peterson, E. D., DeLong, E. R., Muhlbaier, L. H., Rosen, A. B., Buell, H. E., Kiefe, C. I., Kresowik, T. F. (2000). Challenges in comparing risk-adjusted bypass surgery mortality results: Results from the Cooperative Cardiovascular Project. J Am Coll Cardiol 36: 2174-2184 [Abstract] [Full Text]  
• Shahian, D. M., Yip, W., Westcott, G., Jacobson, J. (2000). Selection of a cardiac surgery provider in the managed care era. J. Thorac. Cardiovasc. Surg. 120: 978-989 [Abstract] [Full Text]  
• Friedenberg, R. M. (2000). Managed Care and Social Justice. Radiology 217: 11-13 [Full Text]  
• Jencks, S. F. (2000). Clinical Performance Measurement--A Hard Sell. JAMA 283: 2015-2016 [Full Text]  
• Marshall, M. N., Shekelle, P. G., Leatherman, S., Brook, R. H. (2000). The Public Release of Performance Data: What Do We Expect to Gain? A Review of the Evidence. JAMA 283: 1866-1874 [Abstract] [Full Text]  
• Epstein, A. M. (2000). Public Release of Performance Data: A Progress Report From the Front. JAMA 283: 1884-1886 [Full Text]  
• Marshall, M. N, Shekelle, P. G, Leatherman, S., Brook, R. H (2000). Public disclosure of performance data: learning from the US experience. Qual Saf Health Care 9: 53-57 [Full Text]  
• Eagle, K. A., Guyton, R. A., Davidoff, R., Ewy, G. A., Fonger, J., Gardner, T. J., Gott, J. P., Herrmann, H. C., Marlow, R. A., Nugent, W. C., O'Connor, G. T., Orszulak, T. A., Rieselbach, R. E., Winters, W. L., Yusuf, S., Gibbons, R. J., Alpert, J. S., Eagle, K. A., Gardner, T. J., Garson, A. Jr., Gregoratos, G., Russell, R. O., Smith, S. C. Jr. (1999). ACC/AHA guidelines for coronary artery bypass graft surgery: A report of the American College of Cardiology/ American Heart Association task force on Practice Guidelines (Committee to revise the 1991 Guidelines for Coronary Artery Bypass Graft Surgery). J Am Coll Cardiol 34: 1262-1347 [Full Text]  
• Cohn, L. H. (1999). WHAT THE CARDIOTHORACIC SURGEON OF THE TWENTY-FIRST CENTURY OUGHT TO BE. J. Thorac. Cardiovasc. Surg. 118: 581-587 [Full Text]  
• Burack, J. H., Impellizzeri, P., Homel, P., Cunningham, J. N. Jr (1999). Public reporting of surgical mortality: a survey of New York State cardiothoracic surgeons. Ann. Thorac. Surg. 68: 1195-1200 [Abstract] [Full Text]  
• Thomas, J. W., Hofer, T. P. (1998). Research Evidence on the Validity of Risk-Adjusted Mortality Rate as a Measure of Hospital Quality of Care. Med Care Res Rev 55: 371-404 [Abstract]  
• McColl, A., Roderick, P., Gabbay, J., Smith, H., Moore, M. (1998). Performance indicators for primary care groups: an evidence based approach. BMJ 317: 1354-1360 [Full Text]  
• Mozes, B., Olmer, L., Galai, N., Simchen, E. (1998). A national study of postoperative mortality associated with coronary artery bypass grafting in Israel. Ann. Thorac. Surg. 66: 1254-1262 [Abstract] [Full Text]  
• Rosenthal, G. E., Chren, M.-M., Lasek, R. J., Landefeld, C. S. (1998). What Patients Should Ask of Consumers' Guides to Health Care Quality. Eval Health Prof 21: 316-331 [Abstract]  
• Marshall, E C., Spiegelhalter, D. J, Sanderson, C., McKee, M. (1998). Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates • Commentary: How robust are rankings? The implications of confidence intervals. BMJ 316: 1701-1705 [Abstract] [Full Text]  
• Epstein, A. M. (1998). Rolling Down the Runway: The Challenges Ahead for Quality Report Cards. JAMA 279: 1691-1696 [Abstract] [Full Text]  
• Schneider, E. C., Epstein, A. M. (1998). Use of Public Performance Reports: A Survey of Patients Undergoing Cardiac Surgery. JAMA 279: 1638-1642 [Abstract] [Full Text]  
• Krieger, L. M., Sessa, E. J., Schneider, E., Epstein, A. (1997). Cardiac-Surgery Performance Reports. NEJM 336: 442-443 [Full Text]  
• Blumenthal, D., Epstein, A. M. (1996). The Role of Physicians in the Future of Quality Management- Part Six of Six. NEJM 335: 1328-1332 [Full Text]  
• (1996). PHYSICIANS' VIEWS ON PERFORMANCE ""REPORT CARDS"". JWatch General 1996: 3-3 [Full Text]