FREE NEJM E-TOC    HOME   |   SUBSCRIBE   |   CURRENT ISSUE   |   PAST ISSUES   |   COLLECTIONS   |   Search Term  Advanced Search

Sign in | Get NEJM's E-Mail Table of Contents — Free | Subscribe

Previous Volume 338:1888-1895 June 25, 1998 Number 26 Next

	Full Text PDF Editorial by Naylor, C. D. Add to Personal Archive Add to Citation Manager Notify a Friend E-mail When Cited PubMed Citation

ABSTRACT

Background To assess the overuse and underuse of medical procedures, various methods have been developed, but their reproducibility has not been evaluated. This study estimates the reproducibility of one commonly used method.

Methods We performed a parallel, three-way replication of the RAND–University of California at Los Angeles appropriateness method as applied to two medical procedures, coronary revascularization and hysterectomy. Three nine-member multidisciplinary panels of experts were composed for each procedure by stratified random sampling from a list of experts nominated by the relevant specialty societies. Each panel independently rated the same set of clinical scenarios in terms of the appropriateness of the relevant procedure on a risk–benefit scale ranging from 1 to 9. Final ratings were used to classify the procedure in each scenario as necessary or not necessary (to evaluate underuse) and inappropriate or not inappropriate (to evaluate overuse). Reproducibility was measured by overall agreement and by the kappa statistic. The criteria for underuse and overuse derived from these ratings were then applied to real populations of patients who had undergone coronary revascularization or hysterectomy.

Results The rates of agreement among the three coronary-revascularization panels were 95, 94, and 96 percent for inappropriate-use scenarios and 93, 92, and 92 percent for necessary-use scenarios. Agreement among the three hysterectomy panels was 88, 70, and 74 percent for inappropriate-use scenarios. Scenarios involving necessary use of hysterectomy were not assessed. The three-way kappa statistic to detect overuse was 0.52 for coronary revascularization and 0.51 for hysterectomy. The three-way kappa statistic to detect underuse of coronary revascularization was 0.83. Application of individual panels' criteria to real populations of patients resulted in a 100 percent variation in the proportion of cases classified as inappropriate and a 20 percent variation in the proportion of cases classified as necessary.

Conclusions The appropriateness method is far from perfect. Appropriateness criteria may be useful in comparing levels of appropriate procedures among populations but should not by themselves be used to direct care for individual patients.

Source Information

From the West Los Angeles Veterans Affairs Medical Center, Los Angeles (P.G.S.); RAND, Santa Monica, Calif. (P.G.S., J.P.K., C.J.K., R.E.P.); the Ann Arbor Veterans Affairs Medical Center and the Departments of Internal Medicine and Health Management and Policy, University of Michigan, Ann Arbor (S.J.B.); and the Harvard School of Public Health, Boston (L.L.L.).

Address reprint requests to Dr. Shekelle at RAND, 1700 Main St., P.O. Box 2138, Santa Monica, CA 90407-2138.

Full Text of this Article

This article has been cited by other articles:

• Halm, E. A., Press, M. J., Tuhrim, S., Wang, J., Rojas, M., Chassin, M. R. (2008). Does Managed Care Affect Quality? Appropriateness, Referral Patterns, and Outcomes of Carotid Endarterectomy. American Journal of Medical Quality 23: 448-456 [Abstract]  
• Hemingway, H., Chen, R., Junghans, C., Timmis, A., Eldridge, S., Black, N., Shekelle, P., Feder, G. (2008). Appropriateness Criteria for Coronary Angiography in Angina: Reliability and Validity. ANN INTERN MED 149: 221-231 [Abstract] [Full Text]  
• Sibai, A. M, Tohme, R. A, Saade, G. A, Ghanem, G., Alam, S., for the Lebanese Interventional Coronary Registry, (2008). The appropriateness of use of coronary angiography in Lebanon: implications for health policy. Health Policy Plan 23: 210-217 [Abstract] [Full Text]  
• Coenen, S., Ferech, M., Haaijer-Ruskamp, F. M, Butler, C. C, Vander Stichele, R. H, Verheij, T. J M, Monnet, D. L, Little, P., Goossens, H., the ESAC Project Group, (2007). European Surveillance of Antimicrobial Consumption (ESAC): quality indicators for outpatient antibiotic use in Europe. Qual Saf Health Care 16: 440-445 [Abstract] [Full Text]  
• Francis, H C, Prys-Picard, C O, Fishwick, D, Stenton, C, Burge, P S, Bradshaw, L M, Ayres, J G, Campbell, S M, Niven, R M. (2007). Defining and investigating occupational asthma: a consensus approach. Occup. Environ. Med. 64: 361-365 [Abstract] [Full Text]  
• Griffin, S C, Barber, J A, Manca, A, Sculpher, M J, Thompson, S G, Buxton, M J, Hemingway, H (2007). Cost effectiveness of clinically appropriate decisions on alternative treatments for angina pectoris: prospective observational study. BMJ 334: 624-624 [Abstract] [Full Text]  
• Smith, K. L., Soriano, T. A., Boal, J. (2007). Brief Communication: National Quality-of-Care Standards in Home-Based Primary Care. ANN INTERN MED 146: 188-192 [Abstract] [Full Text]  
• Halm, E. A., Tuhrim, S., Wang, J. J., Rojas, M., Hannan, E. L., Chassin, M. R. (2007). Has evidence changed practice?: Appropriateness of carotid endarterectomy after the clinical trials. Neurology 68: 187-194 [Abstract] [Full Text]  
• Baicker, K., Buckles, K. S., Chandra, A. (2006). Geographic Variation In The Appropriate Use Of Cesarean Delivery. Health Aff (Millwood) 25: w355-w367 [Abstract] [Full Text]  
• Guttmann, A., Razzaq, A., Lindsay, P., Zagorski, B., Anderson, G. M. (2006). Development of Measures of the Quality of Emergency Department Care for Children Using a Structured Panel Process. Pediatrics 118: 114-123 [Abstract] [Full Text]  
• Wang, C. J., McGlynn, E. A., Brook, R. H., Leonard, C. H., Piecuch, R. E., Hsueh, S. I., Schuster, M. A. (2006). Quality-of-Care Indicators for the Neurodevelopmental Follow-up of Very Low Birth Weight Children: Results of an Expert Panel Process.. Pediatrics 117: 2080-2092 [Abstract] [Full Text]  
• Poston, G. J., Adam, R., Alberts, S., Curley, S., Figueras, J., Haller, D., Kunstlinger, F., Mentha, G., Nordlinger, B., Patt, Y., Primrose, J., Roh, M., Rougier, P., Ruers, T., Schmoll, H. J., Valls, C., Vauthey, N. J.-N., Cornelis, M., Kahan, J. P. (2005). OncoSurge: A Strategy for Improving Resectability With Curative Intent in Metastatic Colorectal Cancer. JCO 23: 7125-7134 [Abstract] [Full Text]  
• Hutchings, A, Raine, R, Sanderson, C, Black, N (2005). An experimental study of determinants of the extent of disagreement within clinical guideline development groups. Qual Saf Health Care 14: 240-245 [Abstract] [Full Text]  
• Engels, Y., Campbell, S., Dautzenberg, M., van den Hombergh, P., Brinkmann, H., Szecsenyi, J., Falcoff, H., Seuntjens, L., Kuenzi, B., Grol, R. (2005). Developing a framework of, and quality indicators for, general practice management in Europe. Fam Pract 22: 215-222 [Abstract] [Full Text]  
• Asch, S. M., McGlynn, E. A., Hogan, M. M., Hayward, R. A., Shekelle, P., Rubenstein, L., Keesey, J., Adams, J., Kerr, E. A. (2004). Comparison of Quality of Care for Patients in the Veterans Health Administration and Patients in a National Sample. ANN INTERN MED 141: 938-945 [Abstract] [Full Text]  
• Campbell, S M, Shield, T, Rogers, A, Gask, L (2004). How do stakeholder groups vary in a Delphi technique about primary mental health care and what factors influence their ratings?. Qual Saf Health Care 13: 428-434 [Abstract] [Full Text]  
• Sheldon, T. A, Cullum, N., Dawson, D., Lankshear, A., Lowson, K., Watt, I., West, P., Wright, D., Wright, J. (2004). What's the evidence that NICE guidance has been implemented? Results from a national evaluation using time series analysis, audit of patients' notes, and interviews. BMJ 329: 999- [Abstract] [Full Text]  
• Chertow, G. M., Normand, S.-L. T., McNeil, B. J. (2004). "Renalism": Inappropriately Low Rates of Coronary Angiography in Elderly Individuals with Renal Insufficiency. J. Am. Soc. Nephrol. 15: 2462-2468 [Abstract] [Full Text]  
• Hakim, R. B., Benedict, M. B., Merrick, N. J. (2004). Quality of Care for Women Undergoing a Hysterectomy: Effects of Insurance and Race/Ethnicity. Am. J. Public Health 94: 1399-1405 [Abstract] [Full Text]  
• Schneider, E. C., Epstein, A. M., Malin, J. L., Kahn, K. L., Emanuel, E. J. (2004). Developing a System to Assess the Quality of Cancer Care: ASCO's National Initiative on Cancer Care Quality. JCO 22: 2985-2991 [Full Text]  
• Taffe, P., Burnand, B., Wietlisbach, V., Vader, J.-P. (2004). Influence of Clinical and Economical Factors on the Expert Rating of Appropriateness of Preoperative Use of Recombinant Erythropoietin in Elective Orthopedic Surgery Patients. Med Decis Making 24: 122-130 [Abstract]  
• Shekelle, P. (2004). The Appropriateness Method. Med Decis Making 24: 228-231  
• Mariotto, A. (2003). A Decree against Inappropriate Medical Care in Italy. ANN INTERN MED 139: 958-958 [Full Text]  
• Gandjour, A., Neumann, I., Lauterbach, K. W. (2003). Appropriateness of invasive cardiovascular interventions in German hospitals (2000-2001): an evaluation using the RAND appropriateness criteria. Eur. J. Cardiothorac. Surg. 24: 571-577 [Abstract] [Full Text]  
• Beyersdorf, F. (2003). Editorial comment. Eur. J. Cardiothorac. Surg. 24: 578-579 [Full Text]  
• Schilling, J., Gerstl, P., Kapetanios, E., Lee, C.-Y., Bertel, O. (2003). Assessment of Indications in Interventional Cardiology: Appropriateness and Necessity of Coronary Angiography and Revascularization. American Journal of Medical Quality 18: 155-163 [Abstract]  
• McGlynn, E. A., Asch, S. M., Adams, J., Keesey, J., Hicks, J., DeCristofaro, A., Kerr, E. A. (2003). The Quality of Health Care Delivered to Adults in the United States. NEJM 348: 2635-2645 [Abstract] [Full Text]  
• Halm, E. A., Chassin, M. R., Tuhrim, S., Hollier, L. H., Popp, A. J., Ascher, E., Dardik, H., Faust, G., Riles, T. S. (2003). Revisiting the Appropriateness of Carotid Endarterectomy. Stroke 34: 1464-1471 [Abstract] [Full Text]  
• Barnato, A. E., Garber, A. M. (2003). Performance of the RAND Appropriateness Criteria. Med Decis Making 23: 177-179  
• Marshall, M N, Shekelle, P G, McGlynn, E A, Campbell, S, Brook, R H, Roland, M O (2003). Can health care quality indicators be transferred between countries?. Qual Saf Health Care 12: 8-12 [Abstract] [Full Text]  
• Sackley, C., Pound, K. (2002). Setting priorities for a discharge plan for stroke patients entering nursing home care. Clin Rehabil 16: 859-866 [Abstract]  
• Campbell, S M, Hann, M, Hacker, J, Durie, A, Thapar, A, Roland, M O (2002). Quality assessment for three common conditions in primary care: validity and reliability of review criteria developed by expert panels for angina, asthma and type 2 diabetes. Qual Saf Health Care 11: 125-130 [Abstract] [Full Text]  
• BERNSTEIN, S. J., LAZARO, P., FITCH, K., AGUILAR, M. D., RIGTER, H., KAHAN, J. P. (2002). Appropriateness of coronary revascularization for patients with chronic stable angina or following an acute myocardial infarction: multinational versus Dutch criteria. Int J Qual Health Care 14: 103-109 [Abstract] [Full Text]  
• Mathew, J. P., Fontes, M. L., Garwood, S., Davis, E., White, W. D., McCloskey, G., Fitch, J. C.K., Afifi, S., Lee, D. L., Kraker, P., Rafferty, T. D., Barash, P. G., Gillam, L., Prokop, E. (2002). Transesophageal Echocardiography Interpretation: A Comparative Analysis Between Cardiac Anesthesiologists and Primary Echocardiographers. Anesth. Analg. 94: 302-309 [Abstract] [Full Text]  
• Quintana, J M, Cabriada, J, Lopez de Tejada, I, Varona, M, Oribe, V, Barrios, B, Arostegui, I, Bilbao, A (2002). Development of explicit criteria for cholecystectomy. Qual Saf Health Care 11: 320-326 [Abstract] [Full Text]  
• Campbell, S M, Braspenning, J, Hutchinson, A, Marshall, M (2002). Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care 11: 358-364 [Abstract] [Full Text]  
• McNeil, B. J. (2001). Hidden Barriers to Improvement in the Quality of Care. NEJM 345: 1612-1620 [Full Text]  
• Shekelle, P. G., MacLean, C. H., Morton, S. C., Wenger, N. S. (2001). Assessing Care of Vulnerable Elders: Methods for Developing Quality Indicators. ANN INTERN MED 135: 647-652 [Full Text]  
• Schneider, E. C., Leape, L. L., Weissman, J. S., Piana, R. N., Gatsonis, C., Epstein, A. M. (2001). Racial Differences in Cardiac Revascularization Rates: Does "Overuse" Explain Higher Rates among White Patients?. ANN INTERN MED 135: 328-337 [Abstract] [Full Text]  
• Hearnshaw, H M, Harker, R M, Cheater, F M, Baker, R H, Grimshaw, G M (2001). Expert consensus on the desirable characteristics of review criteria for improvement of health care quality. Qual Saf Health Care 10: 173-178 [Abstract] [Full Text]  
• Hemingway, H., Crook, A. M., Feder, G., Banerjee, S., Dawson, J. R., Magee, P., Philpott, S., Sanders, J., Wood, A., Timmis, A. D. (2001). Underuse of Coronary Revascularization Procedures in Patients Considered Appropriate Candidates for Revascularization. NEJM 344: 645-654 [Abstract] [Full Text]  
• Shekelle, P. G. (2001). Are Appropriateness Criteria Ready for Use in Clinical Practice?. NEJM 344: 677-678 [Full Text]  
• Hsu, J., Go, A., Selby, J., Pogach, L., Woolf, S. H., Rothemich, S. F., Asch, S. M., Sloss, E. M., Brook, R. H., Kravitz, R. L. (2001). Overuse of Administrative Data to Measure Underuse of Care. JAMA 285: 735-737 [Full Text]  
• Buetow, S A, Coster, G D (2000). New Zealand and United Kingdom experiences with the RAND modified Delphi approach to producing angina and heart failure criteria for quality assessment in general practice. Qual Saf Health Care 9: 222-231 [Abstract] [Full Text]  
• Epstein, A. M., Ayanian, J. Z., Keogh, J. H., Noonan, S. J., Armistead, N., Cleary, P. D., Weissman, J. S., David-Kasdan, J. A., Carlson, D., Fuller, J., Marsh, D., Conti, R. M. (2000). Racial Disparities in Access to Renal Transplantation -- Clinically Appropriate or Due to Underuse or Overuse?. NEJM 343: 1537-1544 [Abstract] [Full Text]  
• Hannan, E. L. (2000). The Continuing Quest for Measuring and Improving Access to Necessary Care. JAMA 284: 2374-2376 [Full Text]  
• Quintana, J. M., Arostegui, I., Azkarate, J., Goenaga, J. I., Guisasola, I., Alfageme, A., Diego, A. (2000). Evaluation by explicit criteria of the use of total hip joint replacement. Rheumatology (Oxford) 39: 1234-1241 [Abstract] [Full Text]  
• Fitch, K., Lazaro, P., Aguilar, M. D., Kahan, J. P., van het Loo, M., Bernstein, S. J. (2000). European criteria for the appropriateness and necessity of coronary revascularization procedures. Eur. J. Cardiothorac. Surg. 18: 380-387 [Abstract] [Full Text]  
• Kalant, N., Berlinguet, M., Diodati, J. G., Dragatakis, L., Marcotte, F. (2000). How valid are utilization review tools in assessing appropriate use of acute care beds?. CMAJ 162: 1809-1813 [Abstract] [Full Text]  
• Conigliaro, J., Whittle, J., Good, C. B., Hanusa, B. H., Passman, L. J., Lofgren, R. P., Allman, R., Ubel, P. A., O'Connor, M., Macpherson, D. S. (2000). Understanding Racial Variation in the Use of Coronary Revascularization Procedures: The Role of Clinical Factors. Arch Intern Med 160: 1329-1335 [Abstract] [Full Text]  
• BRODER, M. S., KANOUSE, D. E., MITTMAN, B. S., BERNSTEIN, S. J. (2000). The Appropriateness of Recommendations for Hysterectomy. Obstet Gynecol 95: 199-205 [Abstract] [Full Text]  
• Bernstein, S J, Brorsson, B, Åberg, T, Emanuelsson, H, Brook, R H, Werkö, L (1999). Appropriateness of referral of coronary angiography patients in Sweden. Heart 81: 470-477 [Abstract] [Full Text]  
• Leape, L. L., Hilborne, L. H., Bell, R., Kamberg, C., Brook, R. H. (1999). Underuse of Cardiac Procedures: Do Women, Ethnic Minorities, and the Uninsured Fail To Receive Needed Revascularization?. ANN INTERN MED 130: 183-192 [Abstract] [Full Text]  
• Kravitz, R. L. (1999). Ethnic Differences in Use of Cardiovascular Procedures: New Insights and New Challenges. ANN INTERN MED 130: 231-233 [Full Text]  
• Wassertheil-Smoller, S., Tobin, J., Steingart, R., Hsu, J., Black, N., Ayanian, J. Z., Shekelle, P. G., Park, R.E., Naylor, C. D. (1998). Assessing the Appropriateness of Medical Care. NEJM 339: 1478-1481 [Full Text]  
• Naylor, C. D. (1998). What is Appropriate Care?. NEJM 338: 1918-1920 [Full Text]