FREE NEJM E-TOC    HOME   |   SUBSCRIBE   |   CURRENT ISSUE   |   PAST ISSUES   |   COLLECTIONS   |   Search Term  Advanced Search

Sign in | Get NEJM's E-Mail Table of Contents — Free | Subscribe

Previous Volume 330:1792-1796 June 23, 1994 Number 25 Next

	Full Text Letters Add to Personal Archive Add to Citation Manager Notify a Friend E-mail When Cited PubMed Citation

ABSTRACT

Background Computer-based diagnostic systems are available commercially, but there has been limited evaluation of their performance. We assessed the diagnostic capabilities of four internal medicine diagnostic systems: Dxplain, Iliad, Meditel, and QMR.

Methods Ten expert clinicians created a set of 105 diagnostically challenging clinical case summaries involving actual patients. Clinical data were entered into each program with the vocabulary provided by the program's developer. Each of the systems produced a ranked list of possible diagnoses for each patient, as did the group of experts. We calculated scores on several performance measures for each computer program.

Results No single computer program scored better than the others on all performance measures. Among all cases and all programs, the proportion of correct diagnoses ranged from 0.52 to 0.71, and the mean proportion of relevant diagnoses ranged from 0.19 to 0.37. On average, less than half the diagnoses on the experts' original list of reasonable diagnoses were suggested by any of the programs. However, each program suggested an average of approximately two additional diagnoses per case that the experts found relevant but had not originally considered.

Conclusions The results provide a profile of the strengths and limitations of these computer programs. The programs should be used by physicians who can identify and use the relevant information and ignore the irrelevant information that can be produced.

Source Information

From the University of Alabama at Birmingham (E.S.B., A.A.S., J.R.J., E.V.B., C.G.C.); InforMed, Inc., St. Davids, Pa. (G.D.W.); the University of Florida, Gainesville (J.A.); the University of Chicago, Chicago (A.L.B.); the Cleveland Clinic Foundation, Cleveland (V.W.D.); the University of Texas, Dallas (E.P.F.); the University of Washington, Seattle (L.D.H.); Hahnemann University, Philadelphia (E.L.M.); Georgetown University, Washington, D.C. (C.E.R.); and Baptist Medical Center Montclair, Birmingham, Ala. (O.D.T.).This study was conducted by the Office of Educational Development, University of Alabama at Birmingham School of Medicine, 933 19th St. South, Birmingham, AL 35294-2041, where reprint requests should be addressed to Dr. Berner.

Full Text of this Article

Related Letters:

Computer-Based Diagnostic Systems
Lehmann E. D., Yolton R. L., Berner E. S., Webster G. D., Shugerman A. A.
Extract | Full Text  
N Engl J Med 1994; 331:1023-1024, Oct 13, 1994. Correspondence

This article has been cited by other articles:

• Drummond, W. H. (2009). Neonatal Informatics--Dream of a Paperless NICU: Part Two: Understanding Clinical Expertise. NeoReviews 10: e527-e537 [Abstract] [Full Text]  
• Kostopoulou, O., Oudhoff, J., Nath, R., Delaney, B. C., Munro, C. W., Harries, C., Holder, R. (2008). Predictors of Diagnostic Accuracy and Safe Management in Difficult Diagnostic Problems in Family Medicine. Med Decis Making 28: 668-680 [Abstract]  
• Bottieau, E., Moreira, J., Clerinx, J., Colebunders, R., Van Gompel, A., Van den Ende, J. (2008). Evaluation of the GIDEON Expert Computer Program for the Diagnosis of Imported Febrile Illnesses. Med Decis Making 28: 435-442 [Abstract]  
• Ramnarayan, P., Cronje, N., Brown, R., Negus, R., Coode, B., Moss, P., Hassan, T., Hamer, W., Britto, J. (2007). Validation of a diagnostic reminder system in emergency medicine: a multi-centre study. Emerg. Med. J. 24: 619-624 [Abstract] [Full Text]  
• Apkon, M., Mattera, J. A., Lin, Z., Herrin, J., Bradley, E. H., Carbone, M., Holmboe, E. S., Gross, C. P., Selter, J. G., Rich, A. S., Krumholz, H. M. (2005). A Randomized Outpatient Trial of a Decision-Support Information Technology Tool. Arch Intern Med 165: 2388-2394 [Abstract] [Full Text]  
• Peldschus, K., Herzog, P., Wood, S. A., Cheema, J. I., Costello, P., Schoepf, U. J. (2005). Computer-Aided Diagnosis as a Second Reader: Spectrum of Findings in CT Studies of the Chest Interpreted as Normal. Chest 128: 1517-1523 [Abstract] [Full Text]  
• Bravata, D. M., McDonald, K. M., Szeto, H., Smith, W. M., Rydzak, C., Owens, D. K. (2004). A Conceptual Framework for Evaluating Information Technologies and Decision Support Systems for Bioterrorism Preparedness and Response. Med Decis Making 24: 192-206 [Abstract]  
• Ramnarayan, P., Kapoor, R. R, Coren, M., Nanduri, V., Tomlinson, A. L, Taylor, P. M, Wyatt, J. C, Britto, J. F (2003). Measuring the Impact of Diagnostic Decision Support on the Quality of Clinical Decision Making: Development of a Reliable and Valid Composite Score. J Am Med Inform Assoc 10: 563-572 [Abstract] [Full Text]  
• Berner, E. S (2003). Diagnostic Decision Support Systems: How to Determine the Gold Standard?. J Am Med Inform Assoc 10: 608-610 [Full Text]  
• Ramnarayan, P., Kapoor, R. R., Coren, M., Nanduri, V., Tomlinson, A. L., Taylor, P. M., Wyatt, J. C., Britto, J. F. (2003). Measuring the Impact of Diagnostic Decision Support on the Quality of Clinical Decision Making: Development of a Reliable and Valid Composite Score. J. Am. Med. Inform. Assoc. 10: 563-572 [Abstract] [Full Text]  
• Berner, E. S. (2003). Diagnostic Decision Support Systems: How to Determine the Gold Standard?. J. Am. Med. Inform. Assoc. 10: 608-610 [Full Text]  
• Graber, M A, VanScoy, D (2003). How well does decision support software perform in the emergency department?. Emerg. Med. J. 20: 426-428 [Abstract] [Full Text]  
• Fraser, H. S. F., Long, W. J., Naimi, S. (2003). Evaluation of a Cardiac Diagnostic Program in a Typical Clinical Setting. J. Am. Med. Inform. Assoc. 10: 373-381 [Abstract] [Full Text]  
• Ramnarayan, P, Tomlinson, A, Rao, A, Coren, M, Winrow, A, Britto, J (2003). ISABEL: a web-based differential diagnostic aid for paediatrics: results from an initial performance evaluation. Arch. Dis. Child. 88: 408-413 [Abstract] [Full Text]  
• Regeniter, A., Steiger, J. U., Scholer, A., Huber, P. R., Siede, W. H. (2003). Windows to the Ward: Graphically Oriented Report Forms. Presentation of Complex, Interrelated Laboratory Data for Electrophoresis/Immunofixation, Cerebrospinal Fluid, and Urinary Protein Profiles. Clin. Chem. 49: 41-50 [Abstract] [Full Text]  
• Ogunyemi, O. I., Clarke, J. R., Ash, N., Webber, B. L. (2002). Combining Geometric and Probabilistic Reasoning for Computer-based Penetrating-Trauma Assessment. J. Am. Med. Inform. Assoc. 9: 273-282 [Abstract] [Full Text]  
• Hripcsak, G., Wilcox, A. (2002). Reference Standards, Judges, and Comparison Subjects: Roles for Experts in Evaluating System Performance. J. Am. Med. Inform. Assoc. 9: 1-15 [Abstract] [Full Text]  
• Aronsky, D., Chan, K. J., Haug, P. J. (2001). Evaluation of a Computerized Diagnostic Decision Support System for Patients with Pneumonia: Study Design Considerations. J. Am. Med. Inform. Assoc. 8: 473-485 [Abstract] [Full Text]  
• Redelmeier, D. A., Ferris, L. E., Tu, J. V., Hux, J. E., Schull, M. J. (2001). Problems for clinical judgement: introducing cognitive psychology as one more basic science. CMAJ 164: 358-360 [Full Text]  
• Payne, T. H. (2000). Computer Decision Support Systems. Chest 118 : 47S-52S [Abstract] [Full Text]  
• Gottschalk, A., Hyzer, M. C., Geer, R. T. (2000). A Comparison of Human and Machine-based Predictions of Successful Weaning from Mechanical Ventilation. Med Decis Making 20: 160-169 [Abstract]  
• Friedman, C. P., Elstein, A. S., Wolf, F. M., Murphy, G. C., Franz, T. M., Heckerling, P. S., Fine, P. L., Miller, T. M., Abraham, V. (1999). Enhancement of Clinicians' Diagnostic Reasoning by Computer-Based Consultation: A Multisite Study of 2 Systems. JAMA 282: 1851-1856 [Abstract] [Full Text]  
• Berner, E. S., Maisiak, R. S., Cobbs, C. G., Taunton, O. D. (1999). Effects of a Decision Support System on Physicians' Diagnostic Performance. J. Am. Med. Inform. Assoc. 6: 420-427 [Abstract] [Full Text]  
• Berner, E. S., Maisiak, R. S. (1999). Influence of Case and Physician Characteristics on Perceptions of Decision Support Systems. J. Am. Med. Inform. Assoc. 6: 428-434 [Abstract] [Full Text]  
• Lemaire, J. B., Schaefer, J. P., Martin, L. A., Faris, P., Ainslie, M. D., Hull, R. D. (1999). Effectiveness of the Quick Medical Reference as a diagnostic tool. CMAJ 161: 725-728 [Abstract] [Full Text]  
• Randolph, A. G., Haynes, R. B., Wyatt, J. C., Cook, D. J., Guyatt, G. H. (1999). Users' Guides to the Medical Literature: XVIII. How to Use an Article Evaluating the Clinical Impact of a Computer-Based Clinical Decision Support System. JAMA 282: 67-74 [Full Text]  
• Hripcsak, G., Kuperman, G. J., Friedman, C., Heitjan, D. F. (1999). A Reliability Study for Evaluating Information Extraction from Radiology Reports. J. Am. Med. Inform. Assoc. 6: 143-150 [Abstract] [Full Text]  
• Guerlain, S. A., Smith, P. J., Obradovich, J. H., Rudmann, S., Strohm, P., Smith, J. W., Svirbely, J., Sachs, L. (1999). Interactive Critiquing as a Form of Decision Support: An Empirical Evaluation. Human Factors: The Journal of the Human Factors and Ergonomics Society 41: 72-89 [Abstract]  
• Westberg, E. E., Miller, R. A. (1999). The Basis for Using the Internet to Support the Information Needs of Primary Care. J. Am. Med. Inform. Assoc. 6: 6-25 [Abstract] [Full Text]  
• Wong, J. B., Compton, C. C. (1998). Case 24-1998- A 76-Year-Old Woman with Cardiac and Renal Failure and Gastrointestinal Bleeding. NEJM 339: 329-337 [Full Text]  
• Kramer, J. M., Cath, A. (1996). Medical Resources and the Internet: Making the Connection. Arch Intern Med 156: 833-842 [Abstract]  
• Johnson, K. B., Feldman, M. J. (1995). Medical Informatics and Pediatrics: Decision-Support Systems. Arch Pediatr Adolesc Med 149: 1371-1380 [Abstract]  
• Elson, R. B., Connelly, D. P. (1995). Computerized Patient Records in Primary Care: Their Role in Mediating Guideline-Driven Physician Behavior Change. Arch Fam Med 4: 698-705 [Abstract]  
• McKee, M., Clarke, A. (1995). Guidelines, enthusiasms, uncertainty, and the limits to purchasing. BMJ 310: 101-104 [Full Text]  
• Lehmann, E. D., Yolton, R. L., Berner, E. S., Webster, G. D., Shugerman, A. A. (1994). Computer-Based Diagnostic Systems. NEJM 331: 1023-1024 [Full Text]  
• Kassirer, J. P. (1994). A Report Card on Computer-Assisted Diagnosis -- The Grade: C. NEJM 330: 1824-1825 [Full Text]